Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18841 )
Change subject: IMPALA-10213: Add test for local vs remote scheduling ...................................................................... IMPALA-10213: Add test for local vs remote scheduling Impala already supports locality-aware scheduling with Ozone because it returns location data on partitions. That data doesn't include specific storage ids in getStorageIds, so we skip a warning that will always trigger on Ozone. Updates Ozone to add implicit rules mapping localhost -> 127.0.0.1 for local development. HDFS translates localhost to 127.0.0.1 for host names in its location data, which Impala will identify as colocated with executors in the dev environment. Ozone doesn't, and the default Impala hostname is the machine hostname - not localhost - so without this change all HDFS access in the minicluster is local but all Ozone access is remote. Adds a test to verify local vs remote assignment by using custom clusters with hostnames that either do or don't match storage hostnames. Change-Id: I4e5606528404c3d4fd164c03dec8315345be5f6d Reviewed-on: http://gerrit.cloudera.org:8080/18841 Tested-by: Impala Public Jenkins <[email protected]> Reviewed-by: Joe McDonnell <[email protected]> --- M be/src/runtime/io/disk-io-mgr.cc M be/src/scheduling/executor-group.cc M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M tests/common/skip.py A tests/custom_cluster/test_scheduler_locality.py M tests/metadata/test_stats_extrapolation.py 6 files changed, 65 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Verified Joe McDonnell: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/18841 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I4e5606528404c3d4fd164c03dec8315345be5f6d Gerrit-Change-Number: 18841 Gerrit-PatchSet: 9 Gerrit-Owner: Michael Smith <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Joe McDonnell <[email protected]> Gerrit-Reviewer: Kurt Deschler <[email protected]> Gerrit-Reviewer: Michael Smith <[email protected]>
