Hello Kurt Deschler, Joe McDonnell, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18841
to look at the new patch set (#6).
Change subject: IMPALA-10213: Add test for local vs remote scheduling
......................................................................
IMPALA-10213: Add test for local vs remote scheduling
Impala already supports locality-aware scheduling with Ozone because it
returns location data on partitions. That data doesn't include specific
storage ids in getStorageIds, so we skip a warning that will always
trigger on Ozone.
Updates Ozone to add implicit rules mapping localhost -> 127.0.0.1 for
local development. HDFS translates localhost to 127.0.0.1 for host names
in its location data, which Impala will identify as colocated with
executors in the dev environment. Ozone doesn't, and the default Impala
hostname is the machine hostname - not localhost - so without this
change all HDFS access in the minicluster is local but all Ozone access
is remote.
Adds a test to verify local vs remote assignment by using custom
clusters with hostnames that either do or don't match storage hostnames.
Change-Id: I4e5606528404c3d4fd164c03dec8315345be5f6d
---
M be/src/runtime/io/disk-io-mgr.cc
M be/src/scheduling/executor-group.cc
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
A tests/common/network.py
M tests/common/skip.py
A tests/custom_cluster/test_scheduler_locality.py
M tests/metadata/test_stats_extrapolation.py
7 files changed, 94 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/18841/6
--
To view, visit http://gerrit.cloudera.org:8080/18841
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I4e5606528404c3d4fd164c03dec8315345be5f6d
Gerrit-Change-Number: 18841
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Joe McDonnell <[email protected]>
Gerrit-Reviewer: Kurt Deschler <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
