[
https://issues.apache.org/jira/browse/IMPALA-10213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601419#comment-17601419
]
ASF subversion and git services commented on IMPALA-10213:
----------------------------------------------------------
Commit 79e474d3109e1738c66fd01e9bf28a97b0df9c90 in impala's branch
refs/heads/master from Michael Smith
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=79e474d31 ]
IMPALA-10213: Add test for local vs remote scheduling
Impala already supports locality-aware scheduling with Ozone because it
returns location data on partitions. That data doesn't include specific
storage ids in getStorageIds, so we skip a warning that will always
trigger on Ozone.
Updates Ozone to add implicit rules mapping localhost -> 127.0.0.1 for
local development. HDFS translates localhost to 127.0.0.1 for host names
in its location data, which Impala will identify as colocated with
executors in the dev environment. Ozone doesn't, and the default Impala
hostname is the machine hostname - not localhost - so without this
change all HDFS access in the minicluster is local but all Ozone access
is remote.
Adds a test to verify local vs remote assignment by using custom
clusters with hostnames that either do or don't match storage hostnames.
Change-Id: I4e5606528404c3d4fd164c03dec8315345be5f6d
Reviewed-on: http://gerrit.cloudera.org:8080/18841
Tested-by: Impala Public Jenkins <[email protected]>
Reviewed-by: Joe McDonnell <[email protected]>
> Handle block location for Ozone
> -------------------------------
>
> Key: IMPALA-10213
> URL: https://issues.apache.org/jira/browse/IMPALA-10213
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Reporter: Attila Doroszlai
> Assignee: Michael Smith
> Priority: Major
>
> Currently Impala treats Ozone as a remote filesystem, similar to S3A, ADLS
> etc. Ozone provides block location info in its Hadoop-compatible FS
> implementations. Also, Ozone can be colocated with Impala daemons. It would
> be nice if Impala could be improved to use Ozone's location info to support
> locality of execution.
> Look into unit tests to validate that reads in the mini-cluster are
> DISK_LOCAL rather than REMOTE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
