[
https://issues.apache.org/jira/browse/IMPALA-10213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17572603#comment-17572603
]
Michael Smith commented on IMPALA-10213:
----------------------------------------
This is mostly already enabled, because Ozone support uses Impala's HDFS
implementation. Co-located storage uses local paths (which violates the DCHECK
at
https://github.com/apache/impala/blob/4.1.0/be/src/runtime/io/disk-io-mgr.cc#L843).
We don't run into it in local testing much because Ozone returns storage
addresses as {{localhost}} (where-as HDFS converts them to {{127.0.0.1}}) and
our local dev environments look for the hostname or {{127.0.0.1}}. Explicitly
setting {{--hostname=localhost}} (as done in {{test_client_ssl.py}}) triggers
the local filesystem path (and errors out on the DCHECK).
The main thing we're missing is that Ozone returns an array of NULLs for
https://hadoop.apache.org/docs/stable/api/org/apache/hadoop/fs/BlockLocation.html#getStorageIds--,
which results in warnings about "missing disk id information", and we can't
use separate queues for individual disks on an Ozone backend.
> Handle block location for Ozone
> -------------------------------
>
> Key: IMPALA-10213
> URL: https://issues.apache.org/jira/browse/IMPALA-10213
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Reporter: Attila Doroszlai
> Priority: Major
>
> Currently Impala treats Ozone as a remote filesystem, similar to S3A, ADLS
> etc. Ozone provides block location info in its Hadoop-compatible FS
> implementations. Also, Ozone can be colocated with Impala daemons. It would
> be nice if Impala could be improved to use Ozone's location info to support
> locality of execution.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
