Hello Qifan Chen, Csaba Ringhofer, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/18041
to look at the new patch set (#3).
Change subject: IMPALA-11022: Impala uses wrong file descriptors for Iceberg
tables in local catalog mode
......................................................................
IMPALA-11022: Impala uses wrong file descriptors for Iceberg tables in local
catalog mode
When local catalog mode is used, Impala retrieves the Iceberg
snapshot from CatalogD. The response contains a map of the file
descriptors. The file descriptors contain block location information,
but the hosts are only referred to by indexes. In the Coordinator's
local catalog the host indexes might refer to different hosts than in
CatalogD. This might lead to unnecessary remote reads as scan ranges
are scheduled to random hosts.
This patch properly translates the host index to the coordinators host
list, so block locations remain consistent.
Testing:
* tested manually on a 6-node cluster, and verified that the file
locations are consistent with HDFS
* added unit test to LocalCatalogTest
Change-Id: I253b505846e1cf4d1be445c0d06b2552dc4ba1f8
---
M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
8 files changed, 123 insertions(+), 14 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/18041/3
--
To view, visit http://gerrit.cloudera.org:8080/18041
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I253b505846e1cf4d1be445c0d06b2552dc4ba1f8
Gerrit-Change-Number: 18041
Gerrit-PatchSet: 3
Gerrit-Owner: Zoltan Borok-Nagy <[email protected]>
Gerrit-Reviewer: Csaba Ringhofer <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Qifan Chen <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
