Zoltan Borok-Nagy has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18041 )
Change subject: IMPALA-11022: Impala uses wrong file descriptors for Iceberg tables in local catalog mode ...................................................................... IMPALA-11022: Impala uses wrong file descriptors for Iceberg tables in local catalog mode When local catalog mode is used, Impala retrieves the Iceberg snapshot from CatalogD. The response contains a map of the file descriptors. The file descriptors contain block location information, but the hosts are only referred to by indexes. In the Coordinator's local catalog the host indexes might refer to different hosts than in CatalogD. This might lead to unnecessary remote reads as scan ranges are scheduled to random hosts. This patch properly translates the host index to the coordinators host list, so block locations remain consistent. Testing: * tested manually on a 6-node cluster, and verified that the file locations are consistent with HDFS * added unit test to LocalCatalogTest Change-Id: I253b505846e1cf4d1be445c0d06b2552dc4ba1f8 Reviewed-on: http://gerrit.cloudera.org:8080/18041 Reviewed-by: Qifan Chen <[email protected]> Reviewed-by: Csaba Ringhofer <[email protected]> Tested-by: Impala Public Jenkins <[email protected]> --- M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 8 files changed, 123 insertions(+), 14 deletions(-) Approvals: Qifan Chen: Looks good to me, but someone else must approve Csaba Ringhofer: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/18041 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I253b505846e1cf4d1be445c0d06b2552dc4ba1f8 Gerrit-Change-Number: 18041 Gerrit-PatchSet: 4 Gerrit-Owner: Zoltan Borok-Nagy <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Qifan Chen <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
