Tamas Mate has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/18226 )
Change subject: IMPALA-10737: Optimize the number of Iceberg API Metadata requests ...................................................................... IMPALA-10737: Optimize the number of Iceberg API Metadata requests Iceberg stores the table metadata next to the data files, when this is accessed through the Iceberg API a filesystem call is executed (HDFS, S3, ADLS). These calls were used in various places during query processing and this patch unifies the Iceberg metadata request in the CatalogD similar to other metadata requests: - CatalogD loads and caches the org.apache.iceberg.BaseTable object. - ImpalaDs requests the org.apache.iceberg.BaseTable from the CatalogD and caches it as well. As a result REFRESH/INVALIDATE METADATA is required to reload any Iceberg metadata changes and the metadata load time is improved. This improvement is more significant for smaller queries, where the metadata request has larger impact on the query execution time. Testing: - Passed Iceberg E2E tests. Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62 --- M common/thrift/CatalogObjects.thrift M common/thrift/CatalogService.thrift M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/catalog/FeIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java A fe/src/main/java/org/apache/impala/catalog/IcebergTableLoadingException.java M fe/src/main/java/org/apache/impala/catalog/iceberg/IcebergCtasTarget.java M fe/src/main/java/org/apache/impala/catalog/local/CatalogdMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/catalog/local/LocalIcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java M testdata/workloads/functional-query/queries/QueryTest/iceberg-insert.test 17 files changed, 229 insertions(+), 175 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/18226/8 -- To view, visit http://gerrit.cloudera.org:8080/18226 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9e62a1fb9753ea1b022c7763047d9ccfd1d27d62 Gerrit-Change-Number: 18226 Gerrit-PatchSet: 8 Gerrit-Owner: Tamas Mate <[email protected]> Gerrit-Reviewer: Gergely Fürnstáhl <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Tamas Mate <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
