Noemi Pap-Takacs has uploaded this change for review. ( http://gerrit.cloudera.org:8080/24513
Change subject: IMPALA-15061: Refactor and Improve IcebergFileMetadataLoader ...................................................................... IMPALA-15061: Refactor and Improve IcebergFileMetadataLoader IcebergFileMetadataLoader is responsible for converting the ContentFiles read from Iceberg metadata to Impala's internal representation: IcebergFileDescriptors. On file systems supporting StorageIds, it also means loading the files to get the block location information and write it in the file descriptors. This caching mechanism gets the load off the NameNode. Since file loading and conversion are expensive, we apply as many optimizations as possible. Therefore IcebergFileMetadataLoader can operate in different modes during a metadata refresh, depending on the changes that occurred to the table: FULL_LOAD: At first loading and also in the general case when we need to do a full reload. Optimization: When it is a REFRESH and old FDs are available, they are automatically reused by path. Partitions could have been removed, so their IDs are reassigned from 0 to maintain continuity. NO_REMOVED_PARTITIONS: In this case, files may have been added but no data files were removed. This means that no partitions were removed, but delete files could have been added, changing the data files to delete files relations. Optimization: We need to do a full reload of all files, but we can safely reuse old FDs and preserve their partition IDs. New partitions get new IDs starting from the existing numbering. INCREMENTAL: If only appends occurred between snapshots, they add only data files without deletes and does not change the data files to delete files relations. Optimization: Only the newly added files need processing: they are merged into the existing file store. Only new partitions get new IDs starting from the existing numbering. Besides some refactoring, this patch introduced the IcebergFileMetadataLoader.LoadingMode enum and added the NO_REMOVED_PARTITIONS case as an optimization compared to the previous FULL_LOAD logic. Testing: - ran existing e2e tests - added unit tests in FiletadataLoaderTest.java to cover all LoadingModes. Change-Id: I3faa45a311385c65a7e2ec7f46e3305c5de2ee0b --- M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/IcebergTable.java M fe/src/main/java/org/apache/impala/catalog/local/IcebergMetaProvider.java M fe/src/main/java/org/apache/impala/util/IcebergUtil.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java 5 files changed, 308 insertions(+), 95 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/24513/3 -- To view, visit http://gerrit.cloudera.org:8080/24513 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3faa45a311385c65a7e2ec7f46e3305c5de2ee0b Gerrit-Change-Number: 24513 Gerrit-PatchSet: 3 Gerrit-Owner: Noemi Pap-Takacs <[email protected]> Gerrit-Reviewer: Csaba Ringhofer <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
