Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/21608 )
Change subject: IMPALA-13254: Optimize REFRESH for Iceberg tables ...................................................................... Patch Set 13: (1 comment) http://gerrit.cloudera.org:8080/#/c/21608/13/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java File fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java: http://gerrit.cloudera.org:8080/#/c/21608/13/fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java@243 PS13, Line 243: // When listWithLocations is false, for files support StorageIds, : // call parallelListing here to retrieve the FileStatus : // and specify the listWithLocations parameter as false : nameToFileStatus = parallelListing(fs, filesSupportsStorageIds, false); : for (ContentFile<?> contentFile : filesSupportsStorageIds) { : Path path = FileSystemUtil.createFullyQualifiedPath( : new Path(contentFile.path().toString())); : FileStatus status = nameToFileStatus.get(path); : stats.add(Preconditions.checkNotNull(status)); : } > The difference here is that in the old code, createLocatedFileStatus is cal You're right, we don't load the location info for the fds that are present in oldFdsByPath_ and are unchanged. Good catch about fd.isChanged(), however, if I understand correctly, fd.isChanged() should never be true for files in Iceberg tables, as an Iceberg table is a collection of immutable files (according to the Iceberg spec https://iceberg.apache.org/spec/), i.e. a file should never get overwritten. -- To view, visit http://gerrit.cloudera.org:8080/21608 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8c99a28eb16275efdff52e0ea2711c0c6036719c Gerrit-Change-Number: 21608 Gerrit-PatchSet: 13 Gerrit-Owner: Fu Lili <[email protected]> Gerrit-Reviewer: Fu Lili <[email protected]> Gerrit-Reviewer: Impala Public Jenkins <[email protected]> Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]> Gerrit-Comment-Date: Thu, 15 Aug 2024 17:19:45 +0000 Gerrit-HasComments: Yes
