Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21714
Change subject: IMPALA-13117: (part-1) simplify listWithLocations in FileMetadataLoader ...................................................................... IMPALA-13117: (part-1) simplify listWithLocations in FileMetadataLoader There are two HDFS APIs to list file metadata of a folder - listFiles and listStatus. The first one returns a list of LocatedFileStatus which has the info of block locations. The second one returns a list of FileStatus without block locations. listStatus is usually used on storages that don't have blocks, e.g. S3, or used in scenarios that we want to list basic info (name, length, last modified time) reuse existing file descriptors. This patch simplifies the check on whether to list files with block locations. When the table/partition is marked as HDFS caching enabled, or if there are no existing file descriptors to reuse, we invoke the listFiles API. Also tracks whether there are file changes in FileMetadataLoader to speed up the normal case in hasFilesChangedCompareTo(). Tests - Ran CORE tests Change-Id: Ib690930b57a161c98f07ccc2304860b788e9cd9f --- M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java 5 files changed, 54 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/14/21714/1 -- To view, visit http://gerrit.cloudera.org:8080/21714 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib690930b57a161c98f07ccc2304860b788e9cd9f Gerrit-Change-Number: 21714 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <[email protected]>
