Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/22535
Change subject: IMPALA-13789: Defer creating Path objects in loading file metadata ...................................................................... IMPALA-13789: Defer creating Path objects in loading file metadata When loading file metadata of a HdfsTable, we create a org.apache.hadoop.fs.Path object for each partition dir before actually loading its file metadata. There Path objects have a large memory footprint as the underlying java.net.URI objects have extra fields extracted from the location string. E.g. a location string that takes 160 bytes has a corresponding Path object that takes 704 bytes. See more details of this example in the JIRA description. These Path objects are used as the keys of several Maps for file metadata loaders. We create them before actually loading the metadata. This patch fixes this by using the location strings as the keys and only creating the Path objects when we start loading file metadata of the partition. Tests: - Ran CORE tests - Analyzed the heap dump during file metadata loading, didn't see lots of Path objects anymore. Change-Id: I6ec1fc932eaf7c833ef6ee6cdb08bba235e38271 --- M fe/src/main/java/org/apache/impala/catalog/CatalogHmsAPIHelper.java M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java 8 files changed, 65 insertions(+), 54 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/22535/1 -- To view, visit http://gerrit.cloudera.org:8080/22535 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6ec1fc932eaf7c833ef6ee6cdb08bba235e38271 Gerrit-Change-Number: 22535 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang <[email protected]>
