Hello Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/22535
to look at the new patch set (#2).
Change subject: IMPALA-13789: Defer creating Path objects in loading file
metadata
......................................................................
IMPALA-13789: Defer creating Path objects in loading file metadata
When loading file metadata of a HdfsTable, we create a
org.apache.hadoop.fs.Path object for each partition dir before actually
loading its file metadata. There Path objects have a large memory
footprint as the underlying java.net.URI objects have extra fields
extracted from the location string. E.g. a location string that takes
160 bytes has a corresponding Path object that takes 704 bytes. See more
details of this example in the JIRA description.
These Path objects are used as the keys of several Maps for file
metadata loaders. We create them before actually loading the metadata.
This patch fixes this by using the location strings as the keys and only
creating the Path objects when we start loading file metadata of the
partition.
Tests:
- Ran CORE tests
- Analyzed the heap dump during file metadata loading, didn't see lots
of Path objects anymore.
Change-Id: I6ec1fc932eaf7c833ef6ee6cdb08bba235e38271
---
M fe/src/main/java/org/apache/impala/catalog/CatalogHmsAPIHelper.java
M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/IcebergFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java
M fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java
M fe/src/main/java/org/apache/impala/common/FileSystemUtil.java
M fe/src/test/java/org/apache/impala/catalog/FileMetadataLoaderTest.java
M fe/src/test/java/org/apache/impala/catalog/HdfsPartitionTest.java
8 files changed, 70 insertions(+), 55 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/35/22535/2
--
To view, visit http://gerrit.cloudera.org:8080/22535
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I6ec1fc932eaf7c833ef6ee6cdb08bba235e38271
Gerrit-Change-Number: 22535
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
