Hello Yida Wu, Sai Hemanth Gantasala, Michael Smith, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24588
to look at the new patch set (#3).
Change subject: IMPALA-15117: Optimize getFirstLevelAcidDirPath
......................................................................
IMPALA-15117: Optimize getFirstLevelAcidDirPath
In the original implementation of AcidUtils.getFirstLevelAcidDirPath(),
the FileSystem.isDirectory() API was called recursively for each path
component. This introduces unnecessary RPC overhead during table loading.
This change refactors the logic to leverage Path string operations by
traversing upwards from the file's parent directory to detect the ACID
directory structure
1. Eliminated FileSystem.isDirectory() RPCs:
Since the input filePath is guaranteed to be a valid data file
during metadata load, we can assume that all of its parent paths
are directories
2. Preserved backward compatibility:
The original recursive behavior handled nested subdirectories
inside ACID folders. To maintain the same behavior, this change
preserved the exact same path-merging logic (e.g., returning
"delta_000001_000001_0000/subdir" for a file path like
“partition_dir/delta_000001_000001_0000/sub_dir/filename")
Testing: AcidUtilsTest.testGetFirstLevelAcidDirPath
Change-Id: I390183039181bb62d6818c8e869738ddf01c2f6b
---
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/util/AcidUtils.java
M fe/src/test/java/org/apache/impala/util/AcidUtilsTest.java
3 files changed, 79 insertions(+), 38 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/88/24588/3
--
To view, visit http://gerrit.cloudera.org:8080/24588
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I390183039181bb62d6818c8e869738ddf01c2f6b
Gerrit-Change-Number: 24588
Gerrit-PatchSet: 3
Gerrit-Owner: Jiyoung Yoo <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Michael Smith <[email protected]>
Gerrit-Reviewer: Sai Hemanth Gantasala <[email protected]>
Gerrit-Reviewer: Yida Wu <[email protected]>
