[
https://issues.apache.org/jira/browse/IMPALA-15117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18097707#comment-18097707
]
ASF subversion and git services commented on IMPALA-15117:
----------------------------------------------------------
Commit 60b03fd36fc19acf8ca5bce029285ab7aaac2af8 in impala's branch
refs/heads/master from Jiyoung Yoo
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=60b03fd36 ]
IMPALA-15117: Optimize getFirstLevelAcidDirPath
In the original implementation of AcidUtils.getFirstLevelAcidDirPath(),
the FileSystem.isDirectory() API was called recursively for each path
component. This introduced unnecessary RPC overhead during table loading.
This change refactors the logic to leverage Path string operations by
traversing upwards from the file's parent directory to detect the ACID
directory structure.
1. Eliminated FileSystem.isDirectory() RPCs:
Since the input filePath is guaranteed to be a valid data file
during metadata load, we can assume that all of its parent paths
are directories
2. Preserved backward compatibility:
The original recursive behavior handled nested subdirectories
inside ACID folders. To maintain the same behavior, this change
preserves the exact same path-merging logic (e.g., returning
"delta_000001_000001_0000/sub_dir" for a file path like
"partition_dir/delta_000001_000001_0000/sub_dir/filename")
Testing:
- Passed core tests.
- Added AcidUtilsTest.testGetFirstLevelAcidDirPath.
Change-Id: I390183039181bb62d6818c8e869738ddf01c2f6b
Reviewed-on: http://gerrit.cloudera.org:8080/24588
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Optimize getFirstLevelAcidDirPath
> ---------------------------------
>
> Key: IMPALA-15117
> URL: https://issues.apache.org/jira/browse/IMPALA-15117
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Affects Versions: Impala 4.5.0
> Reporter: Michael Smith
> Assignee: Jiyoung Yoo
> Priority: Major
> Labels: newbie, performance
>
> This could also apply to Hive.
> When loading ACID table metadata, {{AcidUtils.getFirstLevelAcidDirPath}}
> checks whether each candidate ACID subdirectory name (e.g. {{delta_*}},
> {{base_*}}, {{delete_delta_*}}) exists under a partition directory by calling
> {{FileSystem.isDirectory(path)}} for each candidate in a loop. On distributed
> filesystems each call is a separate RPC.
> Observed via async-profiler on a production Impala catalogd connected to
> Apache Ozone (OFS): 258 samples (94% blocked on RPC) in the stack
> {{AcidUtils.getFirstLevelAcidDirPath → FileSystem.isDirectory →
> OzoneBucket.getFileStatus → OM gRPC call}}. For a partition with N candidate
> paths, N separate {{GetFileStatus}} RPCs are issued where 1 {{listStatus}}
> would suffice.
> We could replace the N {{isDirectory(candidatePath)}} calls with a single
> {{listStatus(partitionDir)}} call and resolve membership locally:
> # Call {{fs.listStatus(partitionDir)}} once to get all immediate children
> with their type flags.
> # Build a local {{Set<String>}} of child directory names from the result.
> # Filter candidate ACID names against the set — no additional RPCs.
> This reduces the cost from O(N) RPCs to 1 regardless of filesystem
> implementation. On object-store-backed filesystems (OFS, S3, ABFS) where each
> {{isDirectory}} is a synchronous RPC, the reduction is especially significant.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
