Chi-Hsuan Huang created HDDS-16183:
--------------------------------------
Summary: Avoid the per-group list copy in
OMKeyRequest.sumBlockLengths
Key: HDDS-16183
URL: https://issues.apache.org/jira/browse/HDDS-16183
Project: Apache Ozone
Issue Type: Improvement
Components: OM
Reporter: Chi-Hsuan Huang
h3. Problem
{{OMKeyRequest.sumBlockLengths}} \({{OMKeyRequest.java:887\-897}}\) totals the
length of every block a key points to, but reaches those blocks through
{{OmKeyLocationInfoGroup.getLocationList\(\)}}, which builds a flattened copy
on every call:
{code}
public static long sumBlockLengths\(OmKeyInfo omKeyInfo\) {
long bytesUsed = 0;
for \(OmKeyLocationInfoGroup group: omKeyInfo.getKeyLocationVersions\(\)\) {
for \(OmKeyLocationInfo locationInfo : group.getLocationList\(\)\) {
bytesUsed \+= QuotaUtil.getReplicatedSize\(
locationInfo.getLength\(\), omKeyInfo.getReplicationConfig\(\)\);
}
}
return bytesUsed;
}
{code}
{{getLocationList\(\)}} carries an explicit warning from its author
\({{OmKeyLocationInfoGroup.java:103\-112}}\):
{quote}
Use this expensive method only when absolutely needed\! It creates a new list
so it is not an O\(1\) operation. Use getLocationLists\(\) instead.
{quote}
It is implemented as
{{locationVersionMap.values\(\).stream\(\).flatMap\(List::stream\).collect\(...\)}},
so each call allocates a stream pipeline and a new {{ArrayList}} per version
group. Summing lengths does not need a flattened list; {{getLocationLists\(\)}}
returns {{locationVersionMap.values\(\)}} directly and an extra nested loop
would visit exactly the same block objects.
h3. Where it is called
{{sumBlockLengths}} is used to compute released quota in
{{OMKeyDeleteRequest.java:165}}, {{OMKeyDeleteRequestWithFSO.java:162}},
{{OMKeysDeleteRequest.java:322}}, {{OmKeysDeleteRequestWithFSO.java:115,156}},
{{OMDirectoriesPurgeRequestWithFSO.java:191}} and
{{OMKeyCommitRequest.java:358}}, which are all per\-request.
The call worth looking at is {{KeyManagerImpl.java:868}}, inside the loop that
builds the reclaimable key list for {{KeyDeletingService}}. That one runs per
pending\-delete key on every service iteration, so the allocation is repeated
across the whole scan.
h3. Verification needed first
This is an allocation observation, not a measurement. Nothing here has been
benchmarked, and the effect may be too small to observe next to the RocksDB
reads and the {{OmKeyInfo}} deserialization that dominate the same loop.
Profiling or a benchmark of the {{KeyDeletingService}} scan should come before
or with the change, so that the issue is closed on evidence rather than on the
shape of the code.
h3. Notes
Behavior is unchanged either way: both accessors expose the same block objects,
so the sum is identical, and the existing delete and purge tests cover it.
Noticed while reviewing HDDS\-16127, which added a second caller of
{{sumBlockLengths}} in the quota repair recount. That caller only runs for keys
retaining more than one version, so it is not the motivation here.
Pinned source commit 4766aa8609. Analysis assisted by AI tooling \(Claude Code,
Opus 5\).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
