[
https://issues.apache.org/jira/browse/HDDS-16178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chi-Hsuan Huang updated HDDS-16178:
-----------------------------------
Description:
h3. Problem
On a versioning\-enabled overwrite, blocks that were allocated but never
committed can stay attached to the committed key under a stale
{{createVersion}}. Every caller that flattens a key's location groups, most
importantly {{OMKeyRequest.sumBlockLengths}}, then counts those blocks at their
full allocated length in addition to the committed ones.
h3. Root cause
Blocks handed out by {{allocateBlock}} carry the default {{createVersion}} of 0
and a length of the SCM block size. On an overwrite of a versioning\-enabled
bucket, {{OMKeyRequest.prepareFileInfo}} calls {{addNewVersion\(locations,
false, true\)}}, and {{OmKeyLocationInfoGroup.generateNextVersion}}
\({{OmKeyLocationInfoGroup.java:160\-165}}\) files them under map key {{version
\+ 1}} while the block objects still report {{createVersion=0}}:
{code}
OmKeyLocationInfoGroup generateNextVersion\(List newLocationList\) {
Map> newMap = new HashMap<>\(\);
newMap.put\(version \+ 1, new ArrayList<>\(newLocationList\)\);
return new OmKeyLocationInfoGroup\(version \+ 1, newMap\);
}
{code}
If the open key is written to RocksDB and read back before the commit is
processed, {{OmKeyLocationInfoGroup.getFromProtobuf}}
\({{OmKeyLocationInfoGroup.java:139\-148}}\) regroups the entries by their
{{createVersion}}, so the new group's internal map becomes {{{0: \[allocated
blocks\], 1: \[\]}}}, the empty entry coming from the constructor's
{{putIfAbsent}}.
Commit then only touches the latest key. {{OmKeyInfo.updateLocationInfoList}}
\({{OmKeyInfo.java:330\-333}}\) runs
{{keyLocationInfoGroup.removeBlocks\(latestVersion\)}} and
{{addAll\(latestVersion, updatedBlockLocations\)}}, which clears and
repopulates map key 1 and leaves the entries under map key 0 untouched.
{{getLocationList\(\)}} flattens the whole map, so those stale entries are
returned alongside the committed blocks.
{{OMAllocateBlockRequest}} is not affected, because it appends through
{{appendNewBlocks}}, which sets {{createVersion}} on each block.
h3. Evidence
The regrouping was reproduced in an ozone\-manager unit test. An open key built
as an overwrite whose new blocks carry {{createVersion=0}}, written to the open
key table and read back, has groups {{\[v0\[0\], v1\[0, 1\]\]}}: the block
intended for version 1 sits under map key 0 of the version 1 group.
The survival of those entries through commit follows from
{{OmKeyInfo.java:330\-333}} but has not been reproduced end to end. An
integration test using the real client, with the open key flushed before
commit, would confirm it.
h3. Impact
Any consumer of the flattened block list over\-counts.
{{OMKeyRequest.sumBlockLengths}} is used to release quota in
{{OMKeyDeleteRequest}}, {{OMKeysDeleteRequest}}, the FSO variants,
{{OMDirectoriesPurgeRequestWithFSO}} and {{KeyManagerImpl}}, so a delete can
refund more than was charged. The quota repair recount in HDDS\-16127 counts
only the blocks each version created, so it is not affected, which also means
repair and delete disagree on such a key until this is fixed.
h3. Notes
Found while scoping HDDS\-16127. Pinned source commit 4766aa8609. Analysis
assisted by AI tooling \(Claude Code, Opus 5\).
was:
h3. Problem
On a versioning\-enabled overwrite, blocks that were allocated but never
committed can stay attached to the committed key under a stale
{{createVersion}}. Every caller that flattens a key's location groups, most
importantly {{OMKeyRequest.sumBlockLengths}}, then counts those blocks at their
full allocated length in addition to the committed ones.
h3. Root cause
Blocks handed out by {{allocateBlock}} carry the default {{createVersion}} of 0
and a length of the SCM block size. On an overwrite of a versioning\-enabled
bucket, {{OMKeyRequest.prepareFileInfo}} calls {{addNewVersion\(locations,
false, true\)}}, and {{OmKeyLocationInfoGroup.generateNextVersion}}
\({{OmKeyLocationInfoGroup.java:160\-165}}\) files them under map key {{version
\+ 1}} while the block objects still report {{createVersion=0}}:
{code}
OmKeyLocationInfoGroup generateNextVersion\(List newLocationList\) {
Map> newMap = new HashMap<>\(\);
newMap.put\(version \+ 1, new ArrayList<>\(newLocationList\)\);
return new OmKeyLocationInfoGroup\(version \+ 1, newMap\);
}
{code}
If the open key is written to RocksDB and read back before the commit is
processed, {{OmKeyLocationInfoGroup.getFromProtobuf}}
\({{OmKeyLocationInfoGroup.java:139\-148}}\) regroups the entries by their
{{createVersion}}, so the new group's internal map becomes {{{0: \[allocated
blocks\], 1: \[\]}}}, the empty entry coming from the constructor's
{{putIfAbsent}}.
Commit then only touches the latest key. {{OmKeyInfo.updateLocationInfoList}}
\({{OmKeyInfo.java:330\-333}}\) runs
{{keyLocationInfoGroup.removeBlocks\(latestVersion\)}} and
{{addAll\(latestVersion, updatedBlockLocations\)}}, which clears and
repopulates map key 1 and leaves the entries under map key 0 untouched.
{{getLocationList\(\)}} flattens the whole map, so those stale entries are
returned alongside the committed blocks.
{{OMAllocateBlockRequest}} is not affected, because it appends through
{{appendNewBlocks}}, which sets {{createVersion}} on each block.
h3. Evidence
The regrouping was reproduced in an ozone\-manager unit test. An open key built
as an overwrite whose new blocks carry {{createVersion=0}}, written to the open
key table and read back, has groups {{\[v0\[0\], v1\[0, 1\]\]}}: the block
intended for version 1 sits under map key 0 of the version 1 group.
The survival of those entries through commit follows from
{{OmKeyInfo.java:330\-333}} but has not been reproduced end to end. An
integration test using the real client, with the open key flushed before
commit, would confirm it.
h3. Impact
Any consumer of the flattened block list over\-counts.
{{OMKeyRequest.sumBlockLengths}} is used to release quota in
{{OMKeyDeleteRequest}}, {{OMKeysDeleteRequest}}, the FSO variants,
{{OMDirectoriesPurgeRequestWithFSO}} and {{KeyManagerImpl}}, so a delete can
refund more than was charged. After HDDS\-16127 the quota repair recount also
uses it for versioning\-enabled buckets.
h3. Notes
Found while scoping HDDS\-16127. Pinned source commit 4766aa8609. Analysis
assisted by AI tooling \(Claude Code, Opus 5\).
> Allocated blocks can survive under a stale createVersion after a versioned
> overwrite
> ------------------------------------------------------------------------------------
>
> Key: HDDS-16178
> URL: https://issues.apache.org/jira/browse/HDDS-16178
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: Chi-Hsuan Huang
> Priority: Major
>
> h3. Problem
> On a versioning\-enabled overwrite, blocks that were allocated but never
> committed can stay attached to the committed key under a stale
> {{createVersion}}. Every caller that flattens a key's location groups, most
> importantly {{OMKeyRequest.sumBlockLengths}}, then counts those blocks at
> their full allocated length in addition to the committed ones.
> h3. Root cause
> Blocks handed out by {{allocateBlock}} carry the default {{createVersion}} of
> 0 and a length of the SCM block size. On an overwrite of a
> versioning\-enabled bucket, {{OMKeyRequest.prepareFileInfo}} calls
> {{addNewVersion\(locations, false, true\)}}, and
> {{OmKeyLocationInfoGroup.generateNextVersion}}
> \({{OmKeyLocationInfoGroup.java:160\-165}}\) files them under map key
> {{version \+ 1}} while the block objects still report {{createVersion=0}}:
> {code}
> OmKeyLocationInfoGroup generateNextVersion\(List newLocationList\) {
> Map> newMap = new HashMap<>\(\);
> newMap.put\(version \+ 1, new ArrayList<>\(newLocationList\)\);
> return new OmKeyLocationInfoGroup\(version \+ 1, newMap\);
> }
> {code}
> If the open key is written to RocksDB and read back before the commit is
> processed, {{OmKeyLocationInfoGroup.getFromProtobuf}}
> \({{OmKeyLocationInfoGroup.java:139\-148}}\) regroups the entries by their
> {{createVersion}}, so the new group's internal map becomes {{{0: \[allocated
> blocks\], 1: \[\]}}}, the empty entry coming from the constructor's
> {{putIfAbsent}}.
> Commit then only touches the latest key. {{OmKeyInfo.updateLocationInfoList}}
> \({{OmKeyInfo.java:330\-333}}\) runs
> {{keyLocationInfoGroup.removeBlocks\(latestVersion\)}} and
> {{addAll\(latestVersion, updatedBlockLocations\)}}, which clears and
> repopulates map key 1 and leaves the entries under map key 0 untouched.
> {{getLocationList\(\)}} flattens the whole map, so those stale entries are
> returned alongside the committed blocks.
> {{OMAllocateBlockRequest}} is not affected, because it appends through
> {{appendNewBlocks}}, which sets {{createVersion}} on each block.
> h3. Evidence
> The regrouping was reproduced in an ozone\-manager unit test. An open key
> built as an overwrite whose new blocks carry {{createVersion=0}}, written to
> the open key table and read back, has groups {{\[v0\[0\], v1\[0, 1\]\]}}: the
> block intended for version 1 sits under map key 0 of the version 1 group.
> The survival of those entries through commit follows from
> {{OmKeyInfo.java:330\-333}} but has not been reproduced end to end. An
> integration test using the real client, with the open key flushed before
> commit, would confirm it.
> h3. Impact
> Any consumer of the flattened block list over\-counts.
> {{OMKeyRequest.sumBlockLengths}} is used to release quota in
> {{OMKeyDeleteRequest}}, {{OMKeysDeleteRequest}}, the FSO variants,
> {{OMDirectoriesPurgeRequestWithFSO}} and {{KeyManagerImpl}}, so a delete can
> refund more than was charged. The quota repair recount in HDDS\-16127 counts
> only the blocks each version created, so it is not affected, which also means
> repair and delete disagree on such a key until this is fixed.
> h3. Notes
> Found while scoping HDDS\-16127. Pinned source commit 4766aa8609. Analysis
> assisted by AI tooling \(Claude Code, Opus 5\).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
