[
https://issues.apache.org/jira/browse/HDDS-16127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18104564#comment-18104564
]
Chi-Hsuan Huang commented on HDDS-16127:
----------------------------------------
While working on this, I found that the versioning side of the bug is a bit
broader than the summary here, so I split it up rather than piling everything
into one patch.
The usedBytes part is fixed in PR
[#11018|https://github.com/apache/ozone/pull/11018]. OmKeyInfo.dataSize only
carries the latest version, but a versioned key keeps the blocks of every
version, so the recount was replacing a correct counter with the latest
version's size. The patch converts each retained version once, the same way the
commit path charges it.
The rest have their own root causes, so I filed follow-ups. Most of these fell
out of the same gap: quota repair is a second implementation of the accounting
the request handlers do, and nothing asserts the two agree.
> Quota repair undercounts usedBytes and usedNamespace for versioning-enabled
> buckets by collapsing multi-version keys
> --------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16127
> URL: https://issues.apache.org/jira/browse/HDDS-16127
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: Siyao Meng
> Assignee: Chi-Hsuan Huang
> Priority: Major
> Labels: pull-request-available
> Attachments: HDDS-16127.001.patch
>
>
> h3. Problem
> For a versioning-enabled bucket, {{ozone admin om quota repair}} undercounts
> {{usedBytes}} and {{usedNamespace}} by collapsing a multi-version key to its
> latest version, then persists the undercount.
> h3. Root cause
> Quota repair recomputes each bucket's usage by scanning the active key table
> and charging, per key-table entry, {{OmKeyInfo.getReplicatedSize()}} (derived
> from the single latest {{dataSize}}) and one namespace unit
> (QuotaRepairTask.java:637-651). For a versioning-enabled bucket, live commit
> accounting instead accumulates every version: each overwrite commit takes the
> versioned branch of {{OMKeyCommitRequest}} that gives the old version no
> credit and adds the new version's replicated size plus one namespace unit,
> while the committed key-table entry keeps {{dataSize}} equal to the latest
> version only. The recount therefore collapses an N-version key to its latest
> version, and repair overwrites the physically correct accumulated count.
> h3. Trigger
> # On a versioning-enabled bucket, commit a key twice (v0 = 300 bytes, v1 =
> 600 bytes) through the normal write path. Live accounting becomes
> usedBytes=900, usedNamespace=2 (both versions retained on disk).
> # Run {{ozone admin om quota repair}}. The active-table scan sees one
> key-table entry with dataSize=600 and recounts usedBytes=600, usedNamespace=1.
> # Repair applies the delta (recount minus live), leaving the bucket at 600
> and 1.
> h3. Impact
> Quota enforcement ({{OMKeyRequest.checkBucketQuotaInBytes}}) and bucket usage
> reporting ({{ozone sh bucket info}}) read the persisted counters. After
> repair writes 600 instead of 900, enforcement under-counts and permits writes
> beyond the bucket's true usage. The undercount is durable, and re-running
> repair reproduces it.
> h3. Suggested fix
> Make the active-table recount version-aware for versioning-enabled buckets:
> sum the replicated size over all {{keyLocationVersions}} groups of each
> {{OmKeyInfo}} and count one namespace unit per version, consistent with
> {{OMKeyCommitRequest}}'s per-commit charge, instead of charging only
> {{getReplicatedSize()}} of the latest {{dataSize}} at
> QuotaRepairTask.java:651. Add a versioned-bucket case to
> {{TestQuotaRepairTask}}.
> h3. Notes
> Discovered via code review under the TLA+ verification effort HDDS-16123. All
> code claims were verified against the source and reproduced with an
> ozone-manager JUnit test using two real {{OMKeyCommitRequest}} commits and
> the real {{QuotaRepairTask}} recount (observed live 900/2, repaired 600/1).
> Distinct from HDDS-15997 / PR 10884, which fixes scan hang and partial counts
> on worker failure and does not touch versioning or {{dataSize}}. Pinned
> source commit 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb. Analysis assisted by
> AI tooling (Claude Code, Opus 4.8) via the Specula pipeline.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
