Siyao Meng created HDDS-16127:
---------------------------------
Summary: Quota repair undercounts usedBytes and usedNamespace for
versioning-enabled buckets by collapsing multi-version keys
Key: HDDS-16127
URL: https://issues.apache.org/jira/browse/HDDS-16127
Project: Apache Ozone
Issue Type: Bug
Components: OM
Reporter: Siyao Meng
h3. Problem
For a versioning-enabled bucket, {{ozone admin om quota repair}} undercounts
{{usedBytes}} and {{usedNamespace}} by collapsing a multi-version key to its
latest version, then persists the undercount.
h3. Root cause
Quota repair recomputes each bucket's usage by scanning the active key table
and charging, per key-table entry, {{OmKeyInfo.getReplicatedSize()}} (derived
from the single latest {{dataSize}}) and one namespace unit
(QuotaRepairTask.java:637-651). For a versioning-enabled bucket, live commit
accounting instead accumulates every version: each overwrite commit takes the
versioned branch of {{OMKeyCommitRequest}} that gives the old version no credit
and adds the new version's replicated size plus one namespace unit, while the
committed key-table entry keeps {{dataSize}} equal to the latest version only.
The recount therefore collapses an N-version key to its latest version, and
repair overwrites the physically correct accumulated count.
h3. Trigger
# On a versioning-enabled bucket, commit a key twice (v0 = 300 bytes, v1 = 600
bytes) through the normal write path. Live accounting becomes usedBytes=900,
usedNamespace=2 (both versions retained on disk).
# Run {{ozone admin om quota repair}}. The active-table scan sees one key-table
entry with dataSize=600 and recounts usedBytes=600, usedNamespace=1.
# Repair applies the delta (recount minus live), leaving the bucket at 600 and
1.
h3. Impact
Quota enforcement ({{OMKeyRequest.checkBucketQuotaInBytes}}) and bucket usage
reporting ({{ozone sh bucket info}}) read the persisted counters. After repair
writes 600 instead of 900, enforcement under-counts and permits writes beyond
the bucket's true usage. The undercount is durable, and re-running repair
reproduces it.
h3. Suggested fix
Make the active-table recount version-aware for versioning-enabled buckets: sum
the replicated size over all {{keyLocationVersions}} groups of each
{{OmKeyInfo}} and count one namespace unit per version, consistent with
{{OMKeyCommitRequest}}'s per-commit charge, instead of charging only
{{getReplicatedSize()}} of the latest {{dataSize}} at QuotaRepairTask.java:651.
Add a versioned-bucket case to {{TestQuotaRepairTask}}.
h3. Notes
Discovered via code review under the TLA+ verification effort HDDS-16123. All
code claims were verified against the source and reproduced with an
ozone-manager JUnit test using two real {{OMKeyCommitRequest}} commits and the
real {{QuotaRepairTask}} recount (observed live 900/2, repaired 600/1).
Distinct from HDDS-15997 / PR 10884, which fixes scan hang and partial counts
on worker failure and does not touch versioning or {{dataSize}}. Pinned source
commit 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb. Analysis assisted by AI
tooling (Claude Code, Opus 4.8) via the Specula pipeline.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
