[
https://issues.apache.org/jira/browse/HDDS-16126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16126:
------------------------------
Attachment: HDDS-16126.001.patch
> Quota repair applies a stale name-only delta to a recreated same-name bucket
> and drives usage counters negative
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16126
> URL: https://issues.apache.org/jira/browse/HDDS-16126
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: Siyao Meng
> Priority: Major
> Attachments: HDDS-16126.001.patch
>
>
> h3. Problem
> Online quota repair can apply a stale usage delta to the wrong bucket when a
> bucket is deleted and recreated under the same name during the repair window,
> driving the recreated bucket's {{usedBytes}} and {{usedNamespace}} negative.
> h3. Root cause
> Quota repair scans a point-in-time RocksDB checkpoint and submits per-bucket
> deltas as {{BucketQuotaCount}}, which carries only {{volName}},
> {{bucketName}}, and the deltas, with no bucket identity field
> (OmClientProtocol.proto:2411-2419). On apply,
> {{OMQuotaRepairRequest.updateBucketInfo}} resolves the bucket by name, guards
> only for {{null}}, and applies the delta with {{incrUsedBytes}} and
> {{incrUsedNamespace}} (OMQuotaRepairRequest.java:118-126), which are plain
> {{+=}} with no clamp (OmBucketInfo.java:259-275). The scan captures the
> bucket {{objectID}} locally, but the protobuf discards it, so a delta
> computed against one generation is applied to whatever bucket currently holds
> the name.
> h3. Trigger
> # Bucket gen1 over-counts (for example usedNamespace=1, usedBytes=1000) with
> zero live keys, which is the drift quota repair exists to correct. Bucket
> deletion gates on {{isBucketEmpty}} (key table), not on the usage counters,
> so an over-counted empty bucket is deletable.
> # Repair checkpoints and scans gen1, computing a name-only delta of -1 and
> -1000.
> # Before the repair transaction commits, the bucket is deleted and recreated
> under the same name; gen2 has a new objectID and zero usage.
> # The repair transaction commits; {{updateBucketInfo}} resolves the name,
> finds gen2, and applies the stale delta, moving gen2 from 0 to -1 and -1000.
> h3. Impact
> The recreated bucket's durable usage counters go negative. These feed
> key-write quota enforcement ({{OMKeyRequest.checkBucketQuotaInBytes}}) and
> {{ozone admin}} usage reporting: a negative usedBytes inflates available
> headroom and permits writes beyond the configured quota. The wrong count is
> durable with no automatic re-correction.
> h3. Suggested fix
> Add the scanned bucket {{objectID}} (and/or generation or updateID) to
> {{BucketQuotaCount}} and, in {{updateBucketInfo}}, skip the delta unless the
> loaded bucket's {{objectID}} matches the request's captured identity, reusing
> the generation-fencing pattern already used elsewhere in the protocol. As
> defense in depth, guard the counters against going negative.
> h3. Notes
> Discovered via TLA+ formal verification and model checking under HDDS-16123.
> All code claims were verified against the source and reproduced with an
> ozone-manager JUnit test driving the real {{QuotaRepairTask}} scan and the
> real apply path: the scan emitted a name-only delta (diffUsedBytes=-1000,
> diffUsedNamespace=-1) that landed on the recreated generation's objectID,
> leaving usedNamespace=-1, usedBytes=-1000. Pinned source commit
> 9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb. Analysis assisted by AI tooling
> (Claude Code, Opus 4.8) via the Specula pipeline.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
