Siyao Meng created HDDS-16125:
---------------------------------
Summary: Quota repair aliases the live cached OmBucketInfo and
double-counts bucket usedBytes after crash and Ratis replay
Key: HDDS-16125
URL: https://issues.apache.org/jira/browse/HDDS-16125
Project: Apache Ozone
Issue Type: Bug
Components: OM
Reporter: Siyao Meng
h3. Problem
On an OM bucket, online quota repair can leave the durable {{usedBytes}}
double-counted after a crash and Ratis replay: the bucket durably records usage
2 for a single 1-byte key.
h3. Root cause
{{OMQuotaRepairRequest.updateBucketInfo}} reads the bucket via
{{getBucketTable().get()}} (a copy), then stores that same object into both the
cache ({{addCacheEntry}}) and the client response map ({{bucketMap.put}}) with
no {{copyObject()}} (OMQuotaRepairRequest.java:120,144,146). Every other
mutating request copies its response bucket for replay safety, for example
{{omBucketInfo.copyObject()}} at OMKeyCommitRequest.java:410. A later key
commit fetches the same live cached object via {{OMKeyRequest.getBucketInfo}}
({{getCacheValue}}, no copy; OMKeyRequest.java:903-910) and mutates it in
place. If the double buffer serializes the repair batch after that in-place
mutation but the key-commit batch is not yet durable, a crash strands a durable
prefix that already contains the key commit's increment under the repair
transaction index; on restart Ratis replays the committed key commit and the
count is applied twice.
h3. Trigger
# Bucket {{usedBytes=1}} (drifted; true usage 0). Repair computes delta -1,
live becomes 0; the response and cache entry alias one object.
# The double buffer begins flushing the repair batch (index1) and pauses before
serialization.
# A 1-byte key commit (index2) reads the aliased live object (0) and increments
it to 1.
# The repair batch serializes the mutated aliased object, so durable index1
records usedBytes=1 (repair's own result is 0).
# A crash before index2 flushes loses the unflushed index2 and the in-memory
cache.
# Restart reloads caches from the durable DB (bucket=1) and Ratis replays
committed index2, driving live and durable usedBytes to 2 while true usage is 1.
h3. Impact
{{OmBucketInfo.getUsedBytes()}} is read by
{{OMKeyRequest.checkBucketQuotaInBytes}} on every key write, so an inflated
count rejects legitimate writes early when a space quota is set, and {{ozone
admin}} bucket usage reporting shows the wrong value. The wrong count is
durable, and re-running repair is subject to the same race.
h3. Suggested fix
Store {{bucketInfo.copyObject()}} into the response map (and/or add a distinct
copy to the cache) at OMQuotaRepairRequest.java:146, mirroring
{{OMKeyCommitRequest}}, so the repair response owns an immutable snapshot and a
later key commit's mutation cannot bleed into the repair's durable write.
h3. Notes
Discovered via TLA+ formal verification and model checking under HDDS-16123.
All code claims were verified against the source and reproduced
deterministically on a single-module ozone-manager JUnit test that drives the
real OM state machine, double buffer, and request handlers (observed
liveCount=2, durableCount=2, trueUsage=1). Pinned source commit
9fbf9ee0cb1bd2f5f5d437b6719ebbe5309351fb. Analysis assisted by AI tooling
(Claude Code, Opus 4.8) via the Specula pipeline.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
