Chi-Hsuan Huang created HDDS-16175:
--------------------------------------
Summary: usedNamespace is charged per key version but refunded per
key
Key: HDDS-16175
URL: https://issues.apache.org/jira/browse/HDDS-16175
Project: Apache Ozone
Issue Type: Bug
Components: OM
Reporter: Chi-Hsuan Huang
h3. Problem
A bucket's {{usedNamespace}} is charged once per committed key version but
refunded only once per key. After every key in a versioning\-enabled bucket has
been deleted, {{usedNamespace}} stays at a non\-zero value and never returns to
0. The residue permanently consumes the bucket's namespace quota, so
{{OMKeyRequest.checkBucketQuotaInNamespace}} eventually rejects writes on a
bucket that holds fewer keys than its quota allows.
Repeated {{hsync}} commits of the same key charge the counter the same way,
without ever adding a key.
h3. Root cause
{{OMKeyCommitRequest.java:375}} calls {{omBucketInfo.incrUsedNamespace\(1L\)}}
outside the if/else chain, so every commit increments it, including each new
version of an existing key on a versioning\-enabled bucket and each re\-commit
of an hsync'd key. {{OMKeyDeleteRequest.java:169}} refunds a flat
{{decrUsedNamespace\(1L, ...\)}} regardless of how many versions the key holds,
and the neighbouring line 165 releases bytes for all versions via
{{sumBlockLengths}}. Bytes are therefore symmetric while namespace is not.
The per\-version charge was not intended. {{ff351025bc}} \(HDDS\-6709, "Fix
bucket usedBytes while versioning is true"\) changed only the branch condition:
{code}
\- if \(keyToDelete \!= null\) {
\+ if \(keyToDelete \!= null && \!omBucketInfo.getIsVersionEnabled\(\)\) {
{code}
which routed every commit on a versioning\-enabled bucket into the {{else}}
branch that carries the namespace increment. That commit's test,
{{bucketUsedBytesOverWrite}}, asserts {{getUsedBytes\(\)}} only and makes no
assertion about {{usedNamespace}}. The comment introduced by {{9055a11216}}
\(HDDS\-6556\), "if keyToDelete isn't null, usedNamespace needn't check and
increase", was left in place and still contradicts the code.
The hsync case is a second, later regression: {{c351de9914}} \(HDDS\-13756\)
hoisted {{incrUsedNamespace\(1L\)}} fully outside the if/else, so the
{{isSameHsyncKey}} branch at lines 324\-327, which previously did not charge
namespace, now charges on every re\-commit.
h3. Expected semantics
Every other surface treats {{usedNamespace}} as a count of keys:
* {{a58d3f5a00}} \(HDDS\-4277\) introduced the counter with the commit body
"add namespaceQuotaUsage and update it when create and delete key in a bucket".
* {{hadoop\-hdds/docs/content/feature/Quota.md:57}}: "When bucket namespace
quota is enabled, the total number of keys under the bucket, cannot exceed the
bucket namespace quota", and "Namespace quota is a number that represents how
many unique names can be used".
* All six refund sites \({{OMKeyDeleteRequest}}, {{OMKeyDeleteRequestWithFSO}},
{{OMKeysDeleteRequest}}, {{OMDirectoriesPurgeRequestWithFSO}},
{{OMKeyPurgeRequest}}, and the non\-versioned overwrite branch of
{{OMKeyCommitRequest}}\) refund one unit per key.
* The Recon UI renders it as a used/total quota bar, and {{ozone sh bucket
info}} prints it next to {{quotaInNamespace}}.
* {{OzoneRpcClientTests.java:1921}}: "Test create a file twice will not
increase usedNamespace twice".
No commit message, javadoc, document or test in the repository states that a
multi\-version key should consume more than one namespace unit.
h3. Impact
{{usedNamespace}} drifts upward and never recovers on buckets that use
versioning or hsync. Namespace quota enforcement and {{ozone sh bucket info}}
both read the persisted counter.
h3. Notes
Found while scoping HDDS\-16127, which fixes the corresponding {{usedBytes}}
undercount in quota repair and deliberately leaves {{usedNamespace}} unchanged,
because quota repair currently counts one unit per key and therefore corrects
this drift rather than reproducing it.
Pinned source commit 4766aa8609. Analysis assisted by AI tooling \(Claude Code,
Opus 5\).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
