[
https://issues.apache.org/jira/browse/HDDS-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chi-Hsuan Huang updated HDDS-16176:
-----------------------------------
Description:
h3. Problem
For an EC key whose data spans more than one block, OM charges quota using a
different formula than it refunds with. Commit charges
{{QuotaUtil.getReplicatedSize\(dataSize, repConfig\)}}, converting the whole
key at once. Delete refunds {{OMKeyRequest.sumBlockLengths\(omKeyInfo\)}},
converting each block on its own. EC parity is added per block, so the two do
not agree, and the refund is always the larger of the two.
{{OmBucketInfo.decrUsedBytes}} does not clamp at zero, so deleting such keys
can drive a bucket's {{usedBytes}} negative.
h3. Example
With {{rs\-3\-2\-1024}}, the data stripe is 3 \* 1024 bytes. A key holding two
blocks of 1024 bytes each:
* per key: {{getReplicatedSize\(2048\)}} = 2048 \+ 1024 \* 2 parity = 4096,
which is what commit charges
* per block: \(1024 \+ 1024 \* 2 parity\) \* 2 = 6144, which is what delete
refunds
h3. Which value describes the stored bytes
The per\-block value does. EC encodes each block group independently, so each
block carries its own partial\-stripe padding:
* {{QuotaUtil.getReplicatedSize}} is documented as "Number of bytes required to
store the dataSize with replication".
* {{KeyManagerImpl.java:852\-859}}, which reports reclaimed space to SCM,
computes both {{getReplicatedSize}} and {{getSizePerReplica}} per block, and
passes {{OMKeyRequest.sumBlockLengths\(info\)}} as the quota released.
* {{hadoop\-hdds/docs/content/feature/Quota.md}} describes the space quota as
the storage space a bucket may use, counted after replication.
RATIS replication is linear, so the two formulas agree there and only EC is
affected.
h3. Also affected: snapshotUsedBytes in quota repair
{{QuotaRepairTask.recalculateDeletedKeyUsages}}
\({{QuotaRepairTask.java:513}}\) totals the deleted\-key table with
{{RepeatedOmKeyInfo.getTotalSize\(\)}}, which converts each key as a whole,
while {{OMKeyDeleteRequest.java:165}} moves the per\-block
{{sumBlockLengths\(omKeyInfo\)}} into {{snapshotUsedBytes}}. The same
divergence therefore applies to that counter, and the formula chosen here
should be applied there too. That total is wrong for a second, independent
reason as well, tracked in HDDS\-16184.
h3. Impact
Space quota accounting for EC buckets is internally inconsistent. A bucket can
report a lower {{usedBytes}} than the space its keys occupy, and repeated
create/delete cycles can push {{usedBytes}} below zero.
h3. Notes
Relates to HDDS\-16127, which changes quota repair to recount keys retaining
more than one version by converting each version once, matching the commit
path. That patch deliberately does not adopt the per\-block conversion, so that
repair reproduces what commit charged rather than pre\-empting the decision
here.
Pinned source commit 4766aa8609. Analysis assisted by AI tooling \(Claude Code,
Opus 5\).
was:
h3. Problem
For an EC key whose data spans more than one block, OM charges quota using a
different formula than it refunds with. Commit charges
{{QuotaUtil.getReplicatedSize\(dataSize, repConfig\)}}, converting the whole
key at once. Delete refunds {{OMKeyRequest.sumBlockLengths\(omKeyInfo\)}},
converting each block on its own. EC parity is added per block, so the two do
not agree, and the refund is always the larger of the two.
{{OmBucketInfo.decrUsedBytes}} does not clamp at zero, so deleting such keys
can drive a bucket's {{usedBytes}} negative.
h3. Example
With {{rs\-3\-2\-1024}}, the data stripe is 3 \* 1024 bytes. A key holding two
blocks of 1024 bytes each:
* per key: {{getReplicatedSize\(2048\)}} = 2048 \+ 1024 \* 2 parity = 4096,
which is what commit charges
* per block: \(1024 \+ 1024 \* 2 parity\) \* 2 = 6144, which is what delete
refunds
h3. Which value describes the stored bytes
The per\-block value does. EC encodes each block group independently, so each
block carries its own partial\-stripe padding:
* {{QuotaUtil.getReplicatedSize}} is documented as "Number of bytes required to
store the dataSize with replication".
* {{KeyManagerImpl.java:852\-859}}, which reports reclaimed space to SCM,
computes both {{getReplicatedSize}} and {{getSizePerReplica}} per block, and
passes {{OMKeyRequest.sumBlockLengths\(info\)}} as the quota released.
* {{hadoop\-hdds/docs/content/feature/Quota.md}} describes the space quota as
the storage space a bucket may use, counted after replication.
RATIS replication is linear, so the two formulas agree there and only EC is
affected.
h3. Also affected: snapshotUsedBytes in quota repair
{{QuotaRepairTask.recalculateDeletedKeyUsages}}
\({{QuotaRepairTask.java:513}}\) totals the deleted\-key table with
{{RepeatedOmKeyInfo.getTotalSize\(\)}}, which sums
{{OmKeyInfo.getReplicatedSize\(\)}}, i.e. the latest version's {{dataSize}}
converted as a whole. {{OMKeyDeleteRequest.java:165}} moves
{{sumBlockLengths\(omKeyInfo\)}}, covering all versions and converted per
block, into {{snapshotUsedBytes}}. The recount therefore reports less than what
was moved in, for the same two reasons: it collapses multi\-version keys and it
converts per key.
h3. Impact
Space quota accounting for EC buckets is internally inconsistent. A bucket can
report a lower {{usedBytes}} than the space its keys occupy, and repeated
create/delete cycles can push {{usedBytes}} below zero.
h3. Notes
Relates to HDDS\-16127, which changes quota repair to recount
versioning\-enabled buckets with {{sumBlockLengths}}. The non\-versioned
recount path there still converts per key, so the outcome of this issue
determines whether both paths converge.
Pinned source commit 4766aa8609. Analysis assisted by AI tooling \(Claude Code,
Opus 5\).
> EC replicated size is computed per key on commit but per block on delete
> ------------------------------------------------------------------------
>
> Key: HDDS-16176
> URL: https://issues.apache.org/jira/browse/HDDS-16176
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: Chi-Hsuan Huang
> Priority: Major
>
> h3. Problem
> For an EC key whose data spans more than one block, OM charges quota using a
> different formula than it refunds with. Commit charges
> {{QuotaUtil.getReplicatedSize\(dataSize, repConfig\)}}, converting the whole
> key at once. Delete refunds {{OMKeyRequest.sumBlockLengths\(omKeyInfo\)}},
> converting each block on its own. EC parity is added per block, so the two do
> not agree, and the refund is always the larger of the two.
> {{OmBucketInfo.decrUsedBytes}} does not clamp at zero, so deleting such keys
> can drive a bucket's {{usedBytes}} negative.
> h3. Example
> With {{rs\-3\-2\-1024}}, the data stripe is 3 \* 1024 bytes. A key holding
> two blocks of 1024 bytes each:
> * per key: {{getReplicatedSize\(2048\)}} = 2048 \+ 1024 \* 2 parity = 4096,
> which is what commit charges
> * per block: \(1024 \+ 1024 \* 2 parity\) \* 2 = 6144, which is what delete
> refunds
> h3. Which value describes the stored bytes
> The per\-block value does. EC encodes each block group independently, so each
> block carries its own partial\-stripe padding:
> * {{QuotaUtil.getReplicatedSize}} is documented as "Number of bytes required
> to store the dataSize with replication".
> * {{KeyManagerImpl.java:852\-859}}, which reports reclaimed space to SCM,
> computes both {{getReplicatedSize}} and {{getSizePerReplica}} per block, and
> passes {{OMKeyRequest.sumBlockLengths\(info\)}} as the quota released.
> * {{hadoop\-hdds/docs/content/feature/Quota.md}} describes the space quota as
> the storage space a bucket may use, counted after replication.
> RATIS replication is linear, so the two formulas agree there and only EC is
> affected.
> h3. Also affected: snapshotUsedBytes in quota repair
> {{QuotaRepairTask.recalculateDeletedKeyUsages}}
> \({{QuotaRepairTask.java:513}}\) totals the deleted\-key table with
> {{RepeatedOmKeyInfo.getTotalSize\(\)}}, which converts each key as a whole,
> while {{OMKeyDeleteRequest.java:165}} moves the per\-block
> {{sumBlockLengths\(omKeyInfo\)}} into {{snapshotUsedBytes}}. The same
> divergence therefore applies to that counter, and the formula chosen here
> should be applied there too. That total is wrong for a second, independent
> reason as well, tracked in HDDS\-16184.
> h3. Impact
> Space quota accounting for EC buckets is internally inconsistent. A bucket
> can report a lower {{usedBytes}} than the space its keys occupy, and repeated
> create/delete cycles can push {{usedBytes}} below zero.
> h3. Notes
> Relates to HDDS\-16127, which changes quota repair to recount keys retaining
> more than one version by converting each version once, matching the commit
> path. That patch deliberately does not adopt the per\-block conversion, so
> that repair reproduces what commit charged rather than pre\-empting the
> decision here.
> Pinned source commit 4766aa8609. Analysis assisted by AI tooling \(Claude
> Code, Opus 5\).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
