[
https://issues.apache.org/jira/browse/HDDS-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18108231#comment-18108231
]
Chia-Chuan Ho edited comment on HDDS-16176 at 8/26/26 8:33 AM:
---------------------------------------------------------------
Thanks [~chihsuan] for the detailed report.
I confirmed the inconsistency and agree that quota accounting should use the
per-block-group value, since each EC block group is encoded independently.
The proposed fix is to:
* calculate committed quota from each block in the newly committed location
group, for both regular and FSO keys;
* apply the same calculation to MPU part commit, overwrite, abort, expired
abort, and completion cleanup;
* use the same block-based calculation when quota repair rebuilds usedBytes
and deleted-key snapshotUsedBytes;
* keep OmKeyInfo.getReplicatedSize() unchanged, since it also has non-quota
callers;
* avoid clamping OmBucketInfo.usedBytes at zero, as that could hide the
inconsistency and discard usage belonging to other keys.
Existing EC buckets would need quota repair after upgrading, because OM cannot
distinguish keys charged with the historical per-key formula from keys charged
with the corrected formula without adding persistent versioning or an upgrade
migration.
One wording detail: the per-block value is never smaller than the per-key
value, but it is not always strictly larger. They are equal when block-group
boundaries align with complete EC stripes; the mismatch occurs when multiple
block groups independently incur partial-stripe parity.
Does this scope and migration approach look right to you, especially including
the MPU paths and requiring quota repair for existing buckets? Thanks!
was (Author: JIRAUSER310086):
Thanks [~chihsuan] for the detailed report.
I confirmed the inconsistency and agree that quota accounting should use the
per-block-group value, since each EC block group is encoded independently.
The proposed fix is to: * calculate committed quota from each block in the
newly committed location group, for both regular and FSO keys;
* apply the same calculation to MPU part commit, overwrite, abort, expired
abort, and completion cleanup;
* use the same block-based calculation when quota repair rebuilds usedBytes
and deleted-key snapshotUsedBytes;
* keep OmKeyInfo.getReplicatedSize() unchanged, since it also has non-quota
callers;
* avoid clamping OmBucketInfo.usedBytes at zero, as that could hide the
inconsistency and discard usage belonging to other keys.
Existing EC buckets would need quota repair after upgrading, because OM cannot
distinguish keys charged with the historical per-key formula from keys charged
with the corrected formula without adding persistent versioning or an upgrade
migration.
One wording detail: the per-block value is never smaller than the per-key
value, but it is not always strictly larger. They are equal when block-group
boundaries align with complete EC stripes; the mismatch occurs when multiple
block groups independently incur partial-stripe parity.
Does this scope and migration approach look right to you, especially including
the MPU paths and requiring quota repair for existing buckets? Thanks!
> EC replicated size is computed per key on commit but per block on delete
> ------------------------------------------------------------------------
>
> Key: HDDS-16176
> URL: https://issues.apache.org/jira/browse/HDDS-16176
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: Chi-Hsuan Huang
> Priority: Major
>
> h3. Problem
> For an EC key whose data spans more than one block, OM charges quota using a
> different formula than it refunds with. Commit charges
> {{QuotaUtil.getReplicatedSize\(dataSize, repConfig\)}}, converting the whole
> key at once. Delete refunds {{OMKeyRequest.sumBlockLengths\(omKeyInfo\)}},
> converting each block on its own. EC parity is added per block, so the two do
> not agree, and the refund is always the larger of the two.
> {{OmBucketInfo.decrUsedBytes}} does not clamp at zero, so deleting such keys
> can drive a bucket's {{usedBytes}} negative.
> h3. Example
> With {{rs\-3\-2\-1024}}, the data stripe is 3 \* 1024 bytes. A key holding
> two blocks of 1024 bytes each:
> * per key: {{getReplicatedSize\(2048\)}} = 2048 \+ 1024 \* 2 parity = 4096,
> which is what commit charges
> * per block: \(1024 \+ 1024 \* 2 parity\) \* 2 = 6144, which is what delete
> refunds
> h3. Which value describes the stored bytes
> The per\-block value does. EC encodes each block group independently, so each
> block carries its own partial\-stripe padding:
> * {{QuotaUtil.getReplicatedSize}} is documented as "Number of bytes required
> to store the dataSize with replication".
> * {{KeyManagerImpl.java:852\-859}}, which reports reclaimed space to SCM,
> computes both {{getReplicatedSize}} and {{getSizePerReplica}} per block, and
> passes {{OMKeyRequest.sumBlockLengths\(info\)}} as the quota released.
> * {{hadoop\-hdds/docs/content/feature/Quota.md}} describes the space quota as
> the storage space a bucket may use, counted after replication.
> RATIS replication is linear, so the two formulas agree there and only EC is
> affected.
> h3. Also affected: snapshotUsedBytes in quota repair
> {{QuotaRepairTask.recalculateDeletedKeyUsages}}
> \({{QuotaRepairTask.java:513}}\) totals the deleted\-key table with
> {{RepeatedOmKeyInfo.getTotalSize\(\)}}, which converts each key as a whole,
> while {{OMKeyDeleteRequest.java:165}} moves the per\-block
> {{sumBlockLengths\(omKeyInfo\)}} into {{snapshotUsedBytes}}. The same
> divergence therefore applies to that counter, and the formula chosen here
> should be applied there too. That total is wrong for a second, independent
> reason as well, tracked in HDDS\-16184.
> h3. Impact
> Space quota accounting for EC buckets is internally inconsistent. A bucket
> can report a lower {{usedBytes}} than the space its keys occupy, and repeated
> create/delete cycles can push {{usedBytes}} below zero.
> h3. Notes
> Relates to HDDS\-16127, which changes quota repair to recount keys retaining
> more than one version by converting each version once, matching the commit
> path. That patch deliberately does not adopt the per\-block conversion, so
> that repair reproduces what commit charged rather than pre\-empting the
> decision here.
> Pinned source commit 4766aa8609. Analysis assisted by AI tooling \(Claude
> Code, Opus 5\).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
