[
https://issues.apache.org/jira/browse/HDDS-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18113373#comment-18113373
]
Chi-Hsuan Huang commented on HDDS-16176:
----------------------------------------
Thanks for waiting, and sorry for the slow reply. [~echonesis], I've checked
the accounting sites, and the scope looks right. Please also include the
incomplete MPU quota repair, covering both the legacy and split-table paths, if
you want to pick this up.
Requiring quota repair after upgrading sounds reasonable. One correction to my
earlier reply: the existing versioning test does exercise multiple location
groups, so my “unreachable” statement was too broad. I've also updated the
issue description. Thanks!
> EC replicated size is computed per key on commit but per block on delete
> ------------------------------------------------------------------------
>
> Key: HDDS-16176
> URL: https://issues.apache.org/jira/browse/HDDS-16176
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: Chi-Hsuan Huang
> Priority: Major
>
> h3. Problem
> For an EC key whose data spans more than one block, OM charges quota using a
> different formula than it refunds with. Commit charges
> {{QuotaUtil.getReplicatedSize\(dataSize, repConfig\)}}, converting the whole
> key at once. Delete refunds {{OMKeyRequest.sumBlockLengths\(omKeyInfo\)}},
> converting each block on its own. EC parity is computed independently for
> each block group, so the two calculations can differ. For the same data
> partitioned across block groups, the per-block total is never smaller than
> the whole-key total, but they can be equal, including when block-group
> boundaries align with complete EC stripes.
> {{OmBucketInfo.decrUsedBytes}} does not clamp at zero, so deleting such keys
> can drive a bucket's {{usedBytes}} negative.
> h3. Example
> With {{rs\-3\-2\-1024}}, the data stripe is 3 \* 1024 bytes. A key holding
> two blocks of 1024 bytes each:
> * per key: {{getReplicatedSize\(2048\)}} = 2048 \+ 1024 \* 2 parity = 4096,
> which is what commit charges
> * per block: \(1024 \+ 1024 \* 2 parity\) \* 2 = 6144, which is what delete
> refunds
> h3. Which value describes the stored bytes
> The per\-block value does. EC encodes each block group independently, so each
> block carries its own partial\-stripe padding:
> * {{QuotaUtil.getReplicatedSize}} is documented as "Number of bytes required
> to store the dataSize with replication".
> * {{KeyManagerImpl.java:852\-859}}, which reports reclaimed space to SCM,
> computes both {{getReplicatedSize}} and {{getSizePerReplica}} per block, and
> passes {{OMKeyRequest.sumBlockLengths\(info\)}} as the quota released.
> * {{hadoop\-hdds/docs/content/feature/Quota.md}} describes the space quota as
> the storage space a bucket may use, counted after replication.
> RATIS replication is linear, so the two formulas agree there and only EC is
> affected.
> h3. Also affected: snapshotUsedBytes in quota repair
> {{QuotaRepairTask.recalculateDeletedKeyUsages}}
> \({{QuotaRepairTask.java:513}}\) totals the deleted\-key table with
> {{RepeatedOmKeyInfo.getTotalSize\(\)}}, which converts each key as a whole,
> while {{OMKeyDeleteRequest.java:165}} moves the per\-block
> {{sumBlockLengths\(omKeyInfo\)}} into {{snapshotUsedBytes}}. The same
> divergence therefore applies to that counter, and the formula chosen here
> should be applied there too. That total is wrong for a second, independent
> reason as well, tracked in HDDS\-16184.
> h3. Also affected: live keys and incomplete MPU in quota repair
> The live-key recount in {{QuotaRepairTask}} uses
> {{OmKeyInfo.getReplicatedSize()}}, converting each key's total data size at
> once. It should use the same block-based accounting as commit and delete.
> {{QuotaRepairTask.recalculateMultipartUsages}} recounts committed parts of
> incomplete uploads. Both the legacy embedded-parts path and the split
> {{multipartPartsTable}} path currently convert each part's total data size.
> Both paths should calculate usage per block group, consistently with MPU part
> commit, replacement, abort, expired abort, and completion cleanup.
> h3. Impact
> Space quota accounting for EC buckets is internally inconsistent. A bucket
> can report a lower {{usedBytes}} than the space its keys occupy, and repeated
> create/delete cycles can push {{usedBytes}} below zero.
> h3. Notes
> Relates to HDDS\-16127, which changes quota repair to recount keys retaining
> more than one version by converting each version once, matching the commit
> path. That patch deliberately does not adopt the per\-block conversion, so
> that repair reproduces what commit charged rather than pre\-empting the
> decision here.
> Pinned source commit 4766aa8609. Analysis assisted by AI tooling \(Claude
> Code, Opus 5\).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
