[
https://issues.apache.org/jira/browse/HDDS-16176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chi-Hsuan Huang updated HDDS-16176:
-----------------------------------
Description:
h3. Problem
For an EC key whose data spans more than one block, OM charges quota using a
different formula than it refunds with. Commit charges
{{QuotaUtil.getReplicatedSize\(dataSize, repConfig\)}}, converting the whole
key at once. Delete refunds {{OMKeyRequest.sumBlockLengths\(omKeyInfo\)}},
converting each block on its own. EC parity is computed independently for each
block group, so the two calculations can differ. For the same data partitioned
across block groups, the per-block total is never smaller than the whole-key
total, but they can be equal, including when block-group boundaries align with
complete EC stripes.
{{OmBucketInfo.decrUsedBytes}} does not clamp at zero, so deleting such keys
can drive a bucket's {{usedBytes}} negative.
h3. Example
With {{rs\-3\-2\-1024}}, the data stripe is 3 \* 1024 bytes. A key holding two
blocks of 1024 bytes each:
* per key: {{getReplicatedSize\(2048\)}} = 2048 \+ 1024 \* 2 parity = 4096,
which is what commit charges
* per block: \(1024 \+ 1024 \* 2 parity\) \* 2 = 6144, which is what delete
refunds
h3. Which value describes the stored bytes
The per\-block value does. EC encodes each block group independently, so each
block carries its own partial\-stripe padding:
* {{QuotaUtil.getReplicatedSize}} is documented as "Number of bytes required to
store the dataSize with replication".
* {{KeyManagerImpl.java:852\-859}}, which reports reclaimed space to SCM,
computes both {{getReplicatedSize}} and {{getSizePerReplica}} per block, and
passes {{OMKeyRequest.sumBlockLengths\(info\)}} as the quota released.
* {{hadoop\-hdds/docs/content/feature/Quota.md}} describes the space quota as
the storage space a bucket may use, counted after replication.
RATIS replication is linear, so the two formulas agree there and only EC is
affected.
h3. Also affected: snapshotUsedBytes in quota repair
{{QuotaRepairTask.recalculateDeletedKeyUsages}}
\({{QuotaRepairTask.java:513}}\) totals the deleted\-key table with
{{RepeatedOmKeyInfo.getTotalSize\(\)}}, which converts each key as a whole,
while {{OMKeyDeleteRequest.java:165}} moves the per\-block
{{sumBlockLengths\(omKeyInfo\)}} into {{snapshotUsedBytes}}. The same
divergence therefore applies to that counter, and the formula chosen here
should be applied there too. That total is wrong for a second, independent
reason as well, tracked in HDDS\-16184.
h3. Also affected: live keys and incomplete MPU in quota repair
The live-key recount in {{QuotaRepairTask}} uses
{{OmKeyInfo.getReplicatedSize()}}, converting each key's total data size at
once. It should use the same block-based accounting as commit and delete.
{{QuotaRepairTask.recalculateMultipartUsages}} recounts committed parts of
incomplete uploads. Both the legacy embedded-parts path and the split
{{multipartPartsTable}} path currently convert each part's total data size.
Both paths should calculate usage per block group, consistently with MPU part
commit, replacement, abort, expired abort, and completion cleanup.
h3. Impact
Space quota accounting for EC buckets is internally inconsistent. A bucket can
report a lower {{usedBytes}} than the space its keys occupy, and repeated
create/delete cycles can push {{usedBytes}} below zero.
h3. Notes
Relates to HDDS\-16127, which changes quota repair to recount keys retaining
more than one version by converting each version once, matching the commit
path. That patch deliberately does not adopt the per\-block conversion, so that
repair reproduces what commit charged rather than pre\-empting the decision
here.
Pinned source commit 4766aa8609. Analysis assisted by AI tooling \(Claude Code,
Opus 5\).
was:
h3. Problem
For an EC key whose data spans more than one block, OM charges quota using a
different formula than it refunds with. Commit charges
{{QuotaUtil.getReplicatedSize\(dataSize, repConfig\)}}, converting the whole
key at once. Delete refunds {{OMKeyRequest.sumBlockLengths\(omKeyInfo\)}},
converting each block on its own. EC parity is added per block, so the two do
not agree, and the refund is always the larger of the two.
{{OmBucketInfo.decrUsedBytes}} does not clamp at zero, so deleting such keys
can drive a bucket's {{usedBytes}} negative.
h3. Example
With {{rs\-3\-2\-1024}}, the data stripe is 3 \* 1024 bytes. A key holding two
blocks of 1024 bytes each:
* per key: {{getReplicatedSize\(2048\)}} = 2048 \+ 1024 \* 2 parity = 4096,
which is what commit charges
* per block: \(1024 \+ 1024 \* 2 parity\) \* 2 = 6144, which is what delete
refunds
h3. Which value describes the stored bytes
The per\-block value does. EC encodes each block group independently, so each
block carries its own partial\-stripe padding:
* {{QuotaUtil.getReplicatedSize}} is documented as "Number of bytes required to
store the dataSize with replication".
* {{KeyManagerImpl.java:852\-859}}, which reports reclaimed space to SCM,
computes both {{getReplicatedSize}} and {{getSizePerReplica}} per block, and
passes {{OMKeyRequest.sumBlockLengths\(info\)}} as the quota released.
* {{hadoop\-hdds/docs/content/feature/Quota.md}} describes the space quota as
the storage space a bucket may use, counted after replication.
RATIS replication is linear, so the two formulas agree there and only EC is
affected.
h3. Also affected: snapshotUsedBytes in quota repair
{{QuotaRepairTask.recalculateDeletedKeyUsages}}
\({{QuotaRepairTask.java:513}}\) totals the deleted\-key table with
{{RepeatedOmKeyInfo.getTotalSize\(\)}}, which converts each key as a whole,
while {{OMKeyDeleteRequest.java:165}} moves the per\-block
{{sumBlockLengths\(omKeyInfo\)}} into {{snapshotUsedBytes}}. The same
divergence therefore applies to that counter, and the formula chosen here
should be applied there too. That total is wrong for a second, independent
reason as well, tracked in HDDS\-16184.
h3. Impact
Space quota accounting for EC buckets is internally inconsistent. A bucket can
report a lower {{usedBytes}} than the space its keys occupy, and repeated
create/delete cycles can push {{usedBytes}} below zero.
h3. Notes
Relates to HDDS\-16127, which changes quota repair to recount keys retaining
more than one version by converting each version once, matching the commit
path. That patch deliberately does not adopt the per\-block conversion, so that
repair reproduces what commit charged rather than pre\-empting the decision
here.
Pinned source commit 4766aa8609. Analysis assisted by AI tooling \(Claude Code,
Opus 5\).
> EC replicated size is computed per key on commit but per block on delete
> ------------------------------------------------------------------------
>
> Key: HDDS-16176
> URL: https://issues.apache.org/jira/browse/HDDS-16176
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: Chi-Hsuan Huang
> Priority: Major
>
> h3. Problem
> For an EC key whose data spans more than one block, OM charges quota using a
> different formula than it refunds with. Commit charges
> {{QuotaUtil.getReplicatedSize\(dataSize, repConfig\)}}, converting the whole
> key at once. Delete refunds {{OMKeyRequest.sumBlockLengths\(omKeyInfo\)}},
> converting each block on its own. EC parity is computed independently for
> each block group, so the two calculations can differ. For the same data
> partitioned across block groups, the per-block total is never smaller than
> the whole-key total, but they can be equal, including when block-group
> boundaries align with complete EC stripes.
> {{OmBucketInfo.decrUsedBytes}} does not clamp at zero, so deleting such keys
> can drive a bucket's {{usedBytes}} negative.
> h3. Example
> With {{rs\-3\-2\-1024}}, the data stripe is 3 \* 1024 bytes. A key holding
> two blocks of 1024 bytes each:
> * per key: {{getReplicatedSize\(2048\)}} = 2048 \+ 1024 \* 2 parity = 4096,
> which is what commit charges
> * per block: \(1024 \+ 1024 \* 2 parity\) \* 2 = 6144, which is what delete
> refunds
> h3. Which value describes the stored bytes
> The per\-block value does. EC encodes each block group independently, so each
> block carries its own partial\-stripe padding:
> * {{QuotaUtil.getReplicatedSize}} is documented as "Number of bytes required
> to store the dataSize with replication".
> * {{KeyManagerImpl.java:852\-859}}, which reports reclaimed space to SCM,
> computes both {{getReplicatedSize}} and {{getSizePerReplica}} per block, and
> passes {{OMKeyRequest.sumBlockLengths\(info\)}} as the quota released.
> * {{hadoop\-hdds/docs/content/feature/Quota.md}} describes the space quota as
> the storage space a bucket may use, counted after replication.
> RATIS replication is linear, so the two formulas agree there and only EC is
> affected.
> h3. Also affected: snapshotUsedBytes in quota repair
> {{QuotaRepairTask.recalculateDeletedKeyUsages}}
> \({{QuotaRepairTask.java:513}}\) totals the deleted\-key table with
> {{RepeatedOmKeyInfo.getTotalSize\(\)}}, which converts each key as a whole,
> while {{OMKeyDeleteRequest.java:165}} moves the per\-block
> {{sumBlockLengths\(omKeyInfo\)}} into {{snapshotUsedBytes}}. The same
> divergence therefore applies to that counter, and the formula chosen here
> should be applied there too. That total is wrong for a second, independent
> reason as well, tracked in HDDS\-16184.
> h3. Also affected: live keys and incomplete MPU in quota repair
> The live-key recount in {{QuotaRepairTask}} uses
> {{OmKeyInfo.getReplicatedSize()}}, converting each key's total data size at
> once. It should use the same block-based accounting as commit and delete.
> {{QuotaRepairTask.recalculateMultipartUsages}} recounts committed parts of
> incomplete uploads. Both the legacy embedded-parts path and the split
> {{multipartPartsTable}} path currently convert each part's total data size.
> Both paths should calculate usage per block group, consistently with MPU part
> commit, replacement, abort, expired abort, and completion cleanup.
> h3. Impact
> Space quota accounting for EC buckets is internally inconsistent. A bucket
> can report a lower {{usedBytes}} than the space its keys occupy, and repeated
> create/delete cycles can push {{usedBytes}} below zero.
> h3. Notes
> Relates to HDDS\-16127, which changes quota repair to recount keys retaining
> more than one version by converting each version once, matching the commit
> path. That patch deliberately does not adopt the per\-block conversion, so
> that repair reproduces what commit charged rather than pre\-empting the
> decision here.
> Pinned source commit 4766aa8609. Analysis assisted by AI tooling \(Claude
> Code, Opus 5\).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
