[
https://issues.apache.org/jira/browse/HDDS-17158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Priyesh Karatha reassigned HDDS-17158:
--------------------------------------
Assignee: Priyesh Karatha
> Datanode under-reports pending deletion bytes for EC blocks
> -----------------------------------------------------------
>
> Key: HDDS-17158
> URL: https://issues.apache.org/jira/browse/HDDS-17158
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Datanode
> Reporter: Priyesh Karatha
> Assignee: Priyesh Karatha
> Priority: Major
>
> h3. Problem
> For EC keys, the DN metric \{{TotalPendingBlockBytes}} (BlockDeletingService)
> under-reports pending deletion size. As a result, the DN value Recon shows on
> the storage distribution / pending deletion page
> (\{{pendingDeletion?component=dn}}) does not match SCM. The mismatch only
> shows up some of the time.
> Observed in a test that deleted 3 EC keys (expected 171,966,464 bytes):
> * SCM deletion metrics per cycle: [113246208, 58720256] -> matches expected
> total
> * DN pending deletion per cycle: [113246205, 5872025] -> second cycle is ~10x
> too low (~52.8 MB missing)
> The first cycle is correct; the -3 bytes come from integer division of the
> per-replica size across 5 DNs.
> h3. Root cause
> HDDS-15076 changed the DN to add \{{totalSizePerReplica}} to a container's
> pending deletion bytes when a delete transaction arrives
> (\{{DeleteBlocksCommandHandler#updateMetaData}}). The release path was not
> updated: when the transaction is processed,
> \{{BlockDeletingTask#deleteTransactions}} still subtracts \{{totalBlockSize}}:
> {code:java}
> bytesProcessed += entry.getTotalBlockSize();
> {code}
> For RATIS, \{{totalBlockSize == totalSizePerReplica}}, so the bug does not
> show. For EC, \{{totalBlockSize}} is the logical block size, about \{{data}}
> times the per-replica size (e.g. ~3x for rs-3-2). So each processed EC
> transaction releases far more than it added.
> \{{ContainerData.BlockByteAndCounts#decDeletion}} then clamps the result at 0.
> h3. Why it is intermittent
> If all pending transactions of a container are processed in the same run, the
> clamp to 0 hides the error. The under-count shows only when a newer
> transaction is already pending in the same container while an older one is
> processed in a separate run. That can happen when:
> * the block deletion limit per interval is reached
> * the max lock holding time cutoff is hit
> * the next deletion cycle's transaction arrives in between
> The extra amount subtracted then removes the newer transaction's pending
> bytes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
