priyeshkaratha opened a new pull request, #8755:
URL: https://github.com/apache/ozone/pull/8755
## What changes were proposed in this pull request?
### Accurate accounting of pending-delete bytes per container
- Added `incrPendingDeletions(long)` and `decrPendingDeletions(long)` to
`ContainerData` with a new `pendingDeletionBytes` field that is persisted in
container metadata.
- Integrated the counter with `DeleteBlocksCommandHandler`:
- Introduced helper method `incrPendingDeletionBytes()` which sums the
sizes of blocks in each `DeletedBlocksTransaction` and updates the container’s
counter when blocks are logically marked for deletion.
- When blocks are physically removed in `deleteViaTransactionStore()`, the
counter is decremented accordingly.
### Datanode-level aggregation & reporting
- `StorageReport` now includes an aggregated `pendingDeletionBytes` value
(sum across all containers).
- Introduced a short-lived in-memory cache (default TTL: 10 minutes) for the
aggregated value to avoid recomputing on every heartbeat, reducing CPU overhead.
### Metrics & API exposure
- Exposed `pendingDeletionBytes` as a Datanode metric.
- Extended:
- Datanode-to-SCM heartbeat protobuf to include the new metric.
- Recon REST API (`/datanode`) to surface the pending deletion bytes.
### Upgrade path for existing clusters
- For containers created before the upgrade:
- During open, if `pendingDeletionBytes` is missing, it is computed using
`KeyValueContainerUtil.populateContainerMetadata()` by scanning the block table.
- A background task reprocesses existing delete transactions and block
metadata to compute accurate pending delete sizes.
- Mixed-version compatibility:
- Older DNs/SCMs gracefully ignore unknown fields.
- Recon detects support and falls back to legacy logic if needed.
## What is the link to the Apache JIRA
[HDDS-13192](https://issues.apache.org/jira/browse/HDDS-13192)
## How was this patch tested?
### Unit Tests
- Verified counter updates via unit tests in `ContainerData`.
- Added unit tests for `DeleteBlocksCommandHandler` flows that mark and
delete blocks.
### Integration Testing
- Pending
### Upgrade & Backward Compatibility
- No impacts
### Performance Testing
- Pending
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
