yandrey321 opened a new pull request, #11118:
URL: https://github.com/apache/ozone/pull/11118
## What changes were proposed in this pull request?
`DirectoryDeletingService` (DDS) drains the FSO deleted-directory backlog by
submitting `PurgeDirectories` transactions to the OM. Each transaction is
applied on the single OM state-machine thread while holding the bucket write
lock. Two aspects of the old path let a background purge stall interactive
writes and reads:
1. **Wide lock hold time.**
`OMDirectoriesPurgeRequestWithFSO.validateAndUpdateCache`
did all of its work — protobuf field extraction, RocksDB key-string
construction, replicated block-size summation, hsync client-id lookup —
while
holding the bucket write lock.
2. **Cross-bucket lock stacking.** A single purge transaction could carry
paths
from multiple buckets, so applying it acquired multiple bucket write
locks,
blocking read RPCs (`lookupKey`, `getFileStatus`, `listStatus`) on buckets
unrelated to the deletion.
This PR addresses both:
**Two-phase apply in `OMDirectoriesPurgeRequestWithFSO`**
- *Phase 1 (no lock):* iterate every `PurgePathRequest` and precompute each
entry's delete key, path key, replicated byte size, hsync client id, and
prepared sub-dir/sub-file tombstones into a `PurgeApplyState`.
- *Phase 2 (bucket write lock):* re-validate the bucket object id (snapshot
chain unchanged), then apply the precomputed cache tombstones, quota
deltas,
and hsync open-key cleanup. Snapshot-namespace `copyObject` work is moved
out
of the locked region.
**Per-bucket grouping in `DirectoryDeletingService`**
- `submitPurgePathsWithBatching` groups `PurgePathRequest`s by
`(volumeId, bucketId)` before submission, so each submitted transaction
carries paths from exactly one bucket and the apply side takes exactly one
bucket write lock. Byte-limit batching is preserved within each bucket
group.
Wire format is unchanged — no protobuf, RPC, or RocksDB-schema changes.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16220
## How was this patch tested?
-
`TestOMDirectoriesPurgeRequestAndResponse#testLightParsePreservesReplicatedByteSumAndHsyncForSubFiles`
— proves the lock-free phase-1 light parse yields the exact same replicated
byte sum and hsync open-key handling as the original full `OmKeyInfo` parse
(bucket `usedBytes` decrements to precisely zero).
-
`TestDirectoryDeletingService#testPurgeDirectoriesGroupedByBucketPerTransaction`
— interleaves paths from two buckets and asserts one bucket per submitted
`PurgeDirectories` transaction, order-independently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
