Siyao Meng created HDDS-16164:
---------------------------------
Summary: DirectoryDeletingService could deadlock snapshot purge
while retaining snapshot DB handles
Key: HDDS-16164
URL: https://issues.apache.org/jira/browse/HDDS-16164
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
h2. Problem
{{DirectoryDeletingService}} (DDS) can retain snapshot DB handles while it
submits synchronous OM requests. Each open handle holds a striped
{{SNAPSHOT_DB_LOCK}} read lock.
DDS has two relevant handle owners:
* The task thread holds the current-snapshot handle while it waits for parallel
directory workers and submits the final snapshot-property request.
* Each directory worker can hold previous-snapshot handles through
{{ReclaimableDirFilter}} and {{ReclaimableKeyFilter}} while it submits
{{PurgeDirectories}} requests.
Independently, {{OMDoubleBufferFlushThread}} can apply an earlier
snapshot-purge response that needs a write lock on the same stripe. The DDS and
purge snapshots can have different UUIDs that map to the same stripe.
If the double-buffer flush blocks on a DDS read lock and the unflushed
transaction count reaches {{ozone.om.unflushed.transaction.max.count}}, a
synchronous DDS request cannot be applied. DDS cannot return and close the
handle that blocks the flush, which completes the circular wait.
This risk was identified by code inspection. It was not the path observed in
the HDDS-16118 incident and has not yet been reproduced through DDS.
h2. Impact
The OM leader could stop applying transactions and flushing the double buffer.
Client operations could remain blocked until OM leadership changes or the
affected OM restarts.
h2. Proposed fix
Ensure that DDS closes all snapshot DB handles after it completes snapshot DB
reads and before it submits synchronous OM requests:
* Close the current-snapshot handle on its owner thread.
* Close previous-snapshot handles opened by the reclaimable filters on their
owner threads.
* Retain the snapshot GC locks required for snapshot-chain and reclamation
correctness until the corresponding OM requests complete.
* Preserve DDS parallelism where practical, or process snapshot cleanup on one
thread if this provides a simpler and safer ownership model.
Add a deterministic regression test that uses one snapshot DB lock stripe and
one allowed unflushed transaction to reproduce the complete DDS circular wait.
h2. References
* Follow-up to [HDDS-16118|https://issues.apache.org/jira/browse/HDDS-16118].
* Identified during review of Apache Ozone PR 10987: [review comment by Weichiu
Chuang|https://github.com/apache/ozone/pull/10987#pullrequestreview-4913044902].
* Related but distinct from
[HDDS-13219|https://issues.apache.org/jira/browse/HDDS-13219], which tracks
snapshot read locking during deep cleaning.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
