Siyao Meng created HDDS-16684:
---------------------------------
Summary: OM deadlock: deletion services hold snapshot DB read
locks across a synchronous Ratis submit
Key: HDDS-16684
URL: https://issues.apache.org/jira/browse/HDDS-16684
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
h3. Mechanism
A two edge cycle between a deletion service thread and the OM double buffer
flush thread.
Edge one: a deletion service thread holds {{SNAPSHOT_DB_LOCK}} stripe read
locks for the lifetime of each snapshot handle
({{ReclaimableFilter.java:163-165}}) and performs a synchronous Ratis submit
without releasing them ({{DirectoryDeletingService.java:721-771}} with
{{submitRequest}} at {{:604}}, and {{SnapshotDeletingService.java:172-214}}).
That submit waits for a double buffer permit, released only at the end of
{{flushBatch}} ({{OzoneManagerDoubleBuffer.java:449}}).
Edge two: the {{OMDoubleBufferFlushThread}}, while applying a queued
{{OMSnapshotPurgeResponse}} through
{{OmSnapshotManager.deleteSnapshotCheckpointDirectories}}
({{OmSnapshotManager.java:494-497}}), takes {{snapshotCache.lock(snapshotId)}},
the write side of the same stripe the service thread read holds.
Neither wait times out. {{KeyDeletingService.java:608-615}} releases its
handles before submitting and documents why, which shows the hazard is already
understood elsewhere in the same family of services.
h3. Impact
The single OM commit path parks, so no write transaction can be applied. Both
waits are untimed and only a restart recovers. The JVM's own deadlock detection
cannot see it: {{ThreadMXBean.findDeadlockedThreads()}} returns null because a
{{ReentrantReadWriteLock}} read hold has no owner.
h3. Reproduction
{{repro/test_bugMC-4_ratis_submit_deadlock.sh}} with
{{repro/java/SpeculaMc4RatisDeadlockTest.java}}. Evidence in
{{repro/logs/ev-mc4.log}}: after 60s the service submit has not returned, with
thread dumps showing the service thread parked in
{{OzoneManagerDoubleBuffer.acquireUnFlushedTransactions}} and the flush thread
parked in {{OzoneManagerLock$ResourceLocks.acquireLock}} under
{{SnapshotCache.lock}}.
h3. No patch attached
Fixing this means deciding where the snapshot handles are released. The shape
that fits the codebase is the one KeyDeletingService already uses at
KeyDeletingService.java:608-615, releasing the handles before the synchronous
submit, and ReclaimableFilter.closeSnapshotDbHandles() carries a javadoc saying
exactly that: the GC locks must stay held until the request is applied, but a
synchronous request must not retain snapshot DB read locks because the double
buffer may need the write lock. Applying that to DirectoryDeletingService and
SnapshotDeletingService is a change to when those services give up their
handles, and a regression test for the deadlock itself would be slow and timing
dependent in CI. Filing the analysis so the approach can be agreed first.
Found by TLA+ model checking and code review of the OM snapshot cache under
HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated with
Specula (Claude Opus 5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
