[
https://issues.apache.org/jira/browse/HDDS-16684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chia-Chuan Ho reassigned HDDS-16684:
------------------------------------
Assignee: Chia-Chuan Ho
> OM deadlock: deletion services hold snapshot DB read locks across a
> synchronous Ratis submit
> --------------------------------------------------------------------------------------------
>
> Key: HDDS-16684
> URL: https://issues.apache.org/jira/browse/HDDS-16684
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Chia-Chuan Ho
> Priority: Critical
>
> h3. Mechanism
> A two edge cycle between a deletion service thread and the OM double buffer
> flush thread.
> Edge one: a deletion service thread holds {{SNAPSHOT_DB_LOCK}} stripe read
> locks for the lifetime of each snapshot handle
> ({{ReclaimableFilter.java:163-165}}) and performs a synchronous Ratis submit
> without releasing them ({{DirectoryDeletingService.java:721-771}} with
> {{submitRequest}} at {{:604}}, and {{SnapshotDeletingService.java:172-214}}).
> That submit waits for a double buffer permit, released only at the end of
> {{flushBatch}} ({{OzoneManagerDoubleBuffer.java:449}}).
> Edge two: the {{OMDoubleBufferFlushThread}}, while applying a queued
> {{OMSnapshotPurgeResponse}} through
> {{OmSnapshotManager.deleteSnapshotCheckpointDirectories}}
> ({{OmSnapshotManager.java:494-497}}), takes
> {{snapshotCache.lock(snapshotId)}}, the write side of the same stripe the
> service thread read holds.
> Neither wait times out. {{KeyDeletingService.java:608-615}} releases its
> handles before submitting and documents why, which shows the hazard is
> already understood elsewhere in the same family of services.
> h3. Impact
> The single OM commit path parks, so no write transaction can be applied. Both
> waits are untimed and only a restart recovers. The JVM's own deadlock
> detection cannot see it: {{ThreadMXBean.findDeadlockedThreads()}} returns
> null because a {{ReentrantReadWriteLock}} read hold has no owner.
> h3. Reproduction
> {{repro/test_bugMC-4_ratis_submit_deadlock.sh}} with
> {{repro/java/SpeculaMc4RatisDeadlockTest.java}}. Evidence in
> {{repro/logs/ev-mc4.log}}: after 60s the service submit has not returned,
> with thread dumps showing the service thread parked in
> {{OzoneManagerDoubleBuffer.acquireUnFlushedTransactions}} and the flush
> thread parked in {{OzoneManagerLock$ResourceLocks.acquireLock}} under
> {{SnapshotCache.lock}}.
> h3. No patch attached
> Fixing this means deciding where the snapshot handles are released. The shape
> that fits the codebase is the one KeyDeletingService already uses at
> KeyDeletingService.java:608-615, releasing the handles before the synchronous
> submit, and ReclaimableFilter.closeSnapshotDbHandles() carries a javadoc
> saying exactly that: the GC locks must stay held until the request is
> applied, but a synchronous request must not retain snapshot DB read locks
> because the double buffer may need the write lock. Applying that to
> DirectoryDeletingService and SnapshotDeletingService is a change to when
> those services give up their handles, and a regression test for the deadlock
> itself would be slow and timing dependent in CI. Filing the analysis so the
> approach can be agreed first.
> Found by TLA+ model checking and code review of the OM snapshot cache under
> HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated
> with Specula (Claude Opus 5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
