[
https://issues.apache.org/jira/browse/HDDS-16685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-16685:
----------------------------------
Labels: pull-request-available (was: )
> OM deadlock: nested snapshot get() takes SNAPSHOT_DB_LOCK stripes out of
> order against acquireAll
> -------------------------------------------------------------------------------------------------
>
> Key: HDDS-16685
> URL: https://issues.apache.org/jira/browse/HDDS-16685
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Chia-Chuan Ho
> Priority: Critical
> Labels: pull-request-available
>
> h3. Mechanism
> {{OzoneManagerLock.acquireAll}} ({{:158-164}}) takes every stripe of a
> resource in ascending index order, blocking, with no backoff and no all or
> nothing semantics. {{SnapshotCache.lock()}} reaches it through
> {{acquireResourceWriteLock}} ({{:382}}).
> {{SnapshotCache.get(UUID)}} instead takes exactly one stripe,
> {{acquireReadLock(SNAPSHOT_DB_LOCK, key.toString())}} at {{:217}}, chosen by
> the UUID hash, in whatever order the caller opens snapshots. A snapdiff
> holding the from snapshot on a high stripe and then requesting the to
> snapshot on a low stripe inverts against a concurrent {{acquireAll}} that has
> already taken the low stripe.
> {{MultiSnapshotLocks}} exists and uses the ordered bulk path, so the ordered
> primitive was written on purpose and the snapdiff callers bypass it. The DAG
> tracker added by HDDS-13960 orders resources, not stripes within a resource,
> so it does not cover this case.
> h3. Impact
> Permanent deadlock between the snapdiff thread and the checkpoint or follower
> bootstrap thread. Every later snapshot operation that needs one of the
> already taken stripes blocks behind it. Untimed on both sides, restart only,
> and invisible to JVM deadlock detection because read holds have no owner.
> h3. Reproduction
> {{repro/test_bugMC-5_stripe_inversion_deadlock.sh}} with
> {{repro/java/SpeculaMc5StripeInversionTest.java}}. Evidence in
> {{repro/logs/ev-mc5.log}}: the Level 0 sequential case completes as expected,
> then the two thread case leaves both threads unfinished after 60s, with
> {{findDeadlockedThreads()=null}} and thread dumps showing the snapdiff thread
> parked in {{SnapshotCache.get}} at {{:217}} and the checkpoint thread parked
> in {{acquireAll}} under {{SnapshotCache.lock}}.
> h3. No patch attached
> Fixing this means deciding how nested snapshot acquisition is ordered.
> MultiSnapshotLocks already wraps the ordered bulk path, so routing the
> snapdiff callers through it is the natural fix, but that changes how those
> callers hold snapshots across a diff and the alternative is to order the
> stripes inside SnapshotCache.get itself. A regression test for the deadlock
> would be slow and timing dependent in CI. Filing the analysis so the approach
> can be agreed first.
> Found by TLA+ model checking and code review of the OM snapshot cache under
> HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. Generated
> with Specula (Claude Opus 5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
