[
https://issues.apache.org/jira/browse/HDDS-17293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17293:
------------------------------
Attachment: TestBugMC3MoveTableKeysCheckpointDeadlock.java
> OM double buffer flush of SnapshotMoveTableKeys can deadlock with the DB
> checkpoint servlet on snapshot DB lock stripes, stopping all OM writes
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17293
> URL: https://issues.apache.org/jira/browse/HDDS-17293
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.2.0, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-3-lock-snapshot-db-stripes-in-order-for-move-table-keys.patch,
> TestBugMC3MoveTableKeysCheckpointDeadlock.java
>
>
> h3. Mechanism
> {{OMSnapshotMoveTableKeysResponse.addToDBBatch}} runs on the double buffer
> flush thread. It opens the from snapshot and then, still holding it, the next
> snapshot, both through {{OmSnapshotManager.getSnapshot}}. Each
> {{SnapshotCache.get}} takes a read lock on the one {{SNAPSHOT_DB_LOCK}}
> stripe its snapshot ID hashes to, so the two stripes are locked in caller
> order. {{SnapshotCache.lock()}}, which
> {{OMDBCheckpointServletInodeBasedXfer}} calls when the request includes
> snapshot data, write locks every stripe in ascending index order. When the
> from snapshot is on the higher stripe, the servlet takes the lower stripe and
> waits for the higher one, while the flush thread holds the higher one and
> waits for the lower one. Neither wait has a timeout.
> The flush thread takes no {{BOOTSTRAP_LOCK}}, and
> {{OMDBCheckpointServlet.Lock.acquireWriteLock}} calls
> {{awaitDoubleBufferFlush}} before it takes the bootstrap write lock, so a
> SnapshotMoveTableKeys batch applied in between can still be flushing when the
> servlet sweeps the stripes. SnapshotDeletingService holds the bootstrap read
> lock for its whole run, so only such a batch, applied before the servlet gets
> the bootstrap write lock, can take part. The response already takes its
> {{SNAPSHOT_DB_CONTENT_LOCK}} locks for both snapshots in one ordered
> {{acquireReadLocks}} call, but not the {{SNAPSHOT_DB_LOCK}} locks.
> This is the same stripe order inversion that HDDS-16685 describes for the
> snapshot diff callers, at a different call site. Here the blocked thread is
> the flush thread, so the impact is on every write.
> h3. Trigger
> # SnapshotDeletingService moves the keys of a deleted snapshot into the next
> active snapshot (SnapshotMoveTableKeys with a next snapshot).
> # A follower bootstraps from the leader at the same time. The default
> follower download uses {{/v2/dbCheckpoint}} with snapshot data, so the
> leader's servlet calls {{SnapshotCache.lock()}}.
> # The flush of that SnapshotMoveTableKeys batch has opened the from snapshot
> when the servlet's stripe sweep has passed the next snapshot's stripe but not
> the from snapshot's stripe.
> For a given pair the stripe order is fixed, so about half of all (from, next)
> pairs can hit it. Whether the timing lines up in a real bootstrap depends on
> how far the flush thread lags behind and on how long the servlet spends
> collecting snapshot files before it takes the cache lock. I did not run a
> real follower bootstrap.
> h3. Impact
> The flush thread never finishes the batch, so its unflushed transaction
> permits are not returned. Once {{ozone.om.unflushed.transaction.max.count}}
> transactions (default 10000) are applied,
> {{OzoneManagerStateMachine.applyTransaction}} blocks and no further OM write
> completes. The servlet keeps the bootstrap write lock, so the background
> services that take the bootstrap read lock stop too, and the follower
> bootstrap never finishes. The leader stays leader.
> {{ThreadMXBean.findDeadlockedThreads()}} does not report the cycle because
> read holds have no owner. Only a restart of the OM recovers.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC3MoveTableKeysCheckpointDeadlock.java]
> ({{hadoop-ozone/ozone-manager}}) uses the real lock manager, snapshot cache,
> snapshot manager and double buffer, with snapshots made by the real create
> request. A test thread does what the servlet does around the cache lock
> ({{awaitFlush}}, bootstrap write lock, {{SnapshotCache.lock()}}). The only
> timing help is a pause of the flush thread at the start of the nested
> {{getSnapshot(next)}} until the servlet thread is parked in its sweep. In
> each run the test picks a pair with the from snapshot on the higher stripe
> (for example 303 and 203), and the servlet stays parked in
> {{SnapshotCache.lock()}}, the flush never completes, and with a permit limit
> of 2 only 1 later apply is admitted while the next one blocks. A passing test
> means the defect is present.
> h3. Patch
> [^MC-3-lock-snapshot-db-stripes-in-order-for-move-table-keys.patch], against
> e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. It also applies to master at
> 1ff29ee6b86.
> {{addToDBBatch}} now takes the {{SNAPSHOT_DB_LOCK}} read locks of both
> snapshots with one {{acquireReadLocks}} call, which locks the stripes in
> ascending order, right after the content locks, and releases them in the same
> {{finally}}. The two {{getSnapshot}} calls then reacquire read locks the
> thread already holds, which {{ReentrantReadWriteLock}} grants even when a
> writer is queued. The flush thread and the servlet now lock stripes in the
> same order.
> Covered by {{testMoveTableKeysDoesNotDeadlockWithSnapshotCacheLock}} in the
> existing {{TestOMSnapshotMoveTableKeysResponse}}. It pauses the move before
> it opens the next snapshot, lets {{SnapshotCache.lock()}} start and wait,
> then lets the move continue, for both orders of two snapshots on different
> stripes. Without the change the move does not complete within 30s in one of
> the two orders. With it the snapshot request and response suites,
> {{TestSnapshotCache}}, {{TestOmSnapshotManager}}, {{TestOzoneManagerLock}},
> {{TestMultiSnapshotLocks}}, {{TestOzoneManagerDoubleBuffer}} and
> {{TestSnapshotDeletingService}} (232 tests) pass and checkstyle is clean.
> With the patch the reproduction fails because the servlet thread finishes.
> {{OMSnapshotMoveDeletedKeysResponse}} has the same nesting. Nothing at this
> commit submits that request (SnapshotDeletingService sends
> SnapshotMoveTableKeys, and the request is rejected from clients), so it is
> only applied when replaying log entries written by older OMs. The patch
> leaves it unchanged.
> Found by TLA+ model checking and code review of the Ozone Manager lock
> ordering paths under HDDS-15926, on commit
> e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
