Siyao Meng created HDDS-17293:
---------------------------------
Summary: OM double buffer flush of SnapshotMoveTableKeys can
deadlock with the DB checkpoint servlet on snapshot DB lock stripes, stopping
all OM writes
Key: HDDS-17293
URL: https://issues.apache.org/jira/browse/HDDS-17293
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-3-lock-snapshot-db-stripes-in-order-for-move-table-keys.patch,
TestBugMC3MoveTableKeysCheckpointDeadlock.java
h3. Mechanism
{{OMSnapshotMoveTableKeysResponse.addToDBBatch}} runs on the double buffer
flush thread. It opens the from snapshot and then, still holding it, the next
snapshot, both through {{OmSnapshotManager.getSnapshot}}. Each
{{SnapshotCache.get}} takes a read lock on the one {{SNAPSHOT_DB_LOCK}} stripe
its snapshot ID hashes to, so the two stripes are locked in caller order.
{{SnapshotCache.lock()}}, which {{OMDBCheckpointServletInodeBasedXfer}} calls
when the request includes snapshot data, write locks every stripe in ascending
index order. When the from snapshot is on the higher stripe, the servlet takes
the lower stripe and waits for the higher one, while the flush thread holds the
higher one and waits for the lower one. Neither wait has a timeout.
The flush thread takes no {{BOOTSTRAP_LOCK}}, and
{{OMDBCheckpointServlet.Lock.acquireWriteLock}} calls
{{awaitDoubleBufferFlush}} before it takes the bootstrap write lock, so a
SnapshotMoveTableKeys batch applied in between can still be flushing when the
servlet sweeps the stripes. SnapshotDeletingService holds the bootstrap read
lock for its whole run, so only such a batch, applied before the servlet gets
the bootstrap write lock, can take part. The response already takes its
{{SNAPSHOT_DB_CONTENT_LOCK}} locks for both snapshots in one ordered
{{acquireReadLocks}} call, but not the {{SNAPSHOT_DB_LOCK}} locks.
This is the same stripe order inversion that HDDS-16685 describes for the
snapshot diff callers, at a different call site. Here the blocked thread is the
flush thread, so the impact is on every write.
h3. Trigger
# SnapshotDeletingService moves the keys of a deleted snapshot into the next
active snapshot (SnapshotMoveTableKeys with a next snapshot).
# A follower bootstraps from the leader at the same time. The default follower
download uses {{/v2/dbCheckpoint}} with snapshot data, so the leader's servlet
calls {{SnapshotCache.lock()}}.
# The flush of that SnapshotMoveTableKeys batch has opened the from snapshot
when the servlet's stripe sweep has passed the next snapshot's stripe but not
the from snapshot's stripe.
For a given pair the stripe order is fixed, so about half of all (from, next)
pairs can hit it. Whether the timing lines up in a real bootstrap depends on
how far the flush thread lags behind and on how long the servlet spends
collecting snapshot files before it takes the cache lock. I did not run a real
follower bootstrap.
h3. Impact
The flush thread never finishes the batch, so its unflushed transaction permits
are not returned. Once {{ozone.om.unflushed.transaction.max.count}}
transactions (default 10000) are applied,
{{OzoneManagerStateMachine.applyTransaction}} blocks and no further OM write
completes. The servlet keeps the bootstrap write lock, so the background
services that take the bootstrap read lock stop too, and the follower bootstrap
never finishes. The leader stays leader.
{{ThreadMXBean.findDeadlockedThreads()}} does not report the cycle because read
holds have no owner. Only a restart of the OM recovers.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC3MoveTableKeysCheckpointDeadlock.java]
({{hadoop-ozone/ozone-manager}}) uses the real lock manager, snapshot cache,
snapshot manager and double buffer, with snapshots made by the real create
request. A test thread does what the servlet does around the cache lock
({{awaitFlush}}, bootstrap write lock, {{SnapshotCache.lock()}}). The only
timing help is a pause of the flush thread at the start of the nested
{{getSnapshot(next)}} until the servlet thread is parked in its sweep. In each
run the test picks a pair with the from snapshot on the higher stripe (for
example 303 and 203), and the servlet stays parked in {{SnapshotCache.lock()}},
the flush never completes, and with a permit limit of 2 only 1 later apply is
admitted while the next one blocks. A passing test means the defect is present.
h3. Patch
[^MC-3-lock-snapshot-db-stripes-in-order-for-move-table-keys.patch], against
e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. It also applies to master at
1ff29ee6b86.
{{addToDBBatch}} now takes the {{SNAPSHOT_DB_LOCK}} read locks of both
snapshots with one {{acquireReadLocks}} call, which locks the stripes in
ascending order, right after the content locks, and releases them in the same
{{finally}}. The two {{getSnapshot}} calls then reacquire read locks the thread
already holds, which {{ReentrantReadWriteLock}} grants even when a writer is
queued. The flush thread and the servlet now lock stripes in the same order.
Covered by {{testMoveTableKeysDoesNotDeadlockWithSnapshotCacheLock}} in the
existing {{TestOMSnapshotMoveTableKeysResponse}}. It pauses the move before it
opens the next snapshot, lets {{SnapshotCache.lock()}} start and wait, then
lets the move continue, for both orders of two snapshots on different stripes.
Without the change the move does not complete within 30s in one of the two
orders. With it the snapshot request and response suites,
{{TestSnapshotCache}}, {{TestOmSnapshotManager}}, {{TestOzoneManagerLock}},
{{TestMultiSnapshotLocks}}, {{TestOzoneManagerDoubleBuffer}} and
{{TestSnapshotDeletingService}} (232 tests) pass and checkstyle is clean. With
the patch the reproduction fails because the servlet thread finishes.
{{OMSnapshotMoveDeletedKeysResponse}} has the same nesting. Nothing at this
commit submits that request (SnapshotDeletingService sends
SnapshotMoveTableKeys, and the request is rejected from clients), so it is only
applied when replaying log entries written by older OMs. The patch leaves it
unchanged.
Found by TLA+ model checking and code review of the Ozone Manager lock ordering
paths under HDDS-15926, on commit e0de543cbe9d73f3bf5473a45e1c59cd6eae7173.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
