Siyao Meng created HDDS-16155:
---------------------------------
Summary: Fix OM checkpoint installation deadlock between state
machine pause and double-buffer flush
Key: HDDS-16155
URL: https://issues.apache.org/jira/browse/HDDS-16155
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
h3. Problem
An OM follower can remain indefinitely in checkpoint installation. The leader
continues sending snapshot notifications, but the follower remains frozen and
repeatedly reports that installation is still in progress.
h3. Incident evidence
At the onset of the stall:
* {{jvm_metrics_threads_blocked}} changed from 0 to 1.
* Ratis {{appliedIndex}} reached 264576 while {{applyCompletedIndex}} stopped
at 264524.
* Both indexes then remained unchanged for approximately 5.5 hours.
The follower continued logging:
{code}
Failed appendEntries as snapshot (264587) installation is in progress
appendEntries reply ... INCONSISTENCY,nextIndex=264578,followerCommit=264576
{code}
The leader had advanced to index 1199609 and repeatedly notified the stalled
follower:
{code}
receive installSnapshot: ... notify:(t:2, i:1199609)
reply installSnapshot: ... IN_PROGRESS,snapshotIndex=0
{code}
After the follower was restarted, the blocked-thread count returned to 0 and
its double-buffer flush counter resumed, advancing from 0 to 510.
No JVM thread dump was captured during the incident, so the incident artifacts
cannot directly identify the blocked threads.
h3. Findings and deadlock theory
{{OzoneManagerStateMachine.pause()}} is synchronized on the state-machine
instance. During checkpoint installation, the install-snapshot thread enters
{{pause()}} and stops the OM double buffer:
{code}
InstallSnapshotThread
holds the OzoneManagerStateMachine monitor in synchronized pause()
-> OzoneManagerDoubleBuffer.stop()
-> stopDaemon()
-> daemon.join()
-> waits for OMDoubleBufferFlushThread
{code}
After committing a batch, {{OMDoubleBufferFlushThread}} invokes the
state-machine callback that updates the last applied term/index.
{{updateLastAppliedTermIndex()}} is synchronized on the same state-machine
instance:
{code}
OMDoubleBufferFlushThread
-> updateLastAppliedIndex callback
-> synchronized updateLastAppliedTermIndex()
-> waits for the OzoneManagerStateMachine monitor
{code}
If checkpoint installation calls {{pause()}} after the flush thread commits its
batch but before the callback acquires the monitor, the threads form a cycle:
* The install-snapshot thread holds the state-machine monitor and waits for the
flush thread to exit.
* The flush thread cannot exit because it is waiting for that monitor.
This explains the single blocked thread, the incomplete Ratis apply, and the
follower remaining indefinitely in snapshot installation.
A deterministic regression using a real {{OzoneManagerDoubleBuffer}} and
metadata DB reproduces this cycle: it times out on unpatched master and passes
when lifecycle and term-index synchronization are separated.
h3. Proposed fix
Use separate synchronization domains for lifecycle state and term/index state:
* Keep {{pause()}}, {{unpause()}}, and {{reinitialize()}} synchronized for
lifecycle serialization.
* Protect compound notified, skipped, and applied term/index operations with a
dedicated {{termIndexLock}}.
* Use {{termIndexLock}} in {{notifyTermIndexUpdated()}},
{{updateLastAppliedTermIndex()}}, and snapshot term/index selection.
* Restore the persisted last-applied term/index under the same lock during
checkpoint reload.
The double-buffer flush callback then requires only {{termIndexLock}}, not the
lifecycle monitor held by {{pause()}}. This breaks the cross-thread cycle while
preserving lifecycle serialization and atomic term/index updates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
