[
https://issues.apache.org/jira/browse/HDDS-16155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16155:
------------------------------
Component/s: OM
> Fix OM checkpoint installation deadlock between state machine pause and
> double-buffer flush
> -------------------------------------------------------------------------------------------
>
> Key: HDDS-16155
> URL: https://issues.apache.org/jira/browse/HDDS-16155
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: Siyao Meng
> Priority: Major
>
> h3. Problem
> An OM follower can remain indefinitely in checkpoint installation. The leader
> continues sending snapshot notifications, but the follower remains frozen and
> repeatedly reports that installation is still in progress.
> h3. Incident evidence
> At the onset of the stall:
> * {{jvm_metrics_threads_blocked}} changed from 0 to 1.
> * Ratis {{appliedIndex}} reached 264576 while {{applyCompletedIndex}} stopped
> at 264524.
> * Both indexes then remained unchanged for approximately 5.5 hours.
> The follower continued logging:
> {code}
> Failed appendEntries as snapshot (264587) installation is in progress
> appendEntries reply ... INCONSISTENCY,nextIndex=264578,followerCommit=264576
> {code}
> The leader had advanced to index 1199609 and repeatedly notified the stalled
> follower:
> {code}
> receive installSnapshot: ... notify:(t:2, i:1199609)
> reply installSnapshot: ... IN_PROGRESS,snapshotIndex=0
> {code}
> After the follower was restarted, the blocked-thread count returned to 0 and
> its double-buffer flush counter resumed, advancing from 0 to 510.
> No JVM thread dump was captured during the incident, so the incident
> artifacts cannot directly identify the blocked threads.
> h3. Findings and deadlock theory
> {{OzoneManagerStateMachine.pause()}} is synchronized on the state-machine
> instance. During checkpoint installation, the install-snapshot thread enters
> {{pause()}} and stops the OM double buffer:
> {code}
> InstallSnapshotThread
> holds the OzoneManagerStateMachine monitor in synchronized pause()
> -> OzoneManagerDoubleBuffer.stop()
> -> stopDaemon()
> -> daemon.join()
> -> waits for OMDoubleBufferFlushThread
> {code}
> After committing a batch, {{OMDoubleBufferFlushThread}} invokes the
> state-machine callback that updates the last applied term/index.
> {{updateLastAppliedTermIndex()}} is synchronized on the same state-machine
> instance:
> {code}
> OMDoubleBufferFlushThread
> -> updateLastAppliedIndex callback
> -> synchronized updateLastAppliedTermIndex()
> -> waits for the OzoneManagerStateMachine monitor
> {code}
> If checkpoint installation calls {{pause()}} after the flush thread commits
> its batch but before the callback acquires the monitor, the threads form a
> cycle:
> * The install-snapshot thread holds the state-machine monitor and waits for
> the flush thread to exit.
> * The flush thread cannot exit because it is waiting for that monitor.
> This explains the single blocked thread, the incomplete Ratis apply, and the
> follower remaining indefinitely in snapshot installation.
> A deterministic regression using a real {{OzoneManagerDoubleBuffer}} and
> metadata DB reproduces this cycle: it times out on unpatched master and
> passes when lifecycle and term-index synchronization are separated.
> h3. Proposed fix
> Use separate synchronization domains for lifecycle state and term/index state:
> * Keep {{pause()}}, {{unpause()}}, and {{reinitialize()}} synchronized for
> lifecycle serialization.
> * Protect compound notified, skipped, and applied term/index operations with
> a dedicated {{termIndexLock}}.
> * Use {{termIndexLock}} in {{notifyTermIndexUpdated()}},
> {{updateLastAppliedTermIndex()}}, and snapshot term/index selection.
> * Restore the persisted last-applied term/index under the same lock during
> checkpoint reload.
> The double-buffer flush callback then requires only {{termIndexLock}}, not
> the lifecycle monitor held by {{pause()}}. This breaks the cross-thread cycle
> while preserving lifecycle serialization and atomic term/index updates.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
