Siyao Meng created HDDS-17576:
---------------------------------
Summary: Follower SCM stops applying the Raft log until restart
after an install snapshot notification from a leader missing from its peer list
Key: HDDS-17576
URL: https://issues.apache.org/jira/browse/HDDS-17576
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-3-fail-install-snapshot-future-for-unknown-scm-leader.patch,
TestBugMC3FollowerStuckAfterInstallNotification.java
h3. Mechanism
{{SCMStateMachine.notifyInstallSnapshotFromLeader}} looks the leader up by its
Ratis address in {{SCMHANodeDetails.getPeerNodeDetails}} and runs
{{Preconditions.checkState(leaderDetails.isPresent())}} on the calling Ratis
thread. When no peer matches it throws {{IllegalStateException}} instead of
returning a failed future, as the "missing leader info" branch just above it
does.
Ratis ({{SnapshotInstallationHandler.notifyStateMachineToInstallSnapshot}},
3.3.1) has already set its in-progress install index at that point and clears
it on failure only in the {{whenComplete}} of the returned future. The
synchronous exception skips that, and nothing else resets the index. From then
on the division refuses every {{appendEntries}} ("installation is in progress")
and answers every later install notification with {{IN_PROGRESS}} (or an error,
when a later leader's first available index is below the stale one) without
calling the state machine again, also when the leader is an SCM it knows. The
index is only in memory, so a restart of the SCM should clear it (argued, a
restart was not run).
RATIS-1417 added a catch for this case on Ratis master in 2021, but it was
removed again by RATIS-1481 before the next release, so no Ratis release used
by Ozone has it.
h3. Trigger
The peer list is read from the configuration once, when the
{{StorageContainerManager}} is constructed. A running SCM therefore does not
know an SCM that was bootstrapped after it started:
# Three SCMs with {{ozone.scm.ha.ratis.log.purge.enabled}} set to true.
Bootstrap a fourth SCM and make it leader ({{ozone admin scm transfer}}, or an
election).
# One of the original followers, still running, falls behind the new leader's
purged log (network partition, long pause) far enough that the purge passes its
next index; the leader purges in steps of at least
{{ozone.scm.ha.ratis.log.purge.gap}} (default 1000000 entries) (argued from
code).
# The new leader sends it an install snapshot notification. Its address is not
in the follower's peer list, and the follower is stuck.
With the default {{ozone.scm.ha.ratis.log.purge.enabled=false}} the leader
purges only up to the lowest commit index of its peers, so a lagging follower
keeps receiving entries and is not sent an install notification; a follower
that is restarted instead loads the updated peer list (both argued from code,
not run). An address mismatch between the configured peers and the Ratis peer
address (HDDS-17205) leads to the same exception (code reading, not run).
h3. Impact
* The follower applies no further entries until it is restarted. Moving
leadership back to an SCM it knows does not help.
* The ring runs one replica short. In the four SCM ring of the trigger every
remaining SCM is then needed for a quorum.
No data is lost. A restart with the updated configuration should recover the
SCM (argued, not run).
h3. Reproduction
[^TestBugMC3FollowerStuckAfterInstallNotification.java]
({{hadoop-ozone/integration-test}}, target path and Maven command in the class
comment) follows the trigger on unmodified source, with purge enabled, a purge
gap of 5 and small log segments: {{bootstrapSCM("scm4", true)}}, leadership to
scm4, one follower drops {{appendEntries}} and {{installSnapshot}} RPCs through
Ratis' own {{CodeInjectionForTesting}} hook (a stand in for a partition, only
timing) until scm4 has purged past the follower's next index, then the
partition heals and leadership goes back to the original leader. It passes on
unmodified source when the bug is present:
{noformat}
Leader commit 287, follower applied 53 -> 53 in 60s
{noformat}
The follower logged 12,483 "installation is in progress" refusals, and the
stack of the one state machine call shows
{{SCMStateMachine.notifyInstallSnapshotFromLeader}} (108s).
h3. Patch
[^MC-3-fail-install-snapshot-future-for-unknown-scm-leader.patch], against
424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0. It also applies to master at
257a71ab277.
When no peer matches, {{notifyInstallSnapshotFromLeader}} returns a future that
fails on the install snapshot executor with {{IOException("No SCM peer matches
leader address ...")}} instead of throwing. Ratis then clears its in-progress
index in {{whenComplete}}, the leader retries, and the follower installs the
snapshot once the leader is an SCM it knows. The future fails on the executor
rather than being returned already failed, because Ratis checks right after the
call that the install is still in progress, and an already failed future makes
that check throw on every retry. Nothing changes when the leader is found, and
the existing "missing leader info" branch is unchanged.
While an unknown SCM stays leader the follower still cannot install from it,
and Ratis retries the notification without backoff. Each retry logs one ERROR
line on the follower ("Failed to notify StateMachine to InstallSnapshot ... No
SCM peer matches leader address ..."). In a local mini cluster that was 5269
lines in 30 seconds (about 175 per second); in 47 of those retries the future
failed before Ratis' check, which then logged an {{IllegalStateException}}
stack on the follower and a "Failed InstallSnapshot" warning on the leader.
This lasts until leadership moves to an SCM the follower knows or the follower
is restarted with the new configuration. Teaching a running SCM about new peers
is a separate change.
Covered by {{testInstallSnapshotNotificationFromUnknownLeaderFailsFuture}} in
the existing {{TestSCMStateMachine}}, which also checks that a known leader is
still used for the download. Without the change it fails with the
{{IllegalStateException}}. With it {{TestSCMStateMachine}},
{{TestSCMHAManagerImpl}} and {{TestSCMContext}} pass (23 tests), checkstyle is
clean. With the patch the reproduction fails: the notifications from scm4 fail,
and after leadership moves back the follower installs the snapshot from the
known leader and catches up to the leader's commit index ({{follower applied
285 -> 287}}, leader commit 287).
HDDS-16932 and HDDS-16928 describe the same Ratis in-progress state on an SCM
follower from other triggers (a leadership change during a successful download,
a failed install after Ratis acknowledged it); they are related, not
duplicates. The pending apache/ozone pull request for HDDS-17205 edits the same
{{checkState}} line (adds a message). The two changes are independent;
whichever lands second needs a one line rebase, and this patch uses the same
message text.
Found by TLA+ model checking and code review of the OM and SCM Ratis state
machine integration (leader readiness, snapshot install and log failure
handling) under HDDS-15926, on commit 424482eb9cfc4b835e1bf73bcdaa9e7d70df59f0.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
