Yongzao Dan created RATIS-2661:
----------------------------------
Summary: Recovered Raft group with no persisted configuration has
no path out of STARTING
Key: RATIS-2661
URL: https://issues.apache.org/jira/browse/RATIS-2661
Project: Ratis
Issue Type: Bug
Components: raft-group, server
Affects Versions: 3.2.2
Environment: Apache Ratis 3.2.2, revision
288c032064ce3d168b8a763e248a326459a4a9b7. gRPC transport, three voting peers,
dynamic multi-group usage. Observed in an IoTDB-based 2.0.10.2 deployment. Not
yet verified against newer Ratis releases.
Reporter: Yongzao Dan
Attachments: ratis-3.2.2-uninitialized-group-recovery-sanitized.log
h2. Summary
A dynamically managed Raft group can become permanently unrecoverable if all
peers restart before the initial configuration entry is persisted.
During automatic directory recovery, Ratis reconstructs the group using only
its group ID. If no configuration exists in storage, the recovered group has an
empty peer list. The division remains in STARTING with reason NOT_IN_CONF.
The embedding application still knows the original membership and invokes
setConfiguration with SET_UNCONDITIONALLY, but RaftServerImpl rejects the
request because the lifecycle is not RUNNING. Since a group with an empty
configuration cannot elect a leader, there is no apparent supported path to
restore the membership.
h2. Observed failure sequence
# A new three-peer group was created with the correct initial membership.
# Peer RPC failures prevented the group from forming a majority.
# The group remained at term 0 and reached PRE_VOTE round 7808.
# No Raft log or configuration entry was created; shutdown reported stopIndex =
-1.
# All servers were restarted while preserving their storage directories.
# Each server recovered the group with peers:[] and entered STARTING / FOLLOWER
/ NOT_IN_CONF.
# The application invoked SET_UNCONDITIONALLY with the original three peers.
# Every request failed with ServerNotReadyException because the group was still
STARTING.
# After peer communication recovered, newly created groups elected leaders
normally, but this recovered group remained unavailable.
A sanitized log excerpt containing this sequence is attached.
h2. Suspected root cause
In Ratis 3.2.2, RaftServerProxy.initGroupDir() recovers a directory as:
{code:java}
addGroup(RaftGroup.valueOf(groupId), StartupOption.RECOVER);
{code}
This RaftGroup contains no peers. ServerState.initialize() only replaces that
empty configuration if readRaftConfiguration() returns a persisted
configuration.
RaftServerImpl.start() does not call startAsPeer() when the local peer is
absent from the configuration. It sets the role to FOLLOWER with NOT_IN_CONF,
while the lifecycle remains STARTING. startAsPeer() is the path that
transitions the lifecycle to RUNNING.
RaftServerImpl.setConfigurationAsync() then rejects the recovery request before
inspecting its mode:
{code:java}
assertLifeCycleState(LifeCycle.States.RUNNING);
{code}
The normal configuration-change path also checks for a ready leader. This
creates an unbreakable liveness cycle:
* Membership is required to elect a leader and reach RUNNING.
* RUNNING and a leader are required to restore membership.
Simply relaxing the lifecycle assertion may therefore be insufficient; a
guarded bootstrap path would also need to handle the absence of a leader.
h2. Proposed reproduction
This reproduction is inferred from the production incident and has not yet been
reduced to a standalone Ratis test.
# Start three Ratis servers and add a new group containing all three peers.
# Block peer RPC before the first configuration entry is written or committed.
# Verify that the group remains at term 0 with last log index -1.
# Stop all servers while preserving their group directories.
# Restart them through the StartupOption.RECOVER directory-scanning path.
# Call setConfiguration with SET_UNCONDITIONALLY and the original peer list.
# Verify that every division remains in STARTING and rejects the request.
h2. Expected behavior
Recovery should not leave an uninitialized group in a state with no supported
transition to RUNNING. Ratis should provide a non-destructive way to restore
bootstrap membership when no configuration or Raft log has ever been persisted.
h2. Impact
A transient bootstrap communication failure followed by a restart becomes a
permanent outage for that group. Repeated restarts and configuration retries do
not help. Recovery currently requires out-of-band storage intervention or
application-specific group recreation.
h2. Possible fix directions
* Persist enough bootstrap membership information when the group directory is
created.
* Allow callers to provide membership for recovered group IDs that have no
stored configuration.
* Provide a guarded re-bootstrap operation for STARTING + empty configuration +
empty log. If SET_UNCONDITIONALLY is reused, it would also need a safe
bootstrap path that does not depend on an existing leader.
Is there an existing supported recovery procedure for this state? If so,
documenting that procedure would also help dynamic multi-group applications.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
