[
https://issues.apache.org/jira/browse/RATIS-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18107346#comment-18107346
]
Xinyu Tan commented on RATIS-2661:
----------------------------------
## 1. First-principles analysis
Raft cannot derive its initial membership from an empty state. Leader
election, quorum calculation, log commitment, and
configuration changes all require a known voting configuration. A
configuration-change protocol can safely transform an existing
configuration into another configuration, but it cannot establish the first
configuration when no electorate exists.
Therefore, the initial membership is necessarily an external bootstrap fact.
It must be supplied consistently by the embedding
application, static configuration, or another control-plane service. This
part cannot be solved by Raft consensus alone because
consensus has not yet been established.
## 2. Responsibility of the external coordinator
For a newly created multi-replica group, the external coordinator should
choose:
1. A unique group ID.
2. The complete initial membership.
3. The same peer information for every intended replica.
It should send the same group ID and membership to all intended peers and
retry partial failures. Group creation is not an atomic
distributed operation provided by Raft itself, so the coordinator must expect
that some peers may accept the request while others
are temporarily unavailable.
An empty peer list is useful when adding a new server to an already active
group: the existing leader can later transfer the
committed configuration to that server. However, an empty peer list is not a
valid bootstrap configuration for a brand-new group
because no leader or quorum exists from which the membership could be learned.
## 3. The liveness gap in the current implementation
Ratis should reliably hand the externally supplied bootstrap membership over
to Raft. Currently, groupAdd may succeed while the
initial membership still exists only in memory. The group directory may
already exist, but the first formal configuration entry
may not yet have been persisted.
If every peer restarts during this window, directory recovery retains only
the group ID and reconstructs the group with an empty
membership. The recovered group cannot define a quorum, elect a leader, or
transition to RUNNING.
At the same time, SET_UNCONDITIONALLY is not a bootstrap escape hatch. It
requires the division to be RUNNING and ultimately
relies on a leader-driven configuration-change path. Consequently, membership
is required to reach RUNNING, while RUNNING and a
leader are required to restore membership. This creates a genuine liveness
cycle.
I was also able to reproduce this behavior with a standalone test against the
current 3.3.0-SNAPSHOT code, so the problem is not
limited to the reported 3.2.2 deployment.
## 4. Proposed minimal fix
The simplest fix is to persist the bootstrap membership atomically before
acknowledging a successful groupAdd.
Recovery should follow this precedence:
1. Use the formal configuration reconstructed from the Raft log or persisted
Raft metadata, if available.
2. Otherwise, use the persisted bootstrap membership.
3. Use an empty configuration only when neither source exists.
The bootstrap record is not a second Raft configuration history. It is only a
durable creation fact used before Raft has
successfully recorded its initial configuration. Once the initial
configuration has been persisted through Raft, the formal Raft
state must take precedence, and all later membership changes must continue
through the normal configuration-change protocol.
## 5. Required retry and conflict semantics
To make partial creation safely recoverable, group creation should be
idempotent:
1. The group does not exist: persist the bootstrap membership and create it.
2. The same group ID and the same bootstrap membership are supplied again:
return success or an idempotent “already created”
result.
3. The same group ID is supplied with different bootstrap membership: reject
it as a conflict.
4. The group is already active: membership changes must use setConfiguration,
not another bootstrap request.
For compatibility with existing empty recovered directories, Ratis could also
support a narrowly guarded repair path. It should
only be allowed when the group is demonstrably pristine: STARTING, empty
membership, empty log, and no snapshot or persisted
formal configuration. Supplying the original membership in that state
completes an interrupted bootstrap; it does not overwrite an
established Raft configuration.
## 6. Scope of the change
This does not require redesigning Raft, leader election, or joint consensus.
The consensus implementation can remain unchanged.
The missing piece is the lifecycle boundary between externally coordinated
group creation and the point at which Raft durably owns
the configuration.
If the application wants to abandon a partially created, never-activated
group and use a different initial membership, it should
remove the pristine partial copies and preferably create a new group ID.
Reusing the old group ID with different membership could
conflict with delayed or retried creation requests and should not be treated
as an ordinary recovery operation.
> Recovered Raft group with no persisted configuration has no path out of
> STARTING
> --------------------------------------------------------------------------------
>
> Key: RATIS-2661
> URL: https://issues.apache.org/jira/browse/RATIS-2661
> Project: Ratis
> Issue Type: Bug
> Components: raft-group, server
> Affects Versions: 3.2.2
> Environment: Apache Ratis 3.2.2, revision
> 288c032064ce3d168b8a763e248a326459a4a9b7. gRPC transport, three voting peers,
> dynamic multi-group usage. Observed in an IoTDB-based 2.0.10.2 deployment.
> Not yet verified against newer Ratis releases.
> Reporter: Yongzao Dan
> Assignee: Xinyu Tan
> Priority: Major
> Labels: liveness
> Attachments: ratis-3.2.2-uninitialized-group-recovery-sanitized.log
>
>
> h2. Summary
> A dynamically managed Raft group can become permanently unrecoverable if all
> peers restart before the initial configuration entry is persisted.
> During automatic directory recovery, Ratis reconstructs the group using only
> its group ID. If no configuration exists in storage, the recovered group has
> an empty peer list. The division remains in STARTING with reason NOT_IN_CONF.
> The embedding application still knows the original membership and invokes
> setConfiguration with SET_UNCONDITIONALLY, but RaftServerImpl rejects the
> request because the lifecycle is not RUNNING. Since a group with an empty
> configuration cannot elect a leader, there is no apparent supported path to
> restore the membership.
> h2. Observed failure sequence
> # A new three-peer group was created with the correct initial membership.
> # Peer RPC failures prevented the group from forming a majority.
> # The group remained at term 0 and reached PRE_VOTE round 7808.
> # No Raft log or configuration entry was created; shutdown reported stopIndex
> = -1.
> # All servers were restarted while preserving their storage directories.
> # Each server recovered the group with peers:[] and entered STARTING /
> FOLLOWER / NOT_IN_CONF.
> # The application invoked SET_UNCONDITIONALLY with the original three peers.
> # Every request failed with ServerNotReadyException because the group was
> still STARTING.
> # After peer communication recovered, newly created groups elected leaders
> normally, but this recovered group remained unavailable.
> A sanitized log excerpt containing this sequence is attached.
> h2. Suspected root cause
> In Ratis 3.2.2, RaftServerProxy.initGroupDir() recovers a directory as:
> {code:java}
> addGroup(RaftGroup.valueOf(groupId), StartupOption.RECOVER);
> {code}
> This RaftGroup contains no peers. ServerState.initialize() only replaces that
> empty configuration if readRaftConfiguration() returns a persisted
> configuration.
> RaftServerImpl.start() does not call startAsPeer() when the local peer is
> absent from the configuration. It sets the role to FOLLOWER with NOT_IN_CONF,
> while the lifecycle remains STARTING. startAsPeer() is the path that
> transitions the lifecycle to RUNNING.
> RaftServerImpl.setConfigurationAsync() then rejects the recovery request
> before inspecting its mode:
> {code:java}
> assertLifeCycleState(LifeCycle.States.RUNNING);
> {code}
> The normal configuration-change path also checks for a ready leader. This
> creates an unbreakable liveness cycle:
> * Membership is required to elect a leader and reach RUNNING.
> * RUNNING and a leader are required to restore membership.
> Simply relaxing the lifecycle assertion may therefore be insufficient; a
> guarded bootstrap path would also need to handle the absence of a leader.
> h2. Proposed reproduction
> This reproduction is inferred from the production incident and has not yet
> been reduced to a standalone Ratis test.
> # Start three Ratis servers and add a new group containing all three peers.
> # Block peer RPC before the first configuration entry is written or committed.
> # Verify that the group remains at term 0 with last log index -1.
> # Stop all servers while preserving their group directories.
> # Restart them through the StartupOption.RECOVER directory-scanning path.
> # Call setConfiguration with SET_UNCONDITIONALLY and the original peer list.
> # Verify that every division remains in STARTING and rejects the request.
> h2. Expected behavior
> Recovery should not leave an uninitialized group in a state with no supported
> transition to RUNNING. Ratis should provide a non-destructive way to restore
> bootstrap membership when no configuration or Raft log has ever been
> persisted.
> h2. Impact
> A transient bootstrap communication failure followed by a restart becomes a
> permanent outage for that group. Repeated restarts and configuration retries
> do not help. Recovery currently requires out-of-band storage intervention or
> application-specific group recreation.
> h2. Possible fix directions
> * Persist enough bootstrap membership information when the group directory is
> created.
> * Allow callers to provide membership for recovered group IDs that have no
> stored configuration.
> * Provide a guarded re-bootstrap operation for STARTING + empty configuration
> + empty log. If SET_UNCONDITIONALLY is reused, it would also need a safe
> bootstrap path that does not depend on an existing leader.
> Is there an existing supported recovery procedure for this state? If so,
> documenting that procedure would also help dynamic multi-group applications.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
