[
https://issues.apache.org/jira/browse/RATIS-1825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18102987#comment-18102987
]
Anony Mouse commented on RATIS-1825:
------------------------------------
While building a Jepsen-style test harness for Ratis we scripted the listener
lifecycle end to end, and the results may be useful here - particularly for
[~szetszwo]'s question above about fixing rather than preventing.
What we observe at 3.2.2 is narrower than "not supported": every
*configuration* mechanic works. Staging a listener, promoting it to voter,
demoting it back, and removing it all commit correctly, and the promoted node
participates as a voter afterwards. What does not work is that a listener added
via {{GroupManagementApi.add}} never leaves lifecycle {{STARTING}}, so its
division serves no client request, ever, until the process restarts. It
replicates the whole time, which is what makes it look healthy.
Mechanism, pinned at the 3.2.2 tag:
* A {{groupAdd}}'d division starts in {{STARTING}} and transitions to
{{RUNNING}} only on an AppendEntries whose {{initializing}} flag is false
({{RaftServerImpl:1611}}).
* The leader clears that flag only for followers that {{checkStaging}} has
marked caught-up.
* That mark goes through {{containsInConf(f.getId())}}, and
{{PeerConfiguration.contains(id)}} defaults to FOLLOWER role only - so a staged
LISTENER is never marked, its appends stay {{initializing=true}} indefinitely,
and the division never becomes ready.
Voter adds do not hit this: once the configuration commits they are
FOLLOWER-role members, the mark lands, and they serve normally (our membership
runs confirm that).
Candidate one-line fix, if the analysis holds: {{checkStaging}}'s filter wants
{{containsInConf(id, FOLLOWER, LISTENER)}} rather than the FOLLOWER-only
default.
Still present against the 3.3.0 RC2 artifacts, mechanism source-identical. One
incidental note for anyone reproducing it: a targeted read to a staged listener
fails {{ServerNotReadyException: ... is not in [RUNNING]: current state is
STARTING}}, but a client with retries enabled reports that as a generic retry
failure, which is how it stayed invisible to us at first.
Repro sequence and preserved run artifacts:
https://github.com/hooji/ratis-jepsen (see {{jobs/08-membership-churn/}} and
the listener-probe results under {{results/}}).
Happy to file this as its own issue if you would rather track the {{STARTING}}
problem separately from the conversion-safety question this ticket raises - and
happy to test a patch either way.
> Change listener to follower or change follower to listener is not supported
> ---------------------------------------------------------------------------
>
> Key: RATIS-1825
> URL: https://issues.apache.org/jira/browse/RATIS-1825
> Project: Ratis
> Issue Type: Bug
> Reporter: Lei Qian
> Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Currently ratis does not support changing a Listener node into a Follower or
> a Follower into a Listener.
> Doing so can lead to some unexpected situations, such as a Listener not
> realizing it has become a follower and not starting election or change
> current leader node to a Listener without the node exiting Leader status.
> Need to add checks to prevent users from doing this.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
