Hi Snehasish, Thanks for trying Ratis!
> INFO [2025-12-29 16:05:03,405] [grpc-default-executor-2] [VoteContext]: n4@group-ABB3109A44C2-LISTENER: > reject PRE_VOTE from n2: this server is a listener, who is a non-voting member It may be the case that the setConf was not yet completed before killing the leader. So, n4 remained a listener. I suggest running a `ratis sh group info` command first before killing the leader. Could you try it? Please feel free to let us know if you have more questions. Tsz-Wo On Tue, Jan 6, 2026 at 2:51 AM Snehasish Roy <[email protected]> wrote: > Hello everyone, > > Bumping this thread. Please let me know if further information is needed to > debug this issue. > > > Regards, > Snehasish > > On Mon, 29 Dec, 2025, 4:23 pm Snehasish Roy, <[email protected]> > wrote: > > > Hello everyone, > > > > Happy Holidays. This is my first email to this community so kindly excuse > > me for any mistakes. > > > > I initially started a 3 node Ratis Cluster and then added a listener in > > the Cluster using the setConfiguration(List.of(n1,n2,n3), List.of(n4)) > > based on the following documentation > > https://jojochuang.github.io/ratis-site/docs/developer-guide/listeners > > > > ``` > > INFO [2025-12-29 15:57:01,887] [n1-server-thread1] > [RaftServer$Division]: > > n1@group-ABB3109A44C2-LeaderStateImpl: startSetConfiguration > > SetConfigurationRequest:client-044D31187FB4->n1@group-ABB3109A44C2, > > cid=3, seq=null, RW, null, SET_UNCONDITIONALLY, servers:[n1|0.0.0.0:9000 > , > > n2|0.0.0.0:9001, n3|0.0.0.0:9002], listeners:[n4|0.0.0.0:9003] > > ``` > > > > Then I killed one of the Ratis follower node (n3) followed by promoting > > the listener to the follower using setConfiguration(List.of(n1,n2,n4)) > > command to maintain the cluster size of 3. > > Please note that n3 has been removed from the list of followers and there > > are no more listeners in the cluster and there were no failures observed > > while issuing the command. > > > > ``` > > INFO [2025-12-29 16:02:54,227] [n1-server-thread2] > [RaftServer$Division]: > > n1@group-ABB3109A44C2-LeaderStateImpl: startSetConfiguration > > SetConfigurationRequest:client-2438CA24E2F3->n1@group-ABB3109A44C2, > > cid=4, seq=null, RW, null, SET_UNCONDITIONALLY, servers:[n1|0.0.0.0:9000 > , > > n2|0.0.0.0:9001, n4|0.0.0.0:9003], listeners:[] > > ``` > > > > Then I killed the leader instance n1. Post which n2 attempted to become a > > leader and starts asking for votes from n1 and n4. There is no response > > from n1 as it's not alive and n4 is rejecting the pre_vote request from > n2 > > because it still thinks it's a listener. > > > > Logs from n2 > > ``` > > INFO [2025-12-29 16:04:10,051] [n2@group-ABB3109A44C2-LeaderElection30] > > [LeaderElection]: n2@group-ABB3109A44C2-LeaderElection30 PRE_VOTE round > > 0: submit vote requests at term 1 for conf: {index: 15, cur=peers:[n1| > > 0.0.0.0:9000, n2|0.0.0.0:9001, n4|0.0.0.0:9003]|listeners:[], old=null} > > INFO [2025-12-29 16:04:10,052] [n2@group-ABB3109A44C2-LeaderElection30] > > [LeaderElection]: n2@group-ABB3109A44C2-LeaderElection30 got exception > > when requesting votes: java.util.concurrent.ExecutionException: > > org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: > io > > exception > > INFO [2025-12-29 16:04:10,054] [n2@group-ABB3109A44C2-LeaderElection30] > > [LeaderElection]: n2@group-ABB3109A44C2-LeaderElection30: PRE_VOTE > > REJECTED received 1 response(s) and 1 exception(s): > > INFO [2025-12-29 16:04:10,054] [n2@group-ABB3109A44C2-LeaderElection30] > > [LeaderElection]: Response 0: n2<-n4#0:FAIL-t1-last:(t:1, i:16) > > INFO [2025-12-29 16:04:10,054] [n2@group-ABB3109A44C2-LeaderElection30] > > [LeaderElection]: Exception 1: java.util.concurrent.ExecutionException: > > org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: UNAVAILABLE: > io > > exception > > ``` > > > > > > Due to lack of leader, the cluster is no more stable. > > > > Logs from n4 > > ``` > > INFO [2025-12-29 16:05:03,405] [grpc-default-executor-2] > > [RaftServer$Division]: n4@group-ABB3109A44C2: receive > > requestVote(PRE_VOTE, n2, group-ABB3109A44C2, 1, (t:1, i:16)) > > INFO [2025-12-29 16:05:03,405] [grpc-default-executor-2] [VoteContext]: > > n4@group-ABB3109A44C2-LISTENER: reject PRE_VOTE from n2: this server is > a > > listener, who is a non-voting member > > INFO [2025-12-29 16:05:03,405] [grpc-default-executor-2] > > [RaftServer$Division]: n4@group-ABB3109A44C2 replies to PRE_VOTE vote > > request: n2<-n4#0:FAIL-t1-last:(t:1, i:16). Peer's state: > > n4@group-ABB3109A44C2:t1, leader=n1, voted=null, > > raftlog=Memoized:n4@group-ABB3109A44C2-SegmentedRaftLog > :OPENED:c16:last(t:1, > > i:16), conf=conf: {index: 15, cur=peers:[n1|0.0.0.0:9000, n2| > 0.0.0.0:9001, > > n4|0.0.0.0:9003]|listeners:[], old=null} > > ``` > > > > So my question is how to correctly promote a listener to a follower? Did > I > > miss some step? Or is there a bug in the code? If it's the latter, I > would > > be happy to contribute. Please let me know if you need any more debugging > > information. > > > > Thank you again for looking into this issue. > > > > > > Regards, > > Snehasish > > >
