[ https://issues.apache.org/jira/browse/RATIS-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
runzhiwang updated RATIS-988: ----------------------------- Description: client receive org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, but it did not change leader to s0. {code:java} 2020-06-26T04:32:37.6394628Z 2020-06-26 04:32:37,638 [Thread-450] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s1:c132, s0:c132, s2:c132] 2020-06-26T04:32:40.8611959Z 2020-06-26 04:32:40,860 [Thread-536] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:32:42.2757200Z 2020-06-26 04:32:42,274 [Timer-0] INFO ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - TIMED-PRINT: printing ALL groups 2020-06-26T04:32:42.2758497Z s0: RUNNING LEADER s0@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:32:42.2759383Z s1: RUNNING FOLLOWER s1@group-B309CD2D8643:t2, leader=s0, voted=null, raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:32:42.2760258Z s2: RUNNING FOLLOWER s2@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:32:44.1652305Z 2020-06-26 04:32:44,164 [Thread-624] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:32:47.3785586Z 2020-06-26 04:32:47,377 [Thread-710] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:32:50.5979127Z 2020-06-26 04:32:50,596 [Thread-797] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:32:52.2754050Z 2020-06-26 04:32:52,273 [Timer-0] INFO ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - TIMED-PRINT: printing ALL groups 2020-06-26T04:32:52.2754863Z s0: RUNNING LEADER s0@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:32:52.2755536Z s1: RUNNING FOLLOWER s1@group-B309CD2D8643:t2, leader=s0, voted=null, raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:32:52.2756213Z s2: RUNNING FOLLOWER s2@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:32:53.8130061Z 2020-06-26 04:32:53,811 [Thread-884] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:32:57.1807590Z 2020-06-26 04:32:57,179 [Thread-976] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s1:c132, s0:c132, s2:c132] 2020-06-26T04:33:00.3976035Z 2020-06-26 04:33:00,396 [Thread-1063] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s1:c132, s0:c132, s2:c132] 2020-06-26T04:33:02.2749045Z 2020-06-26 04:33:02,273 [Timer-0] INFO ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - TIMED-PRINT: printing ALL groups 2020-06-26T04:33:02.2750140Z s0: RUNNING LEADER s0@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:02.2750847Z s1: RUNNING FOLLOWER s1@group-B309CD2D8643:t2, leader=s0, voted=null, raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:02.2751499Z s2: RUNNING FOLLOWER s2@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:03.6146366Z 2020-06-26 04:33:03,613 [Thread-1150] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:33:06.8321037Z 2020-06-26 04:33:06,831 [Thread-1237] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s1:c132, s0:c132, s2:c132] 2020-06-26T04:33:10.2640359Z 2020-06-26 04:33:10,263 [Thread-1327] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:33:12.2753778Z 2020-06-26 04:33:12,273 [Timer-0] INFO ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - TIMED-PRINT: printing ALL groups 2020-06-26T04:33:12.2754582Z s0: RUNNING LEADER s0@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:12.2755259Z s1: RUNNING FOLLOWER s1@group-B309CD2D8643:t2, leader=s0, voted=null, raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:12.2755932Z s2: RUNNING FOLLOWER s2@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:13.4787369Z 2020-06-26 04:33:13,477 [Thread-1414] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:33:16.7024358Z 2020-06-26 04:33:16,699 [Thread-1501] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s1:c132, s0:c132, s2:c132] 2020-06-26T04:33:19.9173011Z 2020-06-26 04:33:19,916 [Thread-1588] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:33:22.2758152Z 2020-06-26 04:33:22,274 [Timer-0] INFO ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - TIMED-PRINT: printing ALL groups 2020-06-26T04:33:22.2758994Z s0: RUNNING LEADER s0@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:22.2760683Z s1: RUNNING FOLLOWER s1@group-B309CD2D8643:t2, leader=s0, voted=null, raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:22.2761645Z s2: RUNNING FOLLOWER s2@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:23.2077420Z 2020-06-26 04:33:23,206 [Thread-1677] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:33:26.4214334Z 2020-06-26 04:33:26,420 [Thread-1763] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s1:c132, s0:c132, s2:c132] 2020-06-26T04:33:29.6350401Z 2020-06-26 04:33:29,633 [Thread-1850] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:33:32.2761706Z 2020-06-26 04:33:32,274 [Timer-0] INFO ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - TIMED-PRINT: printing ALL groups 2020-06-26T04:33:32.2762405Z s0: RUNNING LEADER s0@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:32.2762936Z s1: RUNNING FOLLOWER s1@group-B309CD2D8643:t2, leader=s0, voted=null, raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:32.2763447Z s2: RUNNING FOLLOWER s2@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:32.8521952Z 2020-06-26 04:33:32,851 [Thread-1937] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s1:c132, s0:c132, s2:c132] 2020-06-26T04:33:36.0889022Z 2020-06-26 04:33:36,086 [Thread-2025] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] {code} was: {code:java} 2020-06-26T04:32:37.6394628Z 2020-06-26 04:32:37,638 [Thread-450] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s1:c132, s0:c132, s2:c132] 2020-06-26T04:32:40.8611959Z 2020-06-26 04:32:40,860 [Thread-536] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:32:42.2757200Z 2020-06-26 04:32:42,274 [Timer-0] INFO ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - TIMED-PRINT: printing ALL groups 2020-06-26T04:32:42.2758497Z s0: RUNNING LEADER s0@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:32:42.2759383Z s1: RUNNING FOLLOWER s1@group-B309CD2D8643:t2, leader=s0, voted=null, raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:32:42.2760258Z s2: RUNNING FOLLOWER s2@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:32:44.1652305Z 2020-06-26 04:32:44,164 [Thread-624] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:32:47.3785586Z 2020-06-26 04:32:47,377 [Thread-710] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:32:50.5979127Z 2020-06-26 04:32:50,596 [Thread-797] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:32:52.2754050Z 2020-06-26 04:32:52,273 [Timer-0] INFO ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - TIMED-PRINT: printing ALL groups 2020-06-26T04:32:52.2754863Z s0: RUNNING LEADER s0@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:32:52.2755536Z s1: RUNNING FOLLOWER s1@group-B309CD2D8643:t2, leader=s0, voted=null, raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:32:52.2756213Z s2: RUNNING FOLLOWER s2@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:32:53.8130061Z 2020-06-26 04:32:53,811 [Thread-884] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:32:57.1807590Z 2020-06-26 04:32:57,179 [Thread-976] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s1:c132, s0:c132, s2:c132] 2020-06-26T04:33:00.3976035Z 2020-06-26 04:33:00,396 [Thread-1063] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s1:c132, s0:c132, s2:c132] 2020-06-26T04:33:02.2749045Z 2020-06-26 04:33:02,273 [Timer-0] INFO ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - TIMED-PRINT: printing ALL groups 2020-06-26T04:33:02.2750140Z s0: RUNNING LEADER s0@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:02.2750847Z s1: RUNNING FOLLOWER s1@group-B309CD2D8643:t2, leader=s0, voted=null, raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:02.2751499Z s2: RUNNING FOLLOWER s2@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:03.6146366Z 2020-06-26 04:33:03,613 [Thread-1150] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:33:06.8321037Z 2020-06-26 04:33:06,831 [Thread-1237] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s1:c132, s0:c132, s2:c132] 2020-06-26T04:33:10.2640359Z 2020-06-26 04:33:10,263 [Thread-1327] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:33:12.2753778Z 2020-06-26 04:33:12,273 [Timer-0] INFO ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - TIMED-PRINT: printing ALL groups 2020-06-26T04:33:12.2754582Z s0: RUNNING LEADER s0@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:12.2755259Z s1: RUNNING FOLLOWER s1@group-B309CD2D8643:t2, leader=s0, voted=null, raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:12.2755932Z s2: RUNNING FOLLOWER s2@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:13.4787369Z 2020-06-26 04:33:13,477 [Thread-1414] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:33:16.7024358Z 2020-06-26 04:33:16,699 [Thread-1501] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s1:c132, s0:c132, s2:c132] 2020-06-26T04:33:19.9173011Z 2020-06-26 04:33:19,916 [Thread-1588] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:33:22.2758152Z 2020-06-26 04:33:22,274 [Timer-0] INFO ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - TIMED-PRINT: printing ALL groups 2020-06-26T04:33:22.2758994Z s0: RUNNING LEADER s0@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:22.2760683Z s1: RUNNING FOLLOWER s1@group-B309CD2D8643:t2, leader=s0, voted=null, raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:22.2761645Z s2: RUNNING FOLLOWER s2@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:23.2077420Z 2020-06-26 04:33:23,206 [Thread-1677] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:33:26.4214334Z 2020-06-26 04:33:26,420 [Thread-1763] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s1:c132, s0:c132, s2:c132] 2020-06-26T04:33:29.6350401Z 2020-06-26 04:33:29,633 [Thread-1850] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] 2020-06-26T04:33:32.2761706Z 2020-06-26 04:33:32,274 [Timer-0] INFO ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - TIMED-PRINT: printing ALL groups 2020-06-26T04:33:32.2762405Z s0: RUNNING LEADER s0@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:32.2762936Z s1: RUNNING FOLLOWER s1@group-B309CD2D8643:t2, leader=s0, voted=null, raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:32.2763447Z s2: RUNNING FOLLOWER s2@group-B309CD2D8643:t2, leader=s0, voted=s0, raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING 2020-06-26T04:33:32.8521952Z 2020-06-26 04:33:32,851 [Thread-1937] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s1:c132, s0:c132, s2:c132] 2020-06-26T04:33:36.0889022Z 2020-06-26 04:33:36,086 [Thread-2025] INFO client.GrpcClientProtocolService (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, RW, Message:6d3636, reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, FAILED org.apache.ratis.protocol.NotLeaderException: Server s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, commits[s2:c132, s0:c132, s1:c132] {code} > Fix no change leader as expected > -------------------------------- > > Key: RATIS-988 > URL: https://issues.apache.org/jira/browse/RATIS-988 > Project: Ratis > Issue Type: Bug > Reporter: runzhiwang > Assignee: runzhiwang > Priority: Major > > client receive org.apache.ratis.protocol.NotLeaderException: Server > s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, but it did not > change leader to s0. > {code:java} > 2020-06-26T04:32:37.6394628Z 2020-06-26 04:32:37,638 [Thread-450] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s1:c132, s0:c132, s2:c132] > 2020-06-26T04:32:40.8611959Z 2020-06-26 04:32:40,860 [Thread-536] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s2:c132, s0:c132, s1:c132] > 2020-06-26T04:32:42.2757200Z 2020-06-26 04:32:42,274 [Timer-0] INFO > ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - > TIMED-PRINT: printing ALL groups > 2020-06-26T04:32:42.2758497Z s0: RUNNING LEADER > s0@group-B309CD2D8643:t2, leader=s0, voted=s0, > raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:32:42.2759383Z s1: RUNNING FOLLOWER > s1@group-B309CD2D8643:t2, leader=s0, voted=null, > raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:32:42.2760258Z s2: RUNNING FOLLOWER > s2@group-B309CD2D8643:t2, leader=s0, voted=s0, > raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:32:44.1652305Z 2020-06-26 04:32:44,164 [Thread-624] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s2:c132, s0:c132, s1:c132] > 2020-06-26T04:32:47.3785586Z 2020-06-26 04:32:47,377 [Thread-710] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s2:c132, s0:c132, s1:c132] > 2020-06-26T04:32:50.5979127Z 2020-06-26 04:32:50,596 [Thread-797] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s2:c132, s0:c132, s1:c132] > 2020-06-26T04:32:52.2754050Z 2020-06-26 04:32:52,273 [Timer-0] INFO > ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - > TIMED-PRINT: printing ALL groups > 2020-06-26T04:32:52.2754863Z s0: RUNNING LEADER > s0@group-B309CD2D8643:t2, leader=s0, voted=s0, > raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:32:52.2755536Z s1: RUNNING FOLLOWER > s1@group-B309CD2D8643:t2, leader=s0, voted=null, > raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:32:52.2756213Z s2: RUNNING FOLLOWER > s2@group-B309CD2D8643:t2, leader=s0, voted=s0, > raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:32:53.8130061Z 2020-06-26 04:32:53,811 [Thread-884] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s2:c132, s0:c132, s1:c132] > 2020-06-26T04:32:57.1807590Z 2020-06-26 04:32:57,179 [Thread-976] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s1:c132, s0:c132, s2:c132] > 2020-06-26T04:33:00.3976035Z 2020-06-26 04:33:00,396 [Thread-1063] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s1:c132, s0:c132, s2:c132] > 2020-06-26T04:33:02.2749045Z 2020-06-26 04:33:02,273 [Timer-0] INFO > ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - > TIMED-PRINT: printing ALL groups > 2020-06-26T04:33:02.2750140Z s0: RUNNING LEADER > s0@group-B309CD2D8643:t2, leader=s0, voted=s0, > raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:33:02.2750847Z s1: RUNNING FOLLOWER > s1@group-B309CD2D8643:t2, leader=s0, voted=null, > raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:33:02.2751499Z s2: RUNNING FOLLOWER > s2@group-B309CD2D8643:t2, leader=s0, voted=s0, > raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:33:03.6146366Z 2020-06-26 04:33:03,613 [Thread-1150] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s2:c132, s0:c132, s1:c132] > 2020-06-26T04:33:06.8321037Z 2020-06-26 04:33:06,831 [Thread-1237] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s1:c132, s0:c132, s2:c132] > 2020-06-26T04:33:10.2640359Z 2020-06-26 04:33:10,263 [Thread-1327] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s2:c132, s0:c132, s1:c132] > 2020-06-26T04:33:12.2753778Z 2020-06-26 04:33:12,273 [Timer-0] INFO > ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - > TIMED-PRINT: printing ALL groups > 2020-06-26T04:33:12.2754582Z s0: RUNNING LEADER > s0@group-B309CD2D8643:t2, leader=s0, voted=s0, > raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:33:12.2755259Z s1: RUNNING FOLLOWER > s1@group-B309CD2D8643:t2, leader=s0, voted=null, > raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:33:12.2755932Z s2: RUNNING FOLLOWER > s2@group-B309CD2D8643:t2, leader=s0, voted=s0, > raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:33:13.4787369Z 2020-06-26 04:33:13,477 [Thread-1414] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s2:c132, s0:c132, s1:c132] > 2020-06-26T04:33:16.7024358Z 2020-06-26 04:33:16,699 [Thread-1501] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s1:c132, s0:c132, s2:c132] > 2020-06-26T04:33:19.9173011Z 2020-06-26 04:33:19,916 [Thread-1588] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s2:c132, s0:c132, s1:c132] > 2020-06-26T04:33:22.2758152Z 2020-06-26 04:33:22,274 [Timer-0] INFO > ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - > TIMED-PRINT: printing ALL groups > 2020-06-26T04:33:22.2758994Z s0: RUNNING LEADER > s0@group-B309CD2D8643:t2, leader=s0, voted=s0, > raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:33:22.2760683Z s1: RUNNING FOLLOWER > s1@group-B309CD2D8643:t2, leader=s0, voted=null, > raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:33:22.2761645Z s2: RUNNING FOLLOWER > s2@group-B309CD2D8643:t2, leader=s0, voted=s0, > raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:33:23.2077420Z 2020-06-26 04:33:23,206 [Thread-1677] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s2:c132, s0:c132, s1:c132] > 2020-06-26T04:33:26.4214334Z 2020-06-26 04:33:26,420 [Thread-1763] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s1:c132, s0:c132, s2:c132] > 2020-06-26T04:33:29.6350401Z 2020-06-26 04:33:29,633 [Thread-1850] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s2:c132, s0:c132, s1:c132] > 2020-06-26T04:33:32.2761706Z 2020-06-26 04:33:32,274 [Timer-0] INFO > ratis.MiniRaftCluster (MiniRaftCluster.java:lambda$null$5(297)) - > TIMED-PRINT: printing ALL groups > 2020-06-26T04:33:32.2762405Z s0: RUNNING LEADER > s0@group-B309CD2D8643:t2, leader=s0, voted=s0, > raftlog=s0@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f133,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:33:32.2762936Z s1: RUNNING FOLLOWER > s1@group-B309CD2D8643:t2, leader=s0, voted=null, > raftlog=s1@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:33:32.2763447Z s2: RUNNING FOLLOWER > s2@group-B309CD2D8643:t2, leader=s0, voted=s0, > raftlog=s2@group-B309CD2D8643-SegmentedRaftLog:OPENED:c132,f132,i133, conf=0: > [s0:0.0.0.0:43317, s1:0.0.0.0:35077, s2:0.0.0.0:39141], old=null RUNNING > 2020-06-26T04:33:32.8521952Z 2020-06-26 04:33:32,851 [Thread-1937] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s1@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s1@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s1:c132, s0:c132, s2:c132] > 2020-06-26T04:33:36.0889022Z 2020-06-26 04:33:36,086 [Thread-2025] INFO > client.GrpcClientProtocolService > (GrpcClientProtocolService.java:lambda$processClientRequest$0(283)) - Failed > RaftClientRequest:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, seq=0, > RW, Message:6d3636, > reply=RaftClientReply:client-54283F516BF5->s2@group-B309CD2D8643, cid=66, > FAILED org.apache.ratis.protocol.NotLeaderException: Server > s2@group-B309CD2D8643 is not the leader s0:0.0.0.0:43317, logIndex=0, > commits[s2:c132, s0:c132, s1:c132] > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)