Shashikant Banerjee created RATIS-724:
-----------------------------------------
Summary: Leader election seem to timeout in Ratis
Key: RATIS-724
URL: https://issues.apache.org/jira/browse/RATIS-724
Project: Ratis
Issue Type: Bug
Reporter: Shashikant Banerjee
Fix For: 0.5.0
{code:java}
java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
e9477db3-627c-447b-9726-7a0202331e44@group-284D2F681BFD is not in [RUNNING]:
current state is NEWjava.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
e9477db3-627c-447b-9726-7a0202331e44@group-284D2F681BFD is not in [RUNNING]:
current state is NEW at
java.util.concurrent.FutureTask.report(FutureTask.java:122) at
java.util.concurrent.FutureTask.get(FutureTask.java:192) at
org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259)
at
org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197)
at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
e9477db3-627c-447b-9726-7a0202331e44@group-284D2F681BFD is not in [RUNNING]:
current state is NEW at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139)
at
org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265)
at
org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99)
at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204)
at
org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more2019-10-21 17:40:18,702
[4ed74939-427d-455d-852a-df3499c9dbb2@group-284D2F681BFD-LeaderElection1] INFO
impl.LeaderElection (LogUtils.java:infoOrTrace(149)) -
4ed74939-427d-455d-852a-df3499c9dbb2@group-284D2F681BFD-LeaderElection1 got
exception when requesting votes: {}java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED:
deadline exceeded after 2998885738ns at
java.util.concurrent.FutureTask.report(FutureTask.java:122) at
java.util.concurrent.FutureTask.get(FutureTask.java:192) at
org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259)
at
org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197)
at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED:
deadline exceeded after 2998885738ns at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139)
at
org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265)
at
org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99)
at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204)
at
org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more2019-10-21 17:40:18,705
[4ed74939-427d-455d-852a-df3499c9dbb2@group-284D2F681BFD-LeaderElection1] INFO
impl.LeaderElection (LeaderElection.java:logAndReturn(56)) -
4ed74939-427d-455d-852a-df3499c9dbb2@group-284D2F681BFD-LeaderElection1:
Election REJECTED; received 0 response(s) [] and 2 exception(s);
4ed74939-427d-455d-852a-df3499c9dbb2@group-284D2F681BFD:t1, leader=null,
voted=4ed74939-427d-455d-852a-df3499c9dbb2,
raftlog=4ed74939-427d-455d-852a-df3499c9dbb2@group-284D2F681BFD-SegmentedRaftLog:OPENED:c-1,f-1,i0,
conf=-1: [4ed74939-427d-455d-852a-df3499c9dbb2:192.168.29.38:50945,
31cc45f0-16ea-48bc-b870-9e866f150589:192.168.29.38:50943,
e9477db3-627c-447b-9726-7a0202331e44:192.168.29.38:50944], old=null2019-10-21
17:40:18,705
[4ed74939-427d-455d-852a-df3499c9dbb2@group-284D2F681BFD-LeaderElection1] INFO
impl.LeaderElection (LogUtils.java:infoOrTrace(149)) - Exception 0:
{}java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
e9477db3-627c-447b-9726-7a0202331e44@group-284D2F681BFD is not in [RUNNING]:
current state is NEW at
java.util.concurrent.FutureTask.report(FutureTask.java:122) at
java.util.concurrent.FutureTask.get(FutureTask.java:192) at
org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259)
at
org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197)
at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: INTERNAL:
e9477db3-627c-447b-9726-7a0202331e44@group-284D2F681BFD is not in [RUNNING]:
current state is NEW at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139)
at
org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265)
at
org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99)
at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204)
at
org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more2019-10-21 17:40:18,705
[4ed74939-427d-455d-852a-df3499c9dbb2@group-284D2F681BFD-LeaderElection1] INFO
impl.LeaderElection (LogUtils.java:infoOrTrace(149)) - Exception 1:
{}java.util.concurrent.ExecutionException:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED:
deadline exceeded after 2998885738ns at
java.util.concurrent.FutureTask.report(FutureTask.java:122) at
java.util.concurrent.FutureTask.get(FutureTask.java:192) at
org.apache.ratis.server.impl.LeaderElection.waitForResults(LeaderElection.java:259)
at
org.apache.ratis.server.impl.LeaderElection.askForVotes(LeaderElection.java:197)
at org.apache.ratis.server.impl.LeaderElection.run(LeaderElection.java:133) at
java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.ratis.thirdparty.io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED:
deadline exceeded after 2998885738ns at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:233)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:214)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:139)
at
org.apache.ratis.proto.grpc.RaftServerProtocolServiceGrpc$RaftServerProtocolServiceBlockingStub.requestVote(RaftServerProtocolServiceGrpc.java:265)
at
org.apache.ratis.grpc.server.GrpcServerProtocolClient.requestVote(GrpcServerProtocolClient.java:99)
at org.apache.ratis.grpc.server.GrpcService.requestVote(GrpcService.java:204)
at
org.apache.ratis.server.impl.LeaderElection.lambda$submitRequests$1(LeaderElection.java:234)
at java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more2019-10-21 17:40:18,708
[4ed74939-427d-455d-852a-df3499c9dbb2@group-284D2F681BFD-LeaderElection1] INFO
impl.RaftServerImpl (RaftServerImpl.java:setRole(173)) - 4ed74939-427d-455d-85
{code}
This was observed while running ozone with ratis snapshot
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
