Lokesh Jain created RATIS-689:
---------------------------------
Summary: Ratis client should fail fast if raft servers are in
CLOSED state
Key: RATIS-689
URL: https://issues.apache.org/jira/browse/RATIS-689
Project: Ratis
Issue Type: Bug
Reporter: Lokesh Jain
Assignee: Lokesh Jain
If raft servers are in CLOSED state then client should fail fast. Currently the
client retries upto max number of retries before failing the request. We should
fail the request faster in such cases.
{code:java}
19/09/26 11:26:55 DEBUG impl.OrderedAsync: schedule* attempt #178 with policy
RetryLimited(maxAttempts=180, sleepTime=1000ms) for
RaftClientRequest:client-90F3DF988698->044832b2-4f44-46e0-9702-e6a0a7542763@group-93F633896F08,
cid=0, seq=1*, RW,
org.apache.hadoop.hdds.scm.XceiverClientRatis$$Lambda$74/1538849250@4905c46b
19/09/26 11:23:39 DEBUG impl.OrderedAsync: client-90F3DF988698: Failed*
RaftClientRequest:client-90F3DF988698->664c4e90-08f3-46c9-a073-c93ef2a55da3@group-93F633896F08,
cid=0, seq=1*, RW,
org.apache.hadoop.hdds.scm.XceiverClientRatis$$Lambda$74/1538849250@4905c46b
with {}19/09/26 11:23:39 DEBUG impl.OrderedAsync: client-90F3DF988698: Failed*
RaftClientRequest:client-90F3DF988698->664c4e90-08f3-46c9-a073-c93ef2a55da3@group-93F633896F08,
cid=0, seq=1*, RW,
org.apache.hadoop.hdds.scm.XceiverClientRatis$$Lambda$74/1538849250@4905c46b
with {}java.util.concurrent.CompletionException:
org.apache.ratis.protocol.ServerNotReadyException:
664c4e90-08f3-46c9-a073-c93ef2a55da3@group-93F633896F08 is not in [RUNNING]:
current state is CLOSED at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.completeReplyExceptionally(GrpcClientProtocolClient.java:339)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$000(GrpcClientProtocolClient.java:255)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:284)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
at
org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at
org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
at
org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
at
org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)Caused by:
org.apache.ratis.protocol.ServerNotReadyException:
664c4e90-08f3-46c9-a073-c93ef2a55da3@group-93F633896F08 is not in [RUNNING]:
current state is CLOSED at
org.apache.ratis.server.impl.RaftServerImpl.lambda$assertLifeCycleState$9(RaftServerImpl.java:472)
at org.apache.ratis.util.LifeCycle.assertCurrentState(LifeCycle.java:185) at
org.apache.ratis.server.impl.RaftServerImpl.assertLifeCycleState(RaftServerImpl.java:471)
at
org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:536)
at
org.apache.ratis.server.impl.RaftServerProxy.lambda$submitClientRequestAsync$7(RaftServerProxy.java:333)
at
org.apache.ratis.server.impl.RaftServerProxy.lambda$null$5(RaftServerProxy.java:328)
at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:109) at
org.apache.ratis.server.impl.RaftServerProxy.lambda$submitRequest$6(RaftServerProxy.java:328)
at
java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981)
at
java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124)
at
org.apache.ratis.server.impl.RaftServerProxy.submitRequest(RaftServerProxy.java:327)
at
org.apache.ratis.server.impl.RaftServerProxy.submitClientRequestAsync(RaftServerProxy.java:333)
at
org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:220)
at
org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:326)
at
org.apache.ratis.util.SlidingWindow$Server.processRequestsFromHead(SlidingWindow.java:429)
at
org.apache.ratis.util.SlidingWindow$Server.receivedRequest(SlidingWindow.java:421)
at
org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:345)
at
org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:240)
at
org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:168)
at
org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:263)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:686)
... 5 more
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
