[
https://issues.apache.org/jira/browse/RATIS-689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lokesh Jain updated RATIS-689:
------------------------------
Target Version/s: 0.5.0 (was: 0.4.0)
> Ratis client should fail fast if raft servers are in CLOSED state
> -----------------------------------------------------------------
>
> Key: RATIS-689
> URL: https://issues.apache.org/jira/browse/RATIS-689
> Project: Ratis
> Issue Type: Bug
> Reporter: Lokesh Jain
> Assignee: Lokesh Jain
> Priority: Major
>
> If raft servers are in CLOSED state then client should fail fast. Currently
> the client retries upto max number of retries before failing the request. We
> should fail the request faster in such cases.
> {code:java}
> 19/09/26 11:26:55 DEBUG impl.OrderedAsync: schedule* attempt #178 with policy
> RetryLimited(maxAttempts=180, sleepTime=1000ms) for
> RaftClientRequest:client-90F3DF988698->044832b2-4f44-46e0-9702-e6a0a7542763@group-93F633896F08,
> cid=0, seq=1*, RW,
> org.apache.hadoop.hdds.scm.XceiverClientRatis$$Lambda$74/1538849250@4905c46b
> 19/09/26 11:23:39 DEBUG impl.OrderedAsync: client-90F3DF988698: Failed*
> RaftClientRequest:client-90F3DF988698->664c4e90-08f3-46c9-a073-c93ef2a55da3@group-93F633896F08,
> cid=0, seq=1*, RW,
> org.apache.hadoop.hdds.scm.XceiverClientRatis$$Lambda$74/1538849250@4905c46b
> with {}19/09/26 11:23:39 DEBUG impl.OrderedAsync: client-90F3DF988698:
> Failed*
> RaftClientRequest:client-90F3DF988698->664c4e90-08f3-46c9-a073-c93ef2a55da3@group-93F633896F08,
> cid=0, seq=1*, RW,
> org.apache.hadoop.hdds.scm.XceiverClientRatis$$Lambda$74/1538849250@4905c46b
> with {}java.util.concurrent.CompletionException:
> org.apache.ratis.protocol.ServerNotReadyException:
> 664c4e90-08f3-46c9-a073-c93ef2a55da3@group-93F633896F08 is not in [RUNNING]:
> current state is CLOSED at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.completeReplyExceptionally(GrpcClientProtocolClient.java:339)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$000(GrpcClientProtocolClient.java:255)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:284)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:434)
> at
> org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:678)
> at
> org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
> at
> org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:397)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)Caused by:
> org.apache.ratis.protocol.ServerNotReadyException:
> 664c4e90-08f3-46c9-a073-c93ef2a55da3@group-93F633896F08 is not in [RUNNING]:
> current state is CLOSED at
> org.apache.ratis.server.impl.RaftServerImpl.lambda$assertLifeCycleState$9(RaftServerImpl.java:472)
> at org.apache.ratis.util.LifeCycle.assertCurrentState(LifeCycle.java:185) at
> org.apache.ratis.server.impl.RaftServerImpl.assertLifeCycleState(RaftServerImpl.java:471)
> at
> org.apache.ratis.server.impl.RaftServerImpl.submitClientRequestAsync(RaftServerImpl.java:536)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$submitClientRequestAsync$7(RaftServerProxy.java:333)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$null$5(RaftServerProxy.java:328)
> at org.apache.ratis.util.JavaUtils.callAsUnchecked(JavaUtils.java:109) at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$submitRequest$6(RaftServerProxy.java:328)
> at
> java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:981)
> at
> java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2124)
> at
> org.apache.ratis.server.impl.RaftServerProxy.submitRequest(RaftServerProxy.java:327)
> at
> org.apache.ratis.server.impl.RaftServerProxy.submitClientRequestAsync(RaftServerProxy.java:333)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:220)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:326)
> at
> org.apache.ratis.util.SlidingWindow$Server.processRequestsFromHead(SlidingWindow.java:429)
> at
> org.apache.ratis.util.SlidingWindow$Server.receivedRequest(SlidingWindow.java:421)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:345)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:240)
> at
> org.apache.ratis.grpc.client.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:168)
> at
> org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:263)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:686)
> ... 5 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
