[
https://issues.apache.org/jira/browse/RATIS-2000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17808370#comment-17808370
]
Tsz-wo Sze commented on RATIS-2000:
-----------------------------------
{code}
org.apache.ratis.grpc.TestWatchRequestWithGrpc.testWatchRequestAsyncChangeLeader
Time elapsed: 34.557 s <<< ERROR!
java.util.concurrent.TimeoutException
at
java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
at
org.apache.ratis.WatchRequestTests.checkMajority(WatchRequestTests.java:274)
at
org.apache.ratis.WatchRequestTests.runTestWatchRequestAsyncChangeLeader(WatchRequestTests.java:360)
at
org.apache.ratis.WatchRequestTests.runTest(WatchRequestTests.java:153)
at
org.apache.ratis.WatchRequestTests.lambda$testWatchRequestAsyncChangeLeader$6(WatchRequestTests.java:337)
at
org.apache.ratis.server.impl.MiniRaftCluster$Factory$Get.runWithNewCluster(MiniRaftCluster.java:140)
at
org.apache.ratis.server.impl.MiniRaftCluster$Factory$Get.runWithNewCluster(MiniRaftCluster.java:120)
at
org.apache.ratis.WatchRequestTests.testWatchRequestAsyncChangeLeader(WatchRequestTests.java:336)
...
{code}
`testWatchRequestAsyncChangeLeader` may timeout as above. In the log, it has
{code}
2024-01-18 01:41:51,651 [grpc-default-executor-0] ERROR impl.OrderedAsync
(OrderedAsync.java:lambda$sendRequest$9(247)) - client-6F96E8AE8714: Failed*
RaftClientRequest:client-6F96E8AE8714->s2@group-4F85081496F8, cid=64, seq=20,
RW, m7
java.util.concurrent.CompletionException: java.io.IOException:
java.lang.IllegalStateException: Failed to retain: object has already been
completely released.
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
at
java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:607)
at
java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.completeReplyExceptionally(GrpcClientProtocolClient.java:394)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.access$000(GrpcClientProtocolClient.java:300)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:331)
at
org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:481)
at
org.apache.ratis.thirdparty.io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
at
org.apache.ratis.grpc.metrics.intercept.client.MetricClientCallListener.onClose(MetricClientCallListener.java:47)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:567)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:71)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:735)
at
org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:716)
at
org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Caused by: java.io.IOException: java.lang.IllegalStateException: Failed to
retain: object has already been completely released.
at org.apache.ratis.grpc.GrpcUtil.unwrapException(GrpcUtil.java:103)
at org.apache.ratis.grpc.GrpcUtil.unwrapIOException(GrpcUtil.java:163)
at
org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onError(GrpcClientProtocolClient.java:330)
... 13 more
Caused by: java.lang.IllegalStateException: Failed to retain: object has
already been completely released.
at
org.apache.ratis.util.ReferenceCountedObject$4.retain(ReferenceCountedObject.java:183)
at
org.apache.ratis.server.impl.RaftServerProxy.submitClientRequestAsync(RaftServerProxy.java:450)
at
org.apache.ratis.grpc.server.GrpcClientProtocolService$RequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:248)
at
org.apache.ratis.grpc.server.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:364)
at
org.apache.ratis.util.SlidingWindow$Server.processRequestsFromHead(SlidingWindow.java:459)
at
org.apache.ratis.util.SlidingWindow$Server.receivedRequest(SlidingWindow.java:451)
at
org.apache.ratis.grpc.server.GrpcClientProtocolService$OrderedRequestStreamObserver.processClientRequest(GrpcClientProtocolService.java:389)
at
org.apache.ratis.grpc.server.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:272)
at
org.apache.ratis.grpc.server.GrpcClientProtocolService$RequestStreamObserver.onNext(GrpcClientProtocolService.java:191)
at
org.apache.ratis.thirdparty.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:262)
at
org.apache.ratis.thirdparty.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:329)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:314)
at
org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:833)
... 5 more
{code}
> Intermittent timeout in
> TestWatchRequestWithGrpc#testWatchRequestAsyncChangeLeader
> ----------------------------------------------------------------------------------
>
> Key: RATIS-2000
> URL: https://issues.apache.org/jira/browse/RATIS-2000
> Project: Ratis
> Issue Type: Bug
> Reporter: Attila Doroszlai
> Assignee: Tsz-wo Sze
> Priority: Critical
>
> {code}
> [ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 79.851 s <<< FAILURE! - in org.apache.ratis.grpc.TestWatchRequestWithGrpc
> [ERROR]
> org.apache.ratis.grpc.TestWatchRequestWithGrpc.testWatchRequestAsyncChangeLeader
> Time elapsed: 58.211 s <<< ERROR!
> java.util.concurrent.TimeoutException
> at
> java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> at
> org.apache.ratis.WatchRequestTests.checkMajority(WatchRequestTests.java:274)
> at
> org.apache.ratis.WatchRequestTests.runTestWatchRequestAsyncChangeLeader(WatchRequestTests.java:360)
> at
> org.apache.ratis.WatchRequestTests.runTest(WatchRequestTests.java:153)
> at
> org.apache.ratis.WatchRequestTests.lambda$testWatchRequestAsyncChangeLeader$6(WatchRequestTests.java:337)
> at
> org.apache.ratis.server.impl.MiniRaftCluster$Factory$Get.runWithNewCluster(MiniRaftCluster.java:140)
> at
> org.apache.ratis.server.impl.MiniRaftCluster$Factory$Get.runWithNewCluster(MiniRaftCluster.java:120)
> at
> org.apache.ratis.WatchRequestTests.testWatchRequestAsyncChangeLeader(WatchRequestTests.java:336)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
