[
https://issues.apache.org/jira/browse/IGNITE-23670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17897929#comment-17897929
]
Andrey Khitrin commented on IGNITE-23670:
-----------------------------------------
A possible explanation (just a raw hypothesis, I don't have good knowledge of
this code):
# Client tried to get node list from server node several times.
# Server node was busy processing previously loaded data and didn't answer in
expected time.
# Client node got exhausted for some reason (number of attempts, for example;
we have an enough long stack here) and returned `false` from
`ReliableChannel.shouldRetry` method.
# As a result, in `ClientFutureUtils.doWithRetryAsync`, we've entered `else`
branch but without any exception.
> "doWithRetry failed without exception" when calling Ignite.clusterNodes()
> -------------------------------------------------------------------------
>
> Key: IGNITE-23670
> URL: https://issues.apache.org/jira/browse/IGNITE-23670
> Project: Ignite
> Issue Type: Bug
> Components: thin client
> Affects Versions: 3.0
> Environment: 1. 3 server nodes in a single cluster
> 2. 2-3 client nodes
> 3. Some data loaded before this request
> 4. Some transactional work with previously loaded data
> Reporter: Andrey Khitrin
> Priority: Major
>
> In a routine cluster test on commit
> `48de8ec84f4ec0acc10bf34d9f59565191f6c85c`, the following stack trace
> happened:
> {code:java}
> org.apache.ignite.lang.IgniteException: doWithRetry failed without exception
> at
> org.apache.ignite.internal.util.ViewUtils.ensurePublicException(ViewUtils.java:80)
> ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
> at org.apache.ignite.internal.util.ViewUtils.sync(ViewUtils.java:54)
> ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.TcpIgniteClient.clusterNodes(TcpIgniteClient.java:190)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.gridgain.poc.framework.worker.ignite3.task.RestartServerTask.nodesToStop(RestartServerTask.java:238)
> ~[poc-tester-ignite3-0.5.0-SNAPSHOT.jar:?]
> at
> org.gridgain.poc.framework.worker.ignite3.task.RestartServerTask.body0(RestartServerTask.java:112)
> ~[poc-tester-ignite3-0.5.0-SNAPSHOT.jar:?]
> at
> org.gridgain.poc.framework.worker.ignite3.task.AbstractTask.body(AbstractTask.java:352)
> ~[poc-tester-ignite3-0.5.0-SNAPSHOT.jar:?]
> at
> org.gridgain.poc.framework.worker.task.TaskLooper.run(TaskLooper.java:78)
> [poc-tester-core-0.5.0-SNAPSHOT.jar:?]
> at java.lang.Thread.run(Thread.java:829) [?:?]
> Caused by: org.apache.ignite.lang.IgniteException: doWithRetry failed without
> exception
> at
> org.apache.ignite.internal.lang.IgniteExceptionMapperUtil.lambda$mapToPublicException$2(IgniteExceptionMapperUtil.java:88)
> ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.lang.IgniteExceptionMapperUtil.mapCheckingResultIsPublic(IgniteExceptionMapperUtil.java:141)
> ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.lang.IgniteExceptionMapperUtil.mapToPublicException(IgniteExceptionMapperUtil.java:137)
> ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.lang.IgniteExceptionMapperUtil.mapToPublicException(IgniteExceptionMapperUtil.java:88)
> ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.util.ViewUtils.ensurePublicException(ViewUtils.java:78)
> ~[ignite-core-3.0.0-SNAPSHOT.jar:?]
> ... 7 more
> Caused by: java.lang.IllegalStateException: doWithRetry failed without
> exception
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:78)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.lambda$doWithRetryAsync$0(ClientFutureUtils.java:74)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.uniWhenCompleteStage(CompletableFuture.java:883)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.whenComplete(CompletableFuture.java:2251)
> ~[?:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:45)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ReliableChannel.getDefaultChannelAsync(ReliableChannel.java:565)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ReliableChannel.getChannelAsync(ReliableChannel.java:319)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ReliableChannel.lambda$serviceAsync$2(ReliableChannel.java:247)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:56)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ClientFutureUtils.doWithRetryAsync(ClientFutureUtils.java:45)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ReliableChannel.serviceAsync(ReliableChannel.java:246)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.ReliableChannel.serviceAsync(ReliableChannel.java:279)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> at
> org.apache.ignite.internal.client.TcpIgniteClient.clusterNodesAsync(TcpIgniteClient.java:196)
> ~[ignite-client-3.0.0-SNAPSHOT.jar:?]
> ... 6 more
> {code}
> It's doubly suspicious because there's a following comment in the code:
> {code:java}
> if (ctx.errors == null || ctx.errors.isEmpty()) {
> // Should not happen.
> resFut.completeExceptionally(new IllegalStateException("doWithRetry
> failed without exception"));
> {code}
> Nevertheless, it happened.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
