[ 
https://issues.apache.org/jira/browse/FLINK-33278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17778717#comment-17778717
 ] 

Jiabao Sun commented on FLINK-33278:
------------------------------------

The hickup interval is nearly 10 seconds in the logs between 01:02:10.686 and 
01:02:19.074, so I suspected that the RPC timeout.
As for the specific reason, it may need further survey.

The break entrance:
 !screenshot-4.png! 

Stack trace:
{code:java}
1825 [ForkJoinPool-1-worker-1] INFO  
org.apache.flink.runtime.rpc.pekko.PekkoRpcServiceUtils [] - Trying to start 
actor system, external address localhost:0, bind address 0.0.0.0:0.
3007 [flink-pekko.actor.default-dispatcher-5] INFO  
org.apache.pekko.event.slf4j.Slf4jLogger [] - Slf4jLogger started
3041 [flink-pekko.actor.default-dispatcher-5] INFO  
org.apache.pekko.remote.RemoteActorRefProvider [] - Pekko Cluster not in use - 
enabling unsafe features anyway because 
`pekko.remote.use-unsafe-remote-features-outside-cluster` has been enabled.
3042 [flink-pekko.actor.default-dispatcher-5] INFO  
org.apache.pekko.remote.Remoting [] - Starting remoting
3218 [flink-pekko.actor.default-dispatcher-5] INFO  
org.apache.pekko.remote.Remoting [] - Remoting started; listening on addresses 
:[pekko.tcp://flink@localhost:57012]
3534 [ForkJoinPool-1-worker-1] INFO  
org.apache.flink.runtime.rpc.pekko.PekkoRpcServiceUtils [] - Actor system 
started at pekko.tcp://flink@localhost:57012
3571 [ForkJoinPool-1-worker-1] INFO  
org.apache.flink.runtime.rpc.pekko.PekkoRpcServiceUtils [] - Trying to start 
actor system, external address localhost:0, bind address 0.0.0.0:0.
3610 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.event.slf4j.Slf4jLogger [] - Slf4jLogger started
3616 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.remote.RemoteActorRefProvider [] - Pekko Cluster not in use - 
enabling unsafe features anyway because 
`pekko.remote.use-unsafe-remote-features-outside-cluster` has been enabled.
3617 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.remote.Remoting [] - Starting remoting
3627 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.remote.Remoting [] - Remoting started; listening on addresses 
:[pekko.tcp://flink@localhost:57013]
3665 [ForkJoinPool-1-worker-1] INFO  
org.apache.flink.runtime.rpc.pekko.PekkoRpcServiceUtils [] - Actor system 
started at pekko.tcp://flink@localhost:57013
3690 [ForkJoinPool-1-worker-1] INFO  org.apache.flink.util.TestLoggerExtension 
[] - 
================================================================================
Test 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable[failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable()]
 is running.
--------------------------------------------------------------------------------
3696 [ForkJoinPool-1-worker-1] INFO  
org.apache.flink.runtime.rpc.pekko.PekkoRpcServiceUtils [] - Trying to start 
actor system, external address localhost:0, bind address 0.0.0.0:0.
3726 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.event.slf4j.Slf4jLogger [] - Slf4jLogger started
3730 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.remote.RemoteActorRefProvider [] - Pekko Cluster not in use - 
enabling unsafe features anyway because 
`pekko.remote.use-unsafe-remote-features-outside-cluster` has been enabled.
3730 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.remote.Remoting [] - Starting remoting
3746 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.remote.Remoting [] - Remoting started; listening on addresses 
:[pekko.tcp://flink@localhost:57014]
3789 [ForkJoinPool-1-worker-1] INFO  
org.apache.flink.runtime.rpc.pekko.PekkoRpcServiceUtils [] - Actor system 
started at pekko.tcp://flink@localhost:57014
3831 [ForkJoinPool-1-worker-1] INFO  
org.apache.flink.runtime.rpc.pekko.PekkoRpcService [] - Starting RPC endpoint 
for 
org.apache.flink.runtime.rpc.pekko.PekkoRpcActorTest$SerializedValueRespondingEndpoint
 at pekko://flink/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5 .
8669 [ForkJoinPool-1-worker-1] INFO  
org.apache.flink.runtime.rpc.pekko.PekkoRpcService [] - Stopping Pekko RPC 
service.
8772 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.actor.CoordinatedShutdown [] - Running CoordinatedShutdown 
with reason [ActorSystemTerminateReason]
8801 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.remote.RemoteActorRefProvider$RemotingTerminator [] - Shutting 
down remote daemon.
8801 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.remote.RemoteActorRefProvider$RemotingTerminator [] - Remote 
daemon shut down; proceeding with flushing remote transports.
8864 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.remote.RemoteActorRefProvider$RemotingTerminator [] - Remoting 
shut down.
8892 [flink-pekko.actor.default-dispatcher-20] INFO  
org.apache.flink.runtime.rpc.pekko.PekkoRpcService [] - Stopped Pekko RPC 
service.
22759 [flink-pekko.actor.default-dispatcher-6] WARN  
org.apache.pekko.remote.transport.netty.NettyTransport [] - Remote connection 
to [null] failed with java.net.ConnectException: Connection refused: 
localhost/127.0.0.1:57014
22760 [flink-pekko.actor.default-dispatcher-6] WARN  
org.apache.pekko.remote.ReliableDeliverySupervisor [] - Association with remote 
system [pekko.tcp://flink@localhost:57014] has failed, address is now gated for 
[50] ms. Reason: [Association failed with [pekko.tcp://flink@localhost:57014]] 
Caused by: [java.net.ConnectException: Connection refused: 
localhost/127.0.0.1:57014]
22829 [ForkJoinPool-1-worker-1] ERROR org.apache.flink.util.TestLoggerExtension 
[] - 
--------------------------------------------------------------------------------
Test 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable[failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable()]
 failed with:
org.assertj.core.error.AssertJMultipleFailuresError: 
Multiple Failures (1 failure)
-- failure 1 --
[Any cause is instance of class 'class 
org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException'] 
Expecting any element of:
  [java.util.concurrent.CompletionException: 
java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
        at 
java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        ...(63 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed),
    java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        at 
org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
        at 
org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:892)
        ...(61 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed),
    org.apache.pekko.pattern.AskTimeoutException: Ask timed out on 
[Actor[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5#314813027]]
 after [10000 ms]. Message of type 
[org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation]. A typical reason 
for `AskTimeoutException` is that the recipient actor didn't send a reply.
]
to satisfy the given assertions requirements but none did:

java.util.concurrent.CompletionException: 
java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
        at 
java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        ...(63 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed)
error: 
Expecting actual throwable to be an instance of:
  org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException
but was:
  java.util.concurrent.CompletionException: 
java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
        at 
java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        ...(63 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed)

java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        at 
org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
        at 
org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:892)
        ...(61 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed)
error: 
Expecting actual throwable to be an instance of:
  org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException
but was:
  java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        at 
org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
        at 
org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:892)
        ...(61 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed)

org.apache.pekko.pattern.AskTimeoutException: Ask timed out on 
[Actor[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5#314813027]]
 after [10000 ms]. Message of type 
[org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation]. A typical reason 
for `AskTimeoutException` is that the recipient actor didn't send a reply.

error: 
Expecting actual throwable to be an instance of:
  org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException
but was:
  org.apache.pekko.pattern.AskTimeoutException: Ask timed out on 
[Actor[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5#314813027]]
 after [10000 ms]. Message of type 
[org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation]. A typical reason 
for `AskTimeoutException` is that the recipient actor didn't send a reply.
at FlinkAssertions.lambda$anyCauseMatches$3(FlinkAssertions.java:89)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable(RemotePekkoRpcActorTest.java:169)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
        at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
        at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
        at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
        at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
        at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
        at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
        at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
        at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
        at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
        at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
        at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
        at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
        at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
        at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
        at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
        at 
java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)
        at 
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703)
        at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)

================================================================================

java.lang.AssertionError: [Any cause is instance of class 'class 
org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException'] 
Expecting any element of:
  [java.util.concurrent.CompletionException: 
java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
        at 
java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        ...(63 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed),
    java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        at 
org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
        at 
org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:892)
        ...(61 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed),
    org.apache.pekko.pattern.AskTimeoutException: Ask timed out on 
[Actor[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5#314813027]]
 after [10000 ms]. Message of type 
[org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation]. A typical reason 
for `AskTimeoutException` is that the recipient actor didn't send a reply.
]
to satisfy the given assertions requirements but none did:

java.util.concurrent.CompletionException: 
java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
        at 
java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        ...(63 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed)
error: 
Expecting actual throwable to be an instance of:
  org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException
but was:
  java.util.concurrent.CompletionException: 
java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
        at 
java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        ...(63 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed)

java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        at 
org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
        at 
org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:892)
        ...(61 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed)
error: 
Expecting actual throwable to be an instance of:
  org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException
but was:
  java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        at 
org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
        at 
org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:892)
        ...(61 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed)

org.apache.pekko.pattern.AskTimeoutException: Ask timed out on 
[Actor[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5#314813027]]
 after [10000 ms]. Message of type 
[org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation]. A typical reason 
for `AskTimeoutException` is that the recipient actor didn't send a reply.

error: 
Expecting actual throwable to be an instance of:
  org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException
but was:
  org.apache.pekko.pattern.AskTimeoutException: Ask timed out on 
[Actor[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5#314813027]]
 after [10000 ms]. Message of type 
[org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation]. A typical reason 
for `AskTimeoutException` is that the recipient actor didn't send a reply.


        at 
org.apache.flink.core.testutils.FlinkAssertions.lambda$anyCauseMatches$3(FlinkAssertions.java:89)
        at 
java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
        at 
java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
        at 
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
        at 
java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
        at 
java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable(RemotePekkoRpcActorTest.java:169)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
        at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
        at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
        at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
        at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
        at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
        at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
        at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
        at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
        at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
        at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
        at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
        at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
        at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
        at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
        at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
        at 
java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)
        at 
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703)
        at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)


org.assertj.core.error.AssertJMultipleFailuresError: 
Multiple Failures (1 failure)
-- failure 1 --
[Any cause is instance of class 'class 
org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException'] 
Expecting any element of:
  [java.util.concurrent.CompletionException: 
java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
        at 
java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        ...(63 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed),
    java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        at 
org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
        at 
org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:892)
        ...(61 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed),
    org.apache.pekko.pattern.AskTimeoutException: Ask timed out on 
[Actor[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5#314813027]]
 after [10000 ms]. Message of type 
[org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation]. A typical reason 
for `AskTimeoutException` is that the recipient actor didn't send a reply.
]
to satisfy the given assertions requirements but none did:

java.util.concurrent.CompletionException: 
java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
        at 
java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        ...(63 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed)
error: 
Expecting actual throwable to be an instance of:
  org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException
but was:
  java.util.concurrent.CompletionException: 
java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
        at 
java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        ...(63 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed)

java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        at 
org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
        at 
org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:892)
        ...(61 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed)
error: 
Expecting actual throwable to be an instance of:
  org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException
but was:
  java.util.concurrent.TimeoutException: Invocation of 
[RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] at 
recipient 
[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5]
 timed out. This is usually caused by: 1) Pekko failed sending the message 
silently, due to problems like oversized payload or serialization failures. In 
that case, you should find detailed error information in the logs. 2) The 
recipient needs more time for responding, due to problems like slow machines or 
network jitters. In that case, you can try to increase pekko.ask.timeout.
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
        at 
org.assertj.core.api.ThrowableAssert.catchThrowable(ThrowableAssert.java:63)
        at 
org.assertj.core.api.AssertionsForClassTypes.catchThrowable(AssertionsForClassTypes.java:892)
        ...(61 remaining lines not displayed - this can be changed with 
Assertions.setMaxStackTraceElementsDisplayed)

org.apache.pekko.pattern.AskTimeoutException: Ask timed out on 
[Actor[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5#314813027]]
 after [10000 ms]. Message of type 
[org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation]. A typical reason 
for `AskTimeoutException` is that the recipient actor didn't send a reply.

error: 
Expecting actual throwable to be an instance of:
  org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException
but was:
  org.apache.pekko.pattern.AskTimeoutException: Ask timed out on 
[Actor[pekko.tcp://flink@localhost:57014/user/rpc/079e1e41-10aa-4e64-95fb-1ce2a21c11b5#314813027]]
 after [10000 ms]. Message of type 
[org.apache.flink.runtime.rpc.messages.RemoteRpcInvocation]. A typical reason 
for `AskTimeoutException` is that the recipient actor didn't send a reply.
at FlinkAssertions.lambda$anyCauseMatches$3(FlinkAssertions.java:89)

        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at 
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at 
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at 
org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable(RemotePekkoRpcActorTest.java:169)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at 
org.junit.platform.commons.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:727)
        at 
org.junit.jupiter.engine.execution.MethodInvocation.proceed(MethodInvocation.java:60)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$ValidatingInvocation.proceed(InvocationInterceptorChain.java:131)
        at 
org.junit.jupiter.engine.extension.TimeoutExtension.intercept(TimeoutExtension.java:156)
        at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestableMethod(TimeoutExtension.java:147)
        at 
org.junit.jupiter.engine.extension.TimeoutExtension.interceptTestMethod(TimeoutExtension.java:86)
        at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker$ReflectiveInterceptorCall.lambda$ofVoidMethod$0(InterceptingExecutableInvoker.java:103)
        at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.lambda$invoke$0(InterceptingExecutableInvoker.java:93)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain$InterceptedInvocation.proceed(InvocationInterceptorChain.java:106)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.proceed(InvocationInterceptorChain.java:64)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.chainAndInvoke(InvocationInterceptorChain.java:45)
        at 
org.junit.jupiter.engine.execution.InvocationInterceptorChain.invoke(InvocationInterceptorChain.java:37)
        at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:92)
        at 
org.junit.jupiter.engine.execution.InterceptingExecutableInvoker.invoke(InterceptingExecutableInvoker.java:86)
        at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeTestMethod$7(TestMethodTestDescriptor.java:217)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeTestMethod(TestMethodTestDescriptor.java:213)
        at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:138)
        at 
org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:68)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
        at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
        at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService.invokeAll(ForkJoinPoolHierarchicalTestExecutorService.java:129)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
        at 
org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
        at 
org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
        at 
org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
        at 
org.junit.platform.engine.support.hierarchical.ForkJoinPoolHierarchicalTestExecutorService$ExclusiveTask.compute(ForkJoinPoolHierarchicalTestExecutorService.java:185)
        at java.util.concurrent.RecursiveAction.exec(RecursiveAction.java:189)
        at 
java.util.concurrent.ForkJoinTask.doExec$$$capture(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java)
        at 
java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1067)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1703)
        at 
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:172)

22850 [ForkJoinPool-1-worker-1] INFO  
org.apache.flink.runtime.rpc.pekko.PekkoRpcService [] - Stopping Pekko RPC 
service.
22850 [ForkJoinPool-1-worker-1] INFO  
org.apache.flink.runtime.rpc.pekko.PekkoRpcService [] - Stopping Pekko RPC 
service.
22869 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.actor.CoordinatedShutdown [] - Running CoordinatedShutdown 
with reason [ActorSystemTerminateReason]
22869 [flink-pekko.actor.default-dispatcher-5] INFO  
org.apache.pekko.actor.CoordinatedShutdown [] - Running CoordinatedShutdown 
with reason [ActorSystemTerminateReason]
22885 [flink-pekko.actor.default-dispatcher-5] INFO  
org.apache.pekko.remote.RemoteActorRefProvider$RemotingTerminator [] - Shutting 
down remote daemon.
22885 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.remote.RemoteActorRefProvider$RemotingTerminator [] - Shutting 
down remote daemon.
22893 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.remote.RemoteActorRefProvider$RemotingTerminator [] - Remote 
daemon shut down; proceeding with flushing remote transports.
22893 [flink-pekko.actor.default-dispatcher-5] INFO  
org.apache.pekko.remote.RemoteActorRefProvider$RemotingTerminator [] - Remote 
daemon shut down; proceeding with flushing remote transports.
22902 [flink-pekko.actor.default-dispatcher-5] INFO  
org.apache.pekko.remote.RemoteActorRefProvider$RemotingTerminator [] - Remoting 
shut down.
22903 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.pekko.remote.RemoteActorRefProvider$RemotingTerminator [] - Remoting 
shut down.
22913 [flink-pekko.actor.default-dispatcher-5] INFO  
org.apache.flink.runtime.rpc.pekko.PekkoRpcService [] - Stopped Pekko RPC 
service.
22920 [flink-pekko.actor.default-dispatcher-6] INFO  
org.apache.flink.runtime.rpc.pekko.PekkoRpcService [] - Stopped Pekko RPC 
service.
public String getFoo()
{
    return foo;
}
{code}


> RemotePekkoRpcActorTest.failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable
>  fails on AZP
> ----------------------------------------------------------------------------------------------
>
>                 Key: FLINK-33278
>                 URL: https://issues.apache.org/jira/browse/FLINK-33278
>             Project: Flink
>          Issue Type: Bug
>          Components: Runtime / RPC
>    Affects Versions: 1.19.0
>            Reporter: Sergey Nuyanzin
>            Priority: Critical
>              Labels: test-stability
>         Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png, 
> screenshot-4.png
>
>
> This build 
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=53740&view=logs&j=0e7be18f-84f2-53f0-a32d-4a5e4a174679&t=7c1d86e3-35bd-5fd5-3b7c-30c126a78702&l=6563]
> fails as
> {noformat}
> Oct 15 01:02:20 Multiple Failures (1 failure)
> Oct 15 01:02:20 -- failure 1 --
> Oct 15 01:02:20 [Any cause is instance of class 'class 
> org.apache.flink.runtime.rpc.exceptions.RecipientUnreachableException'] 
> Oct 15 01:02:20 Expecting any element of:
> Oct 15 01:02:20   [java.util.concurrent.CompletionException: 
> java.util.concurrent.TimeoutException: Invocation of 
> [RemoteRpcInvocation(SerializedValueRespondingGateway.getSerializedValue())] 
> at recipient 
> [pekko.tcp://flink@localhost:38231/user/rpc/8c211f34-41e5-4efe-93bd-8eca6c590a7f]
>  timed out. This is usually caused by: 1) Pekko failed sending the message 
> silently, due to problems like oversized payload or serialization failures. 
> In that case, you should find detailed error information in the logs. 2) The 
> recipient needs more time for responding, due to problems like slow machines 
> or network jitters. In that case, you can try to increase pekko.ask.timeout.
> Oct 15 01:02:20       at 
> java.util.concurrent.CompletableFuture.reportJoin(CompletableFuture.java:375)
> Oct 15 01:02:20       at 
> java.util.concurrent.CompletableFuture.join(CompletableFuture.java:1947)
> Oct 15 01:02:20       at 
> org.apache.flink.runtime.rpc.pekko.RemotePekkoRpcActorTest.lambda$failsRpcResultImmediatelyIfRemoteRpcServiceIsNotAvailable$1(RemotePekkoRpcActorTest.java:168)
> Oct 15 01:02:20       ...(63 remaining lines not displayed - this can be 
> changed with Assertions.setMaxStackTraceElementsDisplayed),
> ...
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Reply via email to