hi, Xintong：
我仔细查看了下日志，发现在报错"Could not resolve ResourceManager address"之前有如下日志：
2020-08-22 05:39:24,473 INFO org.apache.flink.runtime.jobmaster.JobMaster
- The heartbeat of ResourceManager with id
6724e1ef8ee1c5fe5212eec6182319b6 timed out.
2020-08-22 05:39:24,473 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Close ResourceManager connection
6724e1ef8ee1c5fe5212eec6182319b6: The heartbeat of ResourceManager with id
6724e1ef8ee1c5fe5212eec6182319b6 timed out..
之后就一直报错 "Could not resolve ResourceManager address" 了，
看了下flink 1.9.0 版本的代码，是在rpcService.connect() 时报的错，
可是之后就没有日志输出了，单从报错信息来看只是说无法解析地址，
可是resourcemanager地址是没有问题的。
请问有没有办法可以查看resourcemanager的健康状况呢？
// 代码：
package org.apache.flink.runtime.registration；
public abstract class RetryingRegistration<F extends Serializable, G
extends RpcGateway, S extends RegistrationResponse.Success> {
public void startRegistration() {
...
if (FencedRpcGateway.class.isAssignableFrom(targetType)) {
rpcGatewayFuture = (CompletableFuture<G>) rpcService.connect(
targetAddress,
fencingToken,
targetType.asSubclass(FencedRpcGateway.class));
} else {
// 连接resourcemanager
rpcGatewayFuture = rpcService.connect(targetAddress,
targetType);
}
...
rpcGatewayAcceptFuture.whenCompleteAsync(
(Void v, Throwable failure) -> {
if (failure != null && !canceled) {
final Throwable strippedFailure =
ExceptionUtils.stripCompletionException(failure);
if (log.isDebugEnabled()) {
...
} else {
// 报错
log.info(
"Could not resolve {} address {}, retrying in {}
ms:
{}.",targetName,targetAddress,retryingRegistrationConfiguration.getErrorDelayMillis(),strippedFailure.getMessage());
}
// 重连
startRegistrationLater(retryingRegistrationConfiguration.getErrorDelayMillis());
}
},
rpcService.getExecutor());
}
}
==================================================================================================================
以下是一次提交job的完成报错日志
==================================================================================================================
2020-08-25 16:02:55,737 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Received
JobGraph submission 4a324bc1e1eeb964116686e568cea8ad (Streaming WordCount).
2020-08-25 16:02:55,738 INFO
org.apache.flink.runtime.dispatcher.StandaloneDispatcher - Submitting
job 4a324bc1e1eeb964116686e568cea8ad (Streaming WordCount).
2020-08-25 16:02:56,552 INFO
org.apache.flink.runtime.jobmanager.ZooKeeperSubmittedJobGraphStore -
Added SubmittedJobGraph(4a324bc1e1eeb964116686e568cea8ad) to ZooKeeper.
2020-08-25 16:02:56,554 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcService - Starting
RPC endpoint for org.apache.flink.runtime.jobmaster.JobMaster at
akka://flink/user/jobmanager_42 .
2020-08-25 16:02:56,554 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Initializing job Streaming WordCount
(4a324bc1e1eeb964116686e568cea8ad).
2020-08-25 16:02:56,555 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Using restart strategy
FailureRateRestartStrategy(failuresInterval=300000 msdelayInterval=10000
msmaxFailuresPerInterval=10) for Streaming WordCount
(4a324bc1e1eeb964116686e568cea8ad).
2020-08-25 16:02:56,555 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Job
recovers via failover strategy: New Pipelined Region Failover
2020-08-25 16:02:56,555 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Running initialization on master for job Streaming
WordCount (4a324bc1e1eeb964116686e568cea8ad).
2020-08-25 16:02:56,555 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Successfully ran initialization on master in 0 ms.
2020-08-25 16:02:56,556 INFO
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy
- Start building failover regions.
2020-08-25 16:02:56,556 INFO
org.apache.flink.runtime.executiongraph.failover.flip1.RestartPipelinedRegionStrategy
- Created 1 failover regions.
2020-08-25 16:02:56,559 INFO org.apache.flink.runtime.util.ZooKeeperUtils
- Initialized ZooKeeperCompletedCheckpointStore in
'/checkpoints/4a324bc1e1eeb964116686e568cea8ad'.
2020-08-25 16:02:56,560 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Loading state backend via factory
org.apache.flink.contrib.streaming.state.RocksDBStateBackendFactory
2020-08-25 16:02:56,560 INFO
org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Using
predefined options: DEFAULT.
2020-08-25 16:02:56,560 INFO
org.apache.flink.contrib.streaming.state.RocksDBStateBackend - Using
default options factory:
DefaultConfigurableOptionsFactory{configuredOptions={}}.
2020-08-25 16:02:56,609 INFO
org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
Recovering checkpoints from ZooKeeper.
2020-08-25 16:02:56,611 INFO
org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
Found 0 checkpoints in ZooKeeper.
2020-08-25 16:02:56,611 INFO
org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
Trying to fetch 0 checkpoints from storage.
2020-08-25 16:02:56,611 INFO
org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionService -
Starting ZooKeeperLeaderElectionService
ZooKeeperLeaderElectionService{leaderPath='/leader/4a324bc1e1eeb964116686e568cea8ad/job_manager_lock'}.
2020-08-25 16:02:56,620 INFO
org.apache.flink.runtime.jobmaster.JobManagerRunner - JobManager
runner for job Streaming WordCount (4a324bc1e1eeb964116686e568cea8ad) was
granted leadership with session id 5843c4ec-82d7-441f-befb-e4757c34f3f2 at
akka.tcp://flink@hostname:16098/user/jobmanager_42.
2020-08-25 16:02:56,623 INFO
org.apache.flink.runtime.leaderretrieval.ZooKeeperLeaderRetrievalService
- Starting ZooKeeperLeaderRetrievalService /leader/resource_manager_lock.
2020-08-25 16:02:56,623 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Starting execution of job Streaming WordCount
(4a324bc1e1eeb964116686e568cea8ad) under job master id
befbe4757c34f3f25843c4ec82d7441f.
2020-08-25 16:02:56,623 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Job
Streaming WordCount (4a324bc1e1eeb964116686e568cea8ad) switched from state
CREATED to RUNNING.
2020-08-25 16:02:56,623 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Source:
Collection Source (1/1) (2b6a86725fe87a56e8706654205ac74c) switched from
CREATED to SCHEDULED.
2020-08-25 16:02:56,623 INFO
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl - Cannot
serve slot request, no ResourceManager connected. Adding as pending request
[SlotRequestId{a32583aa56cf8fdfd6c9b42fb2b60c6f}]
2020-08-25 16:02:56,623 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map
(1/4) (896ecc39967bd803dca4656262f0d499) switched from CREATED to SCHEDULED.
2020-08-25 16:02:56,623 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map
(2/4) (081e8d6d45fc1d8b0c0311341039f03e) switched from CREATED to SCHEDULED.
2020-08-25 16:02:56,623 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map
(3/4) (b02a3b8db2b07b4e54d6129d43189e17) switched from CREATED to SCHEDULED.
2020-08-25 16:02:56,623 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Flat Map
(4/4) (c011b9d3544b434cf56dbd454a309644) switched from CREATED to SCHEDULED.
2020-08-25 16:02:56,623 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Keyed
Aggregation -> Sink: Print to Std. Out (1/4)
(b515b0f4a11f7e906f7258211492d8d5) switched from CREATED to SCHEDULED.
2020-08-25 16:02:56,623 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Keyed
Aggregation -> Sink: Print to Std. Out (2/4)
(78b55fa685e6f9ac0db41627e95789a7) switched from CREATED to SCHEDULED.
2020-08-25 16:02:56,623 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Keyed
Aggregation -> Sink: Print to Std. Out (3/4)
(cafa28075572c7cea6dce65fef14e110) switched from CREATED to SCHEDULED.
2020-08-25 16:02:56,623 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Keyed
Aggregation -> Sink: Print to Std. Out (4/4)
(6914cb284564f53914aa2a14b9a9f667) switched from CREATED to SCHEDULED.
2020-08-25 16:02:56,625 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Connecting to ResourceManager akka.tcp://flink@hostname
:16098/user/resourcemanager(869831904fde985ece020bb52023471f)
2020-08-25 16:02:57,483 ERROR
org.apache.flink.runtime.rest.handler.cluster.ClusterOverviewHandler -
Unhandled exception.
akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka://flink/user/dispatcher#-1681919171]] after [10000 ms]. Message
of type [org.apache.flink.runtime.rpc.messages.LocalFencedMessage]. A
typical reason for `AskTimeoutException` is that the recipient actor didn't
send a reply.
at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
at
akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:648)
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
at
scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
at java.lang.Thread.run(Thread.java:748)
2020-08-25 16:03:03,723 ERROR
org.apache.flink.runtime.rest.handler.cluster.ClusterOverviewHandler -
Unhandled exception.
akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka://flink/user/dispatcher#-1681919171]] after [10000 ms]. Message
of type [org.apache.flink.runtime.rpc.messages.LocalFencedMessage]. A
typical reason for `AskTimeoutException` is that the recipient actor didn't
send a reply.
at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
at
akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:648)
at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205)
at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
at
scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at
scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
at java.lang.Thread.run(Thread.java:748)
2020-08-25 16:03:06,646 INFO org.apache.flink.runtime.jobmaster.JobMaster
- Could not resolve ResourceManager address
akka.tcp://flink@hostname:16098/user/resourcemanager, retrying in 10000 ms:
Ask timed out on [ActorSelection[Anchor(akka://flink/),
Path(/user/resourcemanager)]] after [10000 ms]. Message of type
[akka.actor.Identify]. A typical reason for `AskTimeoutException` is that
the recipient actor didn't send a reply..
==================================================================================================================
Xintong Song <[email protected]> 于2020年8月25日周二 上午10:55写道：
> 按你的描述，我理解是在 yarn 上起了一个 flink session，然后往这个 session
> 里提交作业，一开始能正常提交运行，后来再作业开始出现问题？
>
> 具体问题的原因还是要看日志才能确定，这个有可能是 ResourceManager，也有可能是 HA 或者 Akka 的问题，仅凭现象描述很难判断。
>
>
> Thank you~
>
> Xintong Song
>
>
>
> On Tue, Aug 25, 2020 at 10:23 AM song wang <[email protected]>
> wrote:
>
> >
> >
> 你好，这个报错是任务启动了很长一段时间后才发生的，之前一直正常运行，出现这个报错后就提交不了任务了，感觉是ResourceManager出现了问题，ResourceManager有可能自己死掉吗？
> > 另外，这个现场还没有关掉，有什么办法排查下jobmaster和ResourceManager是否正常吗？
> >
> > Xintong Song <[email protected]> 于2020年8月25日周二 上午9:46写道：
> >
> > > 从日志上看是 JobMaster 连不上 ResourceManager。这两个组件应该是在同一个进程内的，通过 akka 进行本地通信。
> > > 需要看下完整的日志，RM 是否成功启动并注册到了 akka 的 actor system，以及注册的路径是否和 JM 尝试连接的路径一致。
> > >
> > >
> > > Thank you~
> > >
> > > Xintong Song
> > >
> > >
> > >
> > > On Mon, Aug 24, 2020 at 3:41 PM song wang <[email protected]>
> > > wrote:
> > >
> > > > 各位老哥， flink
> > > > 运行在yarn上，偶尔报错无法解析ResourceManager地址，可是从对应的host上查找是有flink进程的，请问是什么原因呢？
> > > > flink 版本1.9.0
> > > >
> > > > 部分日志如下：
> > > > ```
> > > > 2020-08-24 15:11:31,566 INFO
> > > org.apache.flink.runtime.jobmaster.JobMaster
> > > > - Could not resolve ResourceManager address
> > > > akka.tcp://flink@hostname:16098/user/resourcemanager, retrying in
> > 10000
> > > > ms:
> > > > Ask timed out on [ActorSelection[Anchor(akka://flink/),
> > > > Path(/user/resourcemanager)]] after [10000 ms]. Message of type
> > > > [akka.actor.Identify]. A typical reason for `AskTimeoutException` is
> > that
> > > > the recipient actor didn't send a reply..
> > > > 2020-08-24 15:11:51,606 INFO
> > > org.apache.flink.runtime.jobmaster.JobMaster
> > > > - Could not resolve ResourceManager address
> > > > akka.tcp://flink@hostname:16098/user/resourcemanager, retrying in
> > 10000
> > > > ms:
> > > > Ask timed out on [ActorSelection[Anchor(akka://flink/),
> > > > Path(/user/resourcemanager)]] after [10000 ms]. Message of type
> > > > [akka.actor.Identify]. A typical reason for `AskTimeoutException` is
> > that
> > > > the recipient actor didn't send a reply..
> > > > 2020-08-24 15:12:11,645 INFO
> > > org.apache.flink.runtime.jobmaster.JobMaster
> > > > - Could not resolve ResourceManager address
> > > > akka.tcp://flink@hostname:16098/user/resourcemanager, retrying in
> > 10000
> > > > ms:
> > > > Ask timed out on [ActorSelection[Anchor(akka://flink/),
> > > > Path(/user/resourcemanager)]] after [10000 ms]. Message of type
> > > > [akka.actor.Identify]. A typical reason for `AskTimeoutException` is
> > that
> > > > the recipient actor didn't send a reply..
> > > > 2020-08-24 15:12:31,687 INFO
> > > org.apache.flink.runtime.jobmaster.JobMaster
> > > > - Could not resolve ResourceManager address
> > > > akka.tcp://flink@hostname:16098/user/resourcemanager, retrying in
> > 10000
> > > > ms:
> > > > Ask timed out on [ActorSelection[Anchor(akka://flink/),
> > > > Path(/user/resourcemanager)]] after [10000 ms]. Message of type
> > > > [akka.actor.Identify]. A typical reason for `AskTimeoutException` is
> > that
> > > > the recipient actor didn't send a reply..
> > > > 2020-08-24 15:12:51,727 INFO
> > > org.apache.flink.runtime.jobmaster.JobMaster
> > > > - Could not resolve ResourceManager address
> > > > akka.tcp://flink@hostname:16098/user/resourcemanager, retrying in
> > 10000
> > > > ms:
> > > > Ask timed out on [ActorSelection[Anchor(akka://flink/),
> > > > Path(/user/resourcemanager)]] after [10000 ms]. Message of type
> > > > [akka.actor.Identify]. A typical reason for `AskTimeoutException` is
> > that
> > > > the recipient actor didn't send a reply..
> > > > 2020-08-24 15:13:08,198 INFO
> > > > org.apache.flink.runtime.executiongraph.ExecutionGraph - Job
> > > > Streaming WordCount (ff0ab7ec3e577a8e0c69e1c8454e5b72) switched from
> > > state
> > > > RUNNING to FAILING.
> > > >
> > >
> >
> org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
> > > > Could not allocate all requires slots within timeout of 300000 ms.
> > Slots
> > > > required: 9, slots allocated: 0, previous allocation IDs: [],
> execution
> > > > status: completed exceptionally:
> > > java.util.concurrent.CompletionException:
> > > > java.util.concurrent.CompletionException:
> > > >
> > > >
> > >
> >
> java.util.concurrent.TimeoutException/java.util.concurrent.CompletableFuture@28d7ad5d
> > > > [Completed
> > > > exceptionally], incomplete:
> > > java.util.concurrent.CompletableFuture@663cdf7e
> > > > [Not
> > > > completed, 1 dependents], incomplete:
> > > > java.util.concurrent.CompletableFuture@2058a7e9[Not completed, 1
> > > > dependents], incomplete:
> > java.util.concurrent.CompletableFuture@5c1121c8
> > > > [Not
> > > > completed, 1 dependents], incomplete:
> > > > java.util.concurrent.CompletableFuture@49b9c252[Not completed, 1
> > > > dependents], incomplete:
> > java.util.concurrent.CompletableFuture@497e3334
> > > > [Not
> > > > completed, 1 dependents], incomplete:
> > > > java.util.concurrent.CompletableFuture@2c7ca21d[Not completed, 1
> > > > dependents], incomplete:
> > java.util.concurrent.CompletableFuture@7936c93b
> > > > [Not
> > > > completed, 1 dependents], incomplete:
> > > > java.util.concurrent.CompletableFuture@7e9a2f1d[Not completed, 1
> > > > dependents]
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.runtime.executiongraph.SchedulingUtils.lambda$scheduleEager$1(SchedulingUtils.java:194)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.runtime.concurrent.FutureUtils$ResultConjunctFuture.handleCompletedFuture(FutureUtils.java:633)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.runtime.concurrent.FutureUtils$ResultConjunctFuture.lambda$new$0(FutureUtils.java:656)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:190)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:700)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:484)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:380)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:998)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:397)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:190)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
> > > > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> > > > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> > > > at
> > scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> > > > at akka.japi.pf
> > > .UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> > > > at
> > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> > > > at
> > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> > > > at
> > scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> > > > at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> > > > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> > > > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> > > > at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> > > > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> > > > at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> > > > at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> > > > at
> akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> > > > at
> > > >
> > > >
> > >
> >
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> > > > at
> > > akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> > > > at
> > > >
> > > >
> > >
> >
> akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
> > > > ```
> > > >
> > >
> >
>
