[
https://issues.apache.org/jira/browse/FLINK-16429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17068798#comment-17068798
]
Till Rohrmann commented on FLINK-16429:
---------------------------------------
[~yuyang08] could you provide us with the logs of the {{Dispatcher}}? I agree
with Stephan that it looks very similar to FLINK-16018.
> failed to restore flink job from checkpoints due to unhandled exceptions
> ------------------------------------------------------------------------
>
> Key: FLINK-16429
> URL: https://issues.apache.org/jira/browse/FLINK-16429
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.9.1
> Reporter: Yu Yang
> Priority: Major
>
> We are trying to restore our flink job from check-points, and run into
> AskTimeoutException related failures at a high frequency. Our environment is
> Hadoop 2.7.1 + Yarn + Flink 1.9.1.
> We hit this issue in 9 out of 10 runs, and were able to restore the
> application from given check-points from time to time. As the application can
> be restored, the check-point files shall not be corrupted. It seems that the
> issue is that jobmaster got timeout when it handles job submission request.
>
> Below is the exception stack trace, it is thrown from
> [https://github.com/apache/flink/blob/2ec645a5bfd3cfadaf0057412401e91da0b21873/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/AbstractHandler.java#L209]
> 2020-03-05 00:04:14,360 ERROR
> org.apache.flink.runtime.rest.handler.job.JobSubmitHandler - Unhandled
> exception: httpRequest uri:/v1/jobs, context:
> ChannelHandlerContext(org.apache.flink.runtime.rest.handler.router.RouterHandler_ROUTED_HANDLER,
> [id: 0xc39aca33, L:/10.1.85.22:41000 - R:/10.1.16.251:44])
> akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka://flink/user/dispatcher#-34498396]] after [10000 ms]. Message of
> type [org.apache.flink.runtime.rpc.messages.LocalFencedMessage]. A typical
> reason for `AskTimeoutException` is that the recipient actor didn't send a
> reply. at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635
> at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635
> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:648
> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205
> at
> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601
> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109
> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599
> at
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235
> at java.lang.Thread.run(Thread.java:748 undefined)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
