Yu Yang created FLINK-16429:
-------------------------------
Summary: failed to restore flink job from checkpoints due to
unhandled exceptions
Key: FLINK-16429
URL: https://issues.apache.org/jira/browse/FLINK-16429
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.9.1
Reporter: Yu Yang
We are trying to restore our flink job from check-points, and run into
AskTimeoutException related failures at a high frequency. Our environment is
Hadoop 2.7.1 + Yarn + Flink 1.9.1.
We hit this issue in 9 out of 10 runs, and were able to restore the application
from given check-points from time to time. As the application can be restored,
the check-point files shall not be corrupted. It seems that the issue is that
jobmaster got timeout when it handles job submission request.
Below is the exception stack trace, it is thrown from
[https://github.com/apache/flink/blob/2ec645a5bfd3cfadaf0057412401e91da0b21873/flink-runtime/src/main/java/org/apache/flink/runtime/rest/handler/AbstractHandler.java#L209]
2020-03-05 00:04:14,360 ERROR
org.apache.flink.runtime.rest.handler.job.JobSubmitHandler - Unhandled
exception: httpRequest uri:/v1/jobs, context:
ChannelHandlerContext(org.apache.flink.runtime.rest.handler.router.RouterHandler_ROUTED_HANDLER,
[id: 0xc39aca33, L:/10.1.85.22:41000 - R:/10.1.16.251:44])
akka.pattern.AskTimeoutException: Ask timed out on
[Actor[akka://flink/user/dispatcher#-34498396]] after [10000 ms]. Message of
type [org.apache.flink.runtime.rpc.messages.LocalFencedMessage]. A typical
reason for `AskTimeoutException` is that the recipient actor didn't send a
reply. at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635)
at akka.pattern.PromiseActorRef$$anonfun$2.apply(AskSupport.scala:635) at
akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:648) at
akka.actor.Scheduler$$anon$4.run(Scheduler.scala:205) at
scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
at
akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:328)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:279)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:283)
at
akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:235)
at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
