[
https://issues.apache.org/jira/browse/FLINK-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896372#comment-16896372
]
Yun Tang commented on FLINK-13452:
----------------------------------
Before FLINK-13060, the process to restart tasks within
{{AdaptedRestartPipelinedRegionStrategyNG}} is:
{code:java}
cancelTasks --> resetTasks --> handleExcpetion
{code}
After FLINK-13060, the process changed to:
{code:java}
resetTasks
|^|
| |
cancelTasks --> strategy restart --> handleExcpetion
{code}
The bug happened due to the future task of {{resetTasks}} failed but die
silently. And as you can see, this bug has no direct relationship between
region failover to load checkpointed state.
Previously I try to catch any exception within {{resetTasks}} itself to
{{failGlobal}} in my [PR-9268|https://github.com/apache/flink/pull/9268].
However, this would meet another problem if {{failGlobal}} come across an
exception and cannot be caught by {{FatalExitExceptionHandler}}.
I still have not found any graceful solution to this bug, [~gjy], [~Zentol] do
you have any suggestions for this?
> Pipelined region failover strategy does not recover Job if checkpoint cannot
> be read
> ------------------------------------------------------------------------------------
>
> Key: FLINK-13452
> URL: https://issues.apache.org/jira/browse/FLINK-13452
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.9.0, 1.10.0
> Reporter: Gary Yao
> Assignee: Yun Tang
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.9.0
>
> Attachments: jobmanager.log
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The job does not recover if a checkpoint cannot be read and
> {{jobmanager.execution.failover-strategy}} is set to _"region"_.
> *Analysis*
> The {{RestartCallback}} created by
> {{AdaptedRestartPipelinedRegionStrategyNG}} throws a \{{RuntimeException}} if
> no checkpoints could be read. When the restart is invoked in a separate
> thread pool, the exception is swallowed. See:
> [https://github.com/apache/flink/blob/21621fbcde534969b748f21e9f8983e3f4e0fb1d/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/failover/AdaptedRestartPipelinedRegionStrategyNG.java#L117-L119]
> [https://github.com/apache/flink/blob/21621fbcde534969b748f21e9f8983e3f4e0fb1d/flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/restart/FixedDelayRestartStrategy.java#L65]
> *Expected behavior*
> * Job should restart
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
