[
https://issues.apache.org/jira/browse/FLINK-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276539#comment-17276539
]
Matthias commented on FLINK-21030:
----------------------------------
I paired with [~chesnay] to look into the issue. We came up with a testcase
covering the issue. We tried to emulate the problem by causing an Exception
when cancelling the Source task (here, I'm not 100% sure whether this is
exactly the same usecase [~TheoD] was describing in the issues description).
But we were able to reproduce the described behavior. It turns out that the
solution is a bit more complex than we thought (please correct me if we're
wrong here): The [current implementation of
SchedulerBase.stopWithSavepoint|https://github.com/XComp/flink/blob/60fb0f5c6e4e23759541a531e034f27f496e93b7/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SchedulerBase.java#L1080]
triggers the creation of a new savepoint and returns the completed checkpoint
in case of success. The stopping of the execution happens implicitly through
the {{CheckpointCoordinator}} which calls
{{AbstractInvokable#notifyheckpointCompleteAsync}} on each executing source
task. The {{stopWithSavepoint}} method has no information about whether the
actual stopping succeeds.
In our case, the stopping failed while
[SchedulerBase.stopWithSavepoint|https://github.com/XComp/flink/blob/60fb0f5c6e4e23759541a531e034f27f496e93b7/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SchedulerBase.java#L1109]
waits for the {{ExecutionGraph}} to terminate. The Source task got restarted.
No termination of the {{ExecutionGraph}} happens: The {{stopWithSavepoint}}
times out, instead.
> Broken job restart for job with disjoint graph
> ----------------------------------------------
>
> Key: FLINK-21030
> URL: https://issues.apache.org/jira/browse/FLINK-21030
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.11.2
> Reporter: Theo Diefenthal
> Assignee: Matthias
> Priority: Blocker
> Fix For: 1.11.4, 1.12.2, 1.13.0
>
>
> Building on top of bugs:
> https://issues.apache.org/jira/browse/FLINK-21028
> and https://issues.apache.org/jira/browse/FLINK-21029 :
> I tried to stop a Flink application on YARN via savepoint which didn't
> succeed due to a possible bug/racecondition in shutdown (Bug 21028). Due to
> some reason, Flink attempted to restart the pipeline after the failure in
> shutdown (21029). The bug here:
> As I mentioned: My jobgraph is disjoint and the pipelines are fully isolated.
> Lets say the original error occured in a single task of pipeline1. Flink then
> restarted the entire pipeline1, but pipeline2 was shutdown successfully and
> switched the state to FINISHED.
> My job thus was in kind of an invalid state after the attempt to stopping:
> One of two pipelines was running, the other was FINISHED. I guess this is
> kind of a bug in the restarting behavior that only all connected components
> of a graph are restarted, but the others aren't...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
