[
https://issues.apache.org/jira/browse/FLINK-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17276909#comment-17276909
]
Matthias commented on FLINK-21030:
----------------------------------
Yes, the notification of tasks happens right at the end of
[CheckpointCoordinator.completePendingCheckpoints:1280|https://github.com/XComp/flink/blob/c7ed2e818a4ead70c6081117f771c54a1bbc0fc5/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CheckpointCoordinator.java#L1280]
in {{sendAcknowledgeMessages}} without any failure handling. Hence, the errors
while canceling the tasks will be not considered when completing the Future
returned by [CheckpointCoordinator. triggerSynchronousSavepoint
|https://github.com/XComp/flink/blob/60fb0f5c6e4e23759541a531e034f27f496e93b7/flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SchedulerBase.java#L1080].
My proposal is to change the interface of
[TaskManagerGateway.notifyCheckpointComplete|https://github.com/XComp/flink/blob/2c4e0ab921ccfaf003073ee50faeae4d4e4f4c93/flink-runtime/src/main/java/org/apache/flink/runtime/jobmanager/slots/TaskManagerGateway.java#L96]
to return the {{CompletableFuture}} already returned by
[TaskExecutorGateway.confirmCheckpoint|https://github.com/XComp/flink/blob/2c4e0ab921ccfaf003073ee50faeae4d4e4f4c93/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutorGateway.java#L152].
This can, then, be forwarded to the {{completePendingCheckpoint}} method and,
then, be composed into the overall {{CompletableFuture}} returned by
{{CheckpointCoordinator.triggerSynchronousSavepoint}}. Do you see any problems
with this approach?
> Broken job restart for job with disjoint graph
> ----------------------------------------------
>
> Key: FLINK-21030
> URL: https://issues.apache.org/jira/browse/FLINK-21030
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
> Affects Versions: 1.11.2
> Reporter: Theo Diefenthal
> Assignee: Matthias
> Priority: Blocker
> Fix For: 1.11.4, 1.12.2, 1.13.0
>
>
> Building on top of bugs:
> https://issues.apache.org/jira/browse/FLINK-21028
> and https://issues.apache.org/jira/browse/FLINK-21029 :
> I tried to stop a Flink application on YARN via savepoint which didn't
> succeed due to a possible bug/racecondition in shutdown (Bug 21028). Due to
> some reason, Flink attempted to restart the pipeline after the failure in
> shutdown (21029). The bug here:
> As I mentioned: My jobgraph is disjoint and the pipelines are fully isolated.
> Lets say the original error occured in a single task of pipeline1. Flink then
> restarted the entire pipeline1, but pipeline2 was shutdown successfully and
> switched the state to FINISHED.
> My job thus was in kind of an invalid state after the attempt to stopping:
> One of two pipelines was running, the other was FINISHED. I guess this is
> kind of a bug in the restarting behavior that only all connected components
> of a graph are restarted, but the others aren't...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
