[
https://issues.apache.org/jira/browse/FLINK-13698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16906060#comment-16906060
]
Stephan Ewen commented on FLINK-13698:
--------------------------------------
I would suggest to have the {{CheckpointCoordinator}} and the
{{Scheduler}}/{{ExecutionGraph}} in the same thread.
That makes a lot of things much easier.
- The checkpoint timer logic should be factored out into the periodic
checkpoint trigger.
- The actual call
- All cleanup and writing of state should be delegated to the I/O executor.
-> mainly state release as a "fire and forget" execution in the I/O executor
-> writing out master-hook data and checkpoint metadata needs to happen in
the I/O executor, with a Future whose completion is again handles in the main
thread executor.
> Rework threading model of CheckpointCoordinator
> -----------------------------------------------
>
> Key: FLINK-13698
> URL: https://issues.apache.org/jira/browse/FLINK-13698
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
> Affects Versions: 1.10.0
> Reporter: Piotr Nowojski
> Assignee: Biao Liu
> Priority: Critical
>
> Currently {{CheckpointCoordinator}} and {{CheckpointFailureManager}} code is
> executed by multiple different threads (mostly {{ioExecutor}}, but not only).
> It's causing multiple concurrency issues, for example:
> https://issues.apache.org/jira/browse/FLINK-13497
> Proper fix would be to rethink threading model there. At first glance it
> doesn't seem that this code should be multi threaded, except of parts doing
> the actual IO operations, so it should be possible to run everything in one
> single ExecutionGraph's thread and just run asynchronously necessary IO
> operations with some feedback loop ("mailbox style").
> I would strongly recommend fixing this issue before adding new features in
> the \{{CheckpointCoordinator}} component.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
