Till Rohrmann created FLINK-6626:
------------------------------------
Summary: Unifying lifecycle management of SubmittedJobGraph- and
CompletedCheckpointStore
Key: FLINK-6626
URL: https://issues.apache.org/jira/browse/FLINK-6626
Project: Flink
Issue Type: Improvement
Components: Distributed Coordination
Affects Versions: 1.3.0, 1.4.0
Reporter: Till Rohrmann
Currently, Flink uses the {{SubmittedJobGraphStore}} to persist {{JobGraphs}}
such that they can be recovered in case of failures. The
{{SubmittedJobGraphStore}} is managed by by the {{JobManager}}. Additionally,
Flink has the {{CompletedCheckpointStore}} which stores checkpoints for a given
{{ExecutionGraph}}/job. The {{CompletedCheckpointStore}} is managed by the
{{CheckpointCoordinator}}.
The {{SubmittedJobGraphStore}} and the {{CompletedCheckpointStore}} are
somewhat related because in the latter we store checkpoints for jobs contained
in the former. I think it would be nice wrt lifecycle management to let the
{{SubmittedJobGraphStore}} manage the lifecycle of the
{{CompletedCheckpointStore}}, because often it does not make much sense to keep
only checkpoints without a job or a job without checkpoints.
An idea would be when we register a job with the {{SubmittedJobGraphStore}}
then it returns a {{CompletedCheckpointStore}}. This store can then be given to
the {{CheckpointCoordinator}} to store the checkpoints. When a job enters a
terminal state it could be the responsibility of the {{SubmittedJobGraphStore}}
to decide what to do with the job data ({{JobGraph}} and {{Checkpoints}}), e.g.
keeping it or cleaning it up.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
