[
https://issues.apache.org/jira/browse/FLINK-6014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15930423#comment-15930423
]
ASF GitHub Bot commented on FLINK-6014:
---------------------------------------
Github user StephanEwen commented on the issue:
https://github.com/apache/flink/pull/3524
Thanks for opening this pull request. Adding a `CompositeStateHandle` and a
`StateRegistry` is a good idea.
Some thoughts:
- What do you think about making the `StateRegistry` into a
`SharedStateRegistry` which only contains the handles to state that is shared
across checkpoints? State that is exclusive to a checkpoint is not handled by
that registry, but remains only in the checkpoint. That way we "isolate" the
existing behavior against the coming changes and do not risk regressions in the
state cleanup code (which is very critical for current users).
- Another reason for the above suggestion is to also bring some other
code into place that has some "fast paths" and "safety nets" for checkpoint
cleanups (currently only with non-shared state), for example dropping a
checkpoint simply by a `rm -r` (see https://github.com/apache/flink/pull/3522
). We have seen that for various users the state cleanup problems are among the
biggest problems they have, which we can address very well with the work
started in the above linked pull request. These things would work together
seamlessly if the registry deals only with shared state handles.
- I am wondering if it is easier to put the registry into the checkpoint
coordinator rather than the checkpoint stores. That way we need the code that
deals with adding / failure handling / etc only once.
> Allow the registration of state objects in checkpoints
> ------------------------------------------------------
>
> Key: FLINK-6014
> URL: https://issues.apache.org/jira/browse/FLINK-6014
> Project: Flink
> Issue Type: Sub-task
> Components: State Backends, Checkpointing
> Reporter: Xiaogang Shi
> Assignee: Xiaogang Shi
>
> This issue is the very first step towards incremental checkpointing. We
> introduce a new state handle named {{CompositeStateHandle}} to be the base of
> the snapshots taken by task components. Known implementation may include
> {{KeyedStateHandle}} (for {{KeyedStateBackend}}s), {{SubtaskState}} (for
> subtasks, splits of {{JobVertex}}) and {{TaskState}} (for {{JobVertex}}s).
> Each {{CompositeStateHandle}} is composed of a collection of {{StateObject}s.
> It should register all its state objects in {{StateRegistry}} when its
> checkpoint is added into {{CompletedCheckpointStore}} (i.e., a pending
> checkpoint completes or a complete checkpoint is reloaded in the recovery).
> When a completed checkpoint is moved out of the {{CompletedCheckpointStore}},
> we should not simply discard all state objects in the checkpoint. With the
> introduction of incremental checkpointing, a {{StateObject}} may be
> referenced by different checkpoints. We should unregister all the state
> objects contained in the {{StateRegistry}} first. Only those state objects
> that are not referenced by any checkpoint can be deleted.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
