[
https://issues.apache.org/jira/browse/FLINK-8753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377654#comment-16377654
]
Stephan Ewen commented on FLINK-8753:
-------------------------------------
It sounds like what you are trying to achieve is closer to a checkpoint then to
a savepoint:
- checkpoint is potentially manually triggered
- checkpoint is not automatically removed once it is subsumed, but it is
retained
- could call it "archive checkpoint" or "detached checkpoint".
The tricky question to me is: if this thing is incremental, then there needs to
be some bookkeeping on how many references are made to the individual shared
state chunks. Someone would still need to hold a reference in the shared state
registry, or the state chunks will be removed once all other checkpoints stop
referencing them.
Alternatively, one could mark the chunks as 'detached', meaning they are not
reference counted any more, but always kept. Then the question is, how can one
determine how to clean the checkpoint up? The only way I can imagine this to
work in practice is on file systems that support hard links - in that case, the
hard links do the ref counting for you.
> Introduce Incremental savepoint
> -------------------------------
>
> Key: FLINK-8753
> URL: https://issues.apache.org/jira/browse/FLINK-8753
> Project: Flink
> Issue Type: New Feature
> Components: State Backends, Checkpointing
> Affects Versions: 1.5.0
> Reporter: Sihua Zhou
> Assignee: Sihua Zhou
> Priority: Major
>
> Right now, savepoint goes through the full checkpoint path, take a savepoint
> could be slowly. In our production, for some long term job it often costs
> more than 10min to complete a savepoint which is unacceptable for a real time
> job, so we have to turn back to use the externalized checkpoint instead
> currently. But the externalized checkpoint has a time interval (checkpoint
> interval) between the last time. So I proposal to introduce the increment
> savepoint which goes through the increment checkpoint path.
> Any advice would be appreciated!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
