GitHub user aljoscha opened a pull request:
https://github.com/apache/flink/pull/4863
[FLINK-7783] Don't always remove checkpoints in
ZooKeeperCompletedCheckpointStore#recover()
This is a bit simpler now than described on the issue: We simply never
delete checkpoints except when they are subsumed. We always fail completely
when we try to retrieve a checkpoint handle, both in `getLatestCheckpoint()`
and `getAllCheckpoints()`. The reason for this are incremental state handles:
`getAllCheckpoints()` is used to rebuild the dependency information between the
incremental state handles when restoring. If we only return a partial list of
state handles from `getAllCheckpoints()` then this logic wouldn't work anymore,
that's why we have to fail in this method as soon as one handle is not
retrievable.
I could change `getLatestCheckpoint()` to ignore broken handles and return
an older one (in fact I have the code for this ready) but it doesn't make sense
because `getAllCheckpoints()` is also called when `getLatestCheckpoint()` is
called, meaning that restoring will fail anyways.
R: @StefanRRichter AND/OR @tillrohrmann
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aljoscha/flink
jira-7783-zookeeper-state-store-fix-simplified
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/4863.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #4863
----
commit bbb105a997594299ab815ec176f1893dd02ad8b9
Author: Aljoscha Krettek <[email protected]>
Date: 2017-10-19T13:06:41Z
[FLINK-7783] Don't always remove checkpoints in
ZooKeeperCompletedCheckpointStore#recover()
----
---
