[
https://issues.apache.org/jira/browse/FLINK-10855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16847702#comment-16847702
]
vinoyang commented on FLINK-10855:
----------------------------------
[~yunta] I think [~till.rohrmann] and I have reached an agreement on how to
achieve that (I'm just looking to see if [~srichter] still has any additions).
I usually assign issues to me because of two considerations: (1) this is also a
problem we encounter internally; (2) I have the ability to handle it;
I have a lot of issues and PR to follow up. But in the period after the 1.8
release, I focused mainly on checkpoint modules, so it seems that I just
randomly assigned an issue to myself, which is not the case. We also have our
own solutions to this problem. Thank you for your suggestion. I don't think a
job with too many tasks will have a significant impact on cleanup. Basically, I
think a simple strategy is that the cleaner records the metadata information
(intervals, timeouts) of checkpoints, which it uses, and it doesn't always scan
the file system.
Thank you for your suggestion. I will try to take it into account when dealing
with other checkpoints issue. I will try my best to balance priorities. Thank
you.
> CheckpointCoordinator does not delete checkpoint directory of late/failed
> checkpoints
> -------------------------------------------------------------------------------------
>
> Key: FLINK-10855
> URL: https://issues.apache.org/jira/browse/FLINK-10855
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
> Affects Versions: 1.5.5, 1.6.2, 1.7.0
> Reporter: Till Rohrmann
> Assignee: vinoyang
> Priority: Major
>
> In case that an acknowledge checkpoint message is late or a checkpoint cannot
> be acknowledged, we discard the subtask state in the
> {{CheckpointCoordinator}}. What's not happening in this case is that we
> delete the parent directory of the checkpoint. This only happens when we
> dispose a {{PendingCheckpoint#dispose}}.
> Due to this behaviour it can happen that a checkpoint fails (e.g. a task not
> being ready) and we delete the checkpoint directory. Next another task writes
> its checkpoint data to the checkpoint directory (thereby creating it again)
> and sending an acknowledge message back to the {{CheckpointCoordinator}}. The
> {{CheckpointCoordinator}} will realize that there is no longer a
> {{PendingCheckpoint}} and will discard the sub task state. This will remove
> the state files from the checkpoint directory but will leave the checkpoint
> directory untouched.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
