[
https://issues.apache.org/jira/browse/FLINK-23949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17411740#comment-17411740
]
Yun Tang edited comment on FLINK-23949 at 9/8/21, 8:24 AM:
-----------------------------------------------------------
merged
master: d2007b74d543edb05af203f115459c51378867d3
release-1.14: 602a177f639154e50f1db3b663ca5f282b6aa49b
release-1.13: b0e532621745d32b218596beceb2e385e766a9a6
-release-1.12: 5f12f4ce88a4e83473e778eb30742ab88d92bdb0-
was (Author: yunta):
merged
master: d2007b74d543edb05af203f115459c51378867d3
release-1.14: 602a177f639154e50f1db3b663ca5f282b6aa49b
release-1.13: b0e532621745d32b218596beceb2e385e766a9a6
release-1.12: 5f12f4ce88a4e83473e778eb30742ab88d92bdb0
> first incremental checkpoint after a savepoint will degenerate into a full
> checkpoint
> -------------------------------------------------------------------------------------
>
> Key: FLINK-23949
> URL: https://issues.apache.org/jira/browse/FLINK-23949
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Affects Versions: 1.11.4, 1.12.5, 1.13.2
> Reporter: Feifan Wang
> Assignee: Feifan Wang
> Priority: Major
> Labels: pull-request-available
> Fix For: 1.14.0, 1.12.6, 1.13.3, 1.15.0
>
> Attachments: image-2021-08-25-00-59-05-779.png
>
>
> In RocksIncrementalSnapshotStrategy we will record the uploaded rocksdb files
> corresponding to the checkpoint id，and clean it in
> _CheckpointListener#notifyCheckpointComplete ._
> {code:java}
> @Override
> public void notifyCheckpointComplete(long completedCheckpointId) {
> synchronized (materializedSstFiles) {
> if (completedCheckpointId > lastCompletedCheckpointId) {
> materializedSstFiles
> .keySet()
> .removeIf(checkpointId -> checkpointId <
> completedCheckpointId);
> lastCompletedCheckpointId = completedCheckpointId;
> }
> }
> }{code}
>
> This works well without savepoint, but when a savepoint is completed, it will
> clean up the _materializedSstFiles_ of the previous checkpoint. It leads to
> the first checkpoint after the savepoint must upload all files in rocksdb.
> !image-2021-08-25-00-59-05-779.png|width=1188,height=163!
> Solving the problem is also very simple, I propose to clean
> _materializedSstFiles_ and update _lastCompletedCheckpointId_ only when
> {color:#ff0000}_materializedSstFiles.keySet().contains(completedCheckpointId)_{color}
> .
> If a _completedCheckpointId_ is not in _materializedSstFiles.keySet()_ ,
> there are only two cases：
> 1. It is a checkpoint but there is a checkpoint with larger id number
> completed before it
> 2. It is a savepoint (savepoint not produce by
> RocksIncrementalSnapshotStrategy)
> In either case we don’t need clean _materializedSstFiles_ and update
> _lastCompletedCheckpointId_ anymore.
> [~yunta] , [~trohrmann] , I have submitted a pull request to solve this
> problem, please evaluate whether it is appropriate.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
