[
https://issues.apache.org/jira/browse/FLINK-28863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17627484#comment-17627484
]
Jinzhong Li edited comment on FLINK-28863 at 11/2/22 6:45 AM:
--------------------------------------------------------------
[~yunta]
1. I found that the changes I metioned above would change native savepoint's
behavior in CLAIM mode.
Now, when job restore from native savepoint in CLAIM mode, the first checkpoint
will be incremental based on savepoint sstFiles.
This means although native savepoint sstFiles stay in the exclusive scope
folder, the checkpoints of the job which restore from the savepoint can still
share sstFiles with it.
If we put
[sstFiles|https://github.com/apache/flink/blob/bb9f2525e6e16d00ef2f0739d9cb96c2e47e35e7/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java#L302]
into privateState Map of IncrementalRemoteKeyedStateHandle for native
savepoint, the above behavior will be changed.
2. Back to the issue itself, i think, if the semantic of
"[sharedState|https://github.com/apache/flink/blob/bb9f2525e6e16d00ef2f0739d9cb96c2e47e35e7/flink-runtime/src/main/java/org/apache/flink/runtime/state/IncrementalRemoteKeyedStateHandle.java#L80]";
in IncrementalRemoteKeyedStateHandle is that state files can be shared by
other checkpoints (include checkpoints after restore), *current behavior is by
design.* In other words, the snapshot artifacts in the exclusive scope
folder can still share sstFiles with checkpoints after restore in CLAIM mode.
[~yunta] WDYT?
was (Author: lijinzhong):
[~yunta]
1. I found that the changes I metioned above would change native savepoint's
behavior in CLAIM mode.
Now, when job restore from native savepoint in CLAIM mode, the first checkpoint
will be incremental based on savepoint sstFiles.
This means although native savepoint sstFiles stay in the exclusive scope
folder, the checkpoints of the job which restore from the savepoint can still
share sstFiles with it.
If we put sstFiles into privateState Map of IncrementalRemoteKeyedStateHandle
for native savepoint, the above behavior will be changed.
2. Back to the issue itself, i think, if the semantic of "sharedState" in
IncrementalRemoteKeyedStateHandle is that state files can be shared by other
checkpoints (include checkpoints after restore), *current behavior is by
design.* In other words, the snapshot artifacts in the exclusive scope
folder can still share sstFiles with checkpoints after restore in CLAIM mode.
[~yunta] WDYT?
> Snapshot result of RocksDB native savepoint should have empty shared-state
> --------------------------------------------------------------------------
>
> Key: FLINK-28863
> URL: https://issues.apache.org/jira/browse/FLINK-28863
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / State Backends
> Reporter: Yun Tang
> Assignee: Jinzhong Li
> Priority: Major
> Fix For: 1.17.0, 1.15.3, 1.16.1
>
>
> The current snapshot result of RocksDB native savepoint has non-empty shared
> state, which is obviously not correct as all snapshot artifacts already stay
> in the exclusive checkpoint scope folder.
> This does not bring real harmful result due to we would not register the
> snapshot results of RocksDB native savepoint.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
