Lihe Ma created FLINK-28699:
-------------------------------
Summary: Native rocksdb full snapshot in non-incremental
checkpointing
Key: FLINK-28699
URL: https://issues.apache.org/jira/browse/FLINK-28699
Project: Flink
Issue Type: Improvement
Components: Runtime / State Backends
Affects Versions: 1.15.1, 1.14.5
Reporter: Lihe Ma
When rocksdb statebackend is used and state.backend.incremental enabled, flink
will figure out newly created sst files generated by rocksdb during checkpoint,
and read all the states from rocksdb and write to files during savepoint [1].
When state.backend.incremental disabled, flink will read all the states from
rocksdb and generate state files in checkpoint and savepoint [2]. This makes
sense in savepoint, cause user can take a savepoint with rocksdb statebackend
and then restore it using another statebackend, but in checkpoint,
deserialisation and serialisation of state results in performance loss.
If the native rocksdb snapshot is introduced in full snapshot, theoretically
better performance can be achieved. At the same time, savepoint remains the
same as before.
#
https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksIncrementalSnapshotStrategy.java
#
https://github.com/apache/flink/blob/master/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/snapshot/RocksFullSnapshotStrategy.java
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
