[
https://issues.apache.org/jira/browse/FLINK-8602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sihua Zhou updated FLINK-8602:
------------------------------
Description:
Right now, RocksDB backend supports {{fully checkpoint}} & {{incremental
checkpoint}}. This issue try to improve the performance of recovery from
{{fully checkpoint}} as well as from {{increment checkpoint }}. It contains 2
sub-task.
1. improve recovery performance for incremental checkpoint when {{hasExtraKeys
= true}}
2. introduce `parallel recovery` from both {{fully checkpoint}} and
{{incremental checkpoint}} (Base on {{ingestExternalFile()}} and
{{SstFileWriter}} provided by RocksDB).
Any advice would be highly appreciated!
was:
Currently, when enable incremental checkpoint, if user change the parallelism
then ```hasExtraKeys``` may be ```true```. If this occur, flink will loop all
rocksdb instance and iterator all data to fetch the data that fails into
current ```KeyGroupRange````, this can be improved as follows:
- 1. For multi rocksdbs, we don't need to iterate the entry of them and insert
into another one, we can use the `ingestExternalFile()` api to merge them.
- 2. For the keyGroup which is not belong the target keyGroupRange, we can
delete them lazily by set the `CompactFilter` for the `ColumnFamily`.
Any advice would be highly appreciated!
> Improve recovery performance for rocksdb backend
> ------------------------------------------------
>
> Key: FLINK-8602
> URL: https://issues.apache.org/jira/browse/FLINK-8602
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
> Affects Versions: 1.5.0
> Reporter: Sihua Zhou
> Assignee: Sihua Zhou
> Priority: Major
>
> Right now, RocksDB backend supports {{fully checkpoint}} & {{incremental
> checkpoint}}. This issue try to improve the performance of recovery from
> {{fully checkpoint}} as well as from {{increment checkpoint }}. It contains 2
> sub-task.
> 1. improve recovery performance for incremental checkpoint when
> {{hasExtraKeys = true}}
> 2. introduce `parallel recovery` from both {{fully checkpoint}} and
> {{incremental checkpoint}} (Base on {{ingestExternalFile()}} and
> {{SstFileWriter}} provided by RocksDB).
> Any advice would be highly appreciated!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
