[
https://issues.apache.org/jira/browse/FLINK-29577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cai Liuyang updated FLINK-29577:
--------------------------------
Description:
For now, RocksDBFullRestoreOperation and RocksDBHeapTimersFullRestoreOperation
does's pass RocksDB::WriteOptions to RocksDBWriteBatchWrapper when restore
kv-data, which will use RocksDBWriteBatchWrapper‘s default WriteOptions(doesn't
disable rocksdb wal explicitly, see code below), so during restoring from full
snapshot, wal is enabled(use more disk and maybe affect
rocksdb-write-performance when restoring)
{code:java}
// First: RocksDBHeapTimersFullRestoreOperation::restoreKVStateData() doesn't
pass WriteOptions to RocksDBWriteBatchWrapper(null as default)
private void restoreKVStateData(
ThrowingIterator<KeyGroup> keyGroups,
Map<Integer, ColumnFamilyHandle> columnFamilies,
Map<Integer, HeapPriorityQueueSnapshotRestoreWrapper<?>>
restoredPQStates)
throws IOException, RocksDBException, StateMigrationException {
// for all key-groups in the current state handle...
try (RocksDBWriteBatchWrapper writeBatchWrapper =
new RocksDBWriteBatchWrapper(this.rocksHandle.getDb(),
writeBatchSize)) {
HeapPriorityQueueSnapshotRestoreWrapper<HeapPriorityQueueElement>
restoredPQ = null;
ColumnFamilyHandle handle = null;
......
}
// Second: RocksDBWriteBatchWrapper::flush function doesn't disable wal
explicitly when user doesn't pass WriteOptions to RocksDBWriteBatchWrapper
public void flush() throws RocksDBException {
if (options != null) {
db.write(options, batch);
} else {
// use the default WriteOptions, if wasn't provided.
try (WriteOptions writeOptions = new WriteOptions()) {
db.write(writeOptions, batch);
}
}
batch.clear();
}
{code}
As we known, rocksdb's wal is usesless for flink, so i think we can disable wal
for RocksDBWriteBatchWrapper's default WriteOptions.
was:
For now, RocksDBFullRestoreOperation and RocksDBHeapTimersFullRestoreOperation
does's pass RocksDB::WriteOptions to RocksDBWriteBatchWrapper when restore
kv-data, which will use RocksDBWriteBatchWrapper‘s default WriteOptions(doesn't
disable rocksdb wal explicitly, see code below), so during restoring from full
snapshot, wal is enabled(use more disk and affect rocksdb-write-performance
when restoring)
{code:java}
// First: RocksDBHeapTimersFullRestoreOperation::restoreKVStateData() doesn't
pass WriteOptions to RocksDBWriteBatchWrapper(null as default)
private void restoreKVStateData(
ThrowingIterator<KeyGroup> keyGroups,
Map<Integer, ColumnFamilyHandle> columnFamilies,
Map<Integer, HeapPriorityQueueSnapshotRestoreWrapper<?>>
restoredPQStates)
throws IOException, RocksDBException, StateMigrationException {
// for all key-groups in the current state handle...
try (RocksDBWriteBatchWrapper writeBatchWrapper =
new RocksDBWriteBatchWrapper(this.rocksHandle.getDb(),
writeBatchSize)) {
HeapPriorityQueueSnapshotRestoreWrapper<HeapPriorityQueueElement>
restoredPQ = null;
ColumnFamilyHandle handle = null;
......
}
// Second: RocksDBWriteBatchWrapper::flush function doesn't disable wal
explicitly when user doesn't pass WriteOptions to RocksDBWriteBatchWrapper
public void flush() throws RocksDBException {
if (options != null) {
db.write(options, batch);
} else {
// use the default WriteOptions, if wasn't provided.
try (WriteOptions writeOptions = new WriteOptions()) {
db.write(writeOptions, batch);
}
}
batch.clear();
}
{code}
As we known, rocksdb's wal is usesless for flink, so i think we can disable wal
for RocksDBWriteBatchWrapper's default WriteOptions.
> Disable rocksdb wal when restore from full snapshot
> ---------------------------------------------------
>
> Key: FLINK-29577
> URL: https://issues.apache.org/jira/browse/FLINK-29577
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Reporter: Cai Liuyang
> Assignee: Cai Liuyang
> Priority: Major
> Labels: pull-request-available
>
> For now, RocksDBFullRestoreOperation and
> RocksDBHeapTimersFullRestoreOperation does's pass RocksDB::WriteOptions to
> RocksDBWriteBatchWrapper when restore kv-data, which will use
> RocksDBWriteBatchWrapper‘s default WriteOptions(doesn't disable rocksdb wal
> explicitly, see code below), so during restoring from full snapshot, wal is
> enabled(use more disk and maybe affect rocksdb-write-performance when
> restoring)
>
> {code:java}
> // First: RocksDBHeapTimersFullRestoreOperation::restoreKVStateData() doesn't
> pass WriteOptions to RocksDBWriteBatchWrapper(null as default)
> private void restoreKVStateData(
> ThrowingIterator<KeyGroup> keyGroups,
> Map<Integer, ColumnFamilyHandle> columnFamilies,
> Map<Integer, HeapPriorityQueueSnapshotRestoreWrapper<?>>
> restoredPQStates)
> throws IOException, RocksDBException, StateMigrationException {
> // for all key-groups in the current state handle...
> try (RocksDBWriteBatchWrapper writeBatchWrapper =
> new RocksDBWriteBatchWrapper(this.rocksHandle.getDb(),
> writeBatchSize)) {
> HeapPriorityQueueSnapshotRestoreWrapper<HeapPriorityQueueElement>
> restoredPQ = null;
> ColumnFamilyHandle handle = null;
> ......
> }
> // Second: RocksDBWriteBatchWrapper::flush function doesn't disable wal
> explicitly when user doesn't pass WriteOptions to RocksDBWriteBatchWrapper
> public void flush() throws RocksDBException {
> if (options != null) {
> db.write(options, batch);
> } else {
> // use the default WriteOptions, if wasn't provided.
> try (WriteOptions writeOptions = new WriteOptions()) {
> db.write(writeOptions, batch);
> }
> }
> batch.clear();
> }
> {code}
>
>
> As we known, rocksdb's wal is usesless for flink, so i think we can disable
> wal for RocksDBWriteBatchWrapper's default WriteOptions.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
