Yue Ma created FLINK-32833:
------------------------------
Summary: Rocksdb CacheIndexAndFilterBlocks must be true when using
shared memory
Key: FLINK-32833
URL: https://issues.apache.org/jira/browse/FLINK-32833
Project: Flink
Issue Type: Improvement
Components: Runtime / State Backends
Affects Versions: 1.17.1
Reporter: Yue Ma
Currently in RocksDBResourceContainer#getColumnOptions, if sharedResources is
used, blockBasedTableConfig will add the following configuration by default.
blockBasedTableConfig.setBlockCache(blockCache);
blockBasedTableConfig.setCacheIndexAndFilterBlocks(true);
blockBasedTableConfig.setCacheIndexAndFilterBlocksWithHighPriority(true);
blockBasedTableConfig.setPinL0FilterAndIndexBlocksInCache(true);
In my understanding, these configurations can help flink better manage the
memory of rocksdb and save some memory overhead in some scenarios. But this may
not be the best practice, mainly for the following reasons:
1. After CacheIndexAndFilterBlocks is set to true, it may cause index and
filter miss when reading, resulting in performance degradation.
2. These parameters may not be bound together with whether shared memory is
used, or some configurations should be supported separately to decide whether
to enable these features
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
