[
https://issues.apache.org/jira/browse/FLINK-32643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17873236#comment-17873236
]
Xinglong Wang commented on FLINK-32643:
---------------------------------------
It seems that setting the block cache has a higher priority than setting the
block cache size:
{code:java}
// java/rocksjni/table.cc
// ...
if (options.no_block_cache) {
options.block_cache = nullptr;
} else {
if (jblock_cache_handle > 0) {
std::shared_ptr<ROCKSDB_NAMESPACE::Cache> *pCache =
reinterpret_cast<std::shared_ptr<ROCKSDB_NAMESPACE::Cache> *>(
jblock_cache_handle);
options.block_cache = *pCache;
} else if (jblock_cache_size >= 0) {
if (jblock_cache_num_shard_bits > 0) {
options.block_cache = ROCKSDB_NAMESPACE::NewLRUCache(
static_cast<size_t>(jblock_cache_size),
static_cast<int>(jblock_cache_num_shard_bits));
} else {
options.block_cache = ROCKSDB_NAMESPACE::NewLRUCache(
static_cast<size_t>(jblock_cache_size));
}
} else {
options.no_block_cache = true;
options.block_cache = nullptr;
}
}
// ...{code}
Moreover, in Flink, within `RocksDBResourceContainer#getColumnOptions`, the
Cache is explicitly set.
So I think `state.backend.rocksdb.block.cache-size` is not effective anymore.
Right? [~masteryhx]
> Introduce off-heap shared state cache across stateful operators in TM
> ---------------------------------------------------------------------
>
> Key: FLINK-32643
> URL: https://issues.apache.org/jira/browse/FLINK-32643
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
> Affects Versions: 1.19.0
> Reporter: Fang Yong
> Priority: Major
>
> Currently each stateful operator will create an independent db instance if it
> uses rocksdb as state backend, and we can configure
> `state.backend.rocksdb.block.cache-size` for each db to speed up state
> performance. This parameter defaults to 8M, and we cannot set it too large,
> such as 512M, this may cause OOM and each DB cannot effectively utilize
> memory. To address this issue, we would like to introduce off-heap shared
> state cache across multiple db instances for stateful operators in TM.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
