[
https://issues.apache.org/jira/browse/FLINK-26800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17513176#comment-17513176
]
Yun Tang commented on FLINK-26800:
----------------------------------
[~liuyufei], RocksDBJava does not publish too many details about
{{WriteBufferManager}}, and we cannot track the stats of each column family
under current status. I think you could create such issue or PR to RocksDB
community and we could then cherry-pick them back if necessary.
> write small data file using share write buffer manager
> ------------------------------------------------------
>
> Key: FLINK-26800
> URL: https://issues.apache.org/jira/browse/FLINK-26800
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
> Affects Versions: 1.12.7
> Reporter: YufeiLiu
> Priority: Major
>
> When setting config {{state.backend.rocksdb.memory.fixed-per-slot}}, all
> rocksdb instances in same slot are using shared WriteBufferManager.
> I meet a extreme circumstances, there are 2 rocksdb using a
> WriteBufferManager size is 32M, if rocksdb-1 write (32*0.9)M data and won't
> have more data in a while, it won't trigger flush, then rocksdb-2 start
> writing data, it will trigger flush every single record and write many small
> file, spend a lot of time of backgroud compaction.
> rocksdb-2 only flush current CF data, it won't recovery until rocksdb-1 flush
> data in memtable.
> I can disable memory managed option to avoid this case, but I can't limit the
> memory usage.
> Maybe can create a tracker monitor all rocksdb memory usage, trigger a force
> flush if it's necessary?
> [1] RocksDB Write Buffer Manager
> https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
