YufeiLiu created FLINK-26800:
--------------------------------
Summary: write small data file using share write buffer manager
Key: FLINK-26800
URL: https://issues.apache.org/jira/browse/FLINK-26800
Project: Flink
Issue Type: Bug
Components: Runtime / State Backends
Affects Versions: 1.12.7
Reporter: YufeiLiu
When setting config {{state.backend.rocksdb.memory.fixed-per-slot}}, all
rocksdb instances in same slot are using shared WriteBufferManager.
I meet a extreme circumstances, there are 2 rocksdb using a WriteBufferManager
size is 32M, if rocksdb-1 write (32*0.9)M data and won't have more data in a
while, it won't trigger flush, then rocksdb-2 start writing data, it will
trigger flush every single record and write many small file, spend a lot of
time of backgroud compaction.
rocksdb-2 only flush current CF data, it won't recovery until rocksdb-1 flush
data in memtable.
I can disable memory managed option to avoid this case, but I can't limit the
memory usage.
Maybe can create a tracker monitor all rocksdb memory usage, trigger a force
flush if it's necessary?
[1] RocksDB Write Buffer Manager
https://github.com/facebook/rocksdb/wiki/Write-Buffer-Manager
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
