[
https://issues.apache.org/jira/browse/FLINK-4164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367596#comment-15367596
]
Aljoscha Krettek commented on FLINK-4164:
-----------------------------------------
I would suggest to copy the existing RocksDBStateBackend and change it to use
only one column. In process of this you might also have to adapt
RocksDBReducingState and the other specific state classes. Once you have that
you can test how the new backend behaves.
> Use only one rocksdb column family to store all states in an operator
> ---------------------------------------------------------------------
>
> Key: FLINK-4164
> URL: https://issues.apache.org/jira/browse/FLINK-4164
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
> Reporter: LiuBiao
> Priority: Minor
>
> I see now in master branch, there is only one rocksdb instance in an
> operator, each state will be assigned into separated column families. It's a
> good improvement, but I think there are still some problems. Column families
> in rocksdb do not share memtable and sst files(correct me if I am wrong,
> there is not much docs about column families in rocksdb). If user have
> thousands of states, the overhead will be costly. Memory may will be
> exhausted, also there will be too much small sst files.
> If we use prefix in key instead of column family, the memory that rocksdb
> costed will be under control. The shortcoming is that we can not optimize
> states separately, and performance may be a little worse than before. But I
> think it is worth to do, it is meaning for some cases.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
