ASF GitHub Bot commented on FLINK-8297:

Github user aljoscha commented on the issue:

    It doesn't have to be the same while were working on it in RocksDB but when 
checkpointing it should be written in the same format. This might be trickier 
than it sounds, btw, but I don't think we can have different savepoint formats 
for such a feature.

> RocksDBListState stores whole list in single byte[]
> ---------------------------------------------------
>                 Key: FLINK-8297
>                 URL: https://issues.apache.org/jira/browse/FLINK-8297
>             Project: Flink
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 1.4.0, 1.3.2
>            Reporter: Jan Lukavsk√Ĺ
>            Priority: Major
> RocksDBListState currently keeps whole list of data in single RocksDB 
> key-value pair, which implies that the list actually must fit into memory. 
> Larger lists are not supported and end up with OOME or other error. The 
> RocksDBListState could be modified so that individual items in list are 
> stored in separate keys in RocksDB and can then be iterated over. A simple 
> implementation could reuse existing RocksDBMapState, with key as index to the 
> list and a single RocksDBValueState keeping track of how many items has 
> already been added to the list. Because this implementation might be less 
> efficient in come cases, it would be good to make it opt-in by a construct 
> like
> {{new RocksDBStateBackend().enableLargeListsPerKey()}}

This message was sent by Atlassian JIRA

Reply via email to