ASF GitHub Bot commented on FLINK-8297:

Github user aljoscha commented on the issue:

    Thanks for the update. Regarding migration, I think the format that we 
write to the savepoint must be the same between the two `ListState` 
    @StefanRRichter could you please have a look at this? And also 
@StephanEwen, I think you did a similar PoC implementation of this a while back.
    @je-ik It would make sense to change the title of the PR to reflect what 
this does now better.

> RocksDBListState stores whole list in single byte[]
> ---------------------------------------------------
>                 Key: FLINK-8297
>                 URL: https://issues.apache.org/jira/browse/FLINK-8297
>             Project: Flink
>          Issue Type: Improvement
>          Components: Core
>    Affects Versions: 1.4.0, 1.3.2
>            Reporter: Jan Lukavsk√Ĺ
>            Priority: Major
> RocksDBListState currently keeps whole list of data in single RocksDB 
> key-value pair, which implies that the list actually must fit into memory. 
> Larger lists are not supported and end up with OOME or other error. The 
> RocksDBListState could be modified so that individual items in list are 
> stored in separate keys in RocksDB and can then be iterated over. A simple 
> implementation could reuse existing RocksDBMapState, with key as index to the 
> list and a single RocksDBValueState keeping track of how many items has 
> already been added to the list. Because this implementation might be less 
> efficient in come cases, it would be good to make it opt-in by a construct 
> like
> {{new RocksDBStateBackend().enableLargeListsPerKey()}}

This message was sent by Atlassian JIRA

Reply via email to