Jan Lukavský created FLINK-8297:
-----------------------------------
Summary: RocksDBListState stores whole list in single byte[]
Key: FLINK-8297
URL: https://issues.apache.org/jira/browse/FLINK-8297
Project: Flink
Issue Type: Improvement
Components: Core
Affects Versions: 1.3.2, 1.4.0
Reporter: Jan Lukavský
RocksDBListState currently keeps whole list of data in single RocksDB key-value
pair, which implies that the list actually must fit into memory. Larger lists
are not supported and end up with OOME or other error. The RocksDBListState
could be modified so that individual items in list are stored in separate keys
in RocksDB and can then be iterated over. A simple implementation could reuse
existing RocksDBMapState, with key as index to the list and a single
RocksDBValueState keeping track of how many items has already been added to the
list. Because this implementation might be less efficient in come cases, it
would be good to make it opt-in by a construct like
{{new RocksDBStateBackend().enableLargeListsPerKey()}}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
