Hi Ken, just for my clarification, the `RocksDBMapState#entries` method does not satisfy your requirements? This method does not allow you to iterate across different keys of your keyed stream of course. But it should allow you to iterate over the different entries for a given key of your keyed stream.
Cheers, Till On Mon, Feb 19, 2018 at 12:10 AM, Ken Krugler <kkrugler_li...@transpac.com> wrote: > Hi there, > > I’ve got a MapState where I need to iterate over the entries. > > This currently isn’t supported (at least for Rocks DB), AFAIK, though > there is an issue/PR <https://issues.apache.org/jira/browse/FLINK-8297> to > improve this. > > The best solution I’ve seen is what Fabian proposed, which involves > keeping a ValueState with a count of entries, and then having the key for > the MapState be the index. > > I cannot comment on the internal design, but you could put the data into a > RocksDBStateBackend MapState<Integer, X> where the value X is your data > type and the key is the list index. You would need another ValueState for > the current number of elements that you put into the MapState. > A MapState allows to fetch and traverse the key, value, or entry set of the > Map without loading it completely into memory. > The sets are traversed in sort order of the key, so should be in insertion > order (given that you properly increment the list index). > > > This effectively lets you iterate over all of the map entries for a given > (keyed) state - though it doesn’t solve the “I have to iterate over _every_ > entry” situation. > > Is this currently the best option? > > Thanks, > > — Ken > > -------------------------------------------- > http://about.me/kkrugler > +1 530-210-6378 <(530)%20210-6378> > >