Hi Till,

> On Feb 19, 2018, at 8:14 AM, Till Rohrmann <trohrm...@apache.org> wrote:
> Hi Ken,
> just for my clarification, the `RocksDBMapState#entries` method does not 
> satisfy your requirements? This method does not allow you to iterate across 
> different keys of your keyed stream of course. But it should allow you to 
> iterate over the different entries for a given key of your keyed stream.

As per my email to Fabian, I should have been more precise in my requirements.

I need to do incremental iteration of the entries, versus a complete iteration.

And I'm assuming I can't keep the iterator around across calls to the function.


— Ken

> On Mon, Feb 19, 2018 at 12:10 AM, Ken Krugler <kkrugler_li...@transpac.com 
> <mailto:kkrugler_li...@transpac.com>> wrote:
> Hi there,
> I’ve got a MapState where I need to iterate over the entries.
> This currently isn’t supported (at least for Rocks DB), AFAIK, though there 
> is an issue/PR <https://issues.apache.org/jira/browse/FLINK-8297> to improve 
> this.
> The best solution I’ve seen is what Fabian proposed, which involves keeping a 
> ValueState with a count of entries, and then having the key for the MapState 
> be the index.
>> I cannot comment on the internal design, but you could put the data into a
>> RocksDBStateBackend MapState<Integer, X> where the value X is your data
>> type and the key is the list index. You would need another ValueState for
>> the current number of elements that you put into the MapState.
>> A MapState allows to fetch and traverse the key, value, or entry set of the
>> Map without loading it completely into memory.
>> The sets are traversed in sort order of the key, so should be in insertion
>> order (given that you properly increment the list index).
> This effectively lets you iterate over all of the map entries for a given 
> (keyed) state - though it doesn’t solve the “I have to iterate over _every_ 
> entry” situation.
> Is this currently the best option?

Ken Krugler
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr

Reply via email to