Hi Fabian,

> I'd like to clarify what I said before.
> By using MapState mainly gain two things:
> - position access by index
> - the full list does not need to be deserialized to read values (which is how 
> ListState works).
> Point access should obviously done by get(index). 
> However, iterating over the list should be done by iterating over the entry 
> (or value) set. The entry set iterator will prefetch multiple entries and 
> only deserialize the key / values when you access them. This reduces the 
> number of RocksDB look-ups.

Sorry, I should have been more precise in my description below. I have to do 
incremental iteration (e.g. process the next 10 entries).

I’m assuming I can’t hold onto the iterator across calls to a function, right?

If so, then making get(index) calls via the technique described below is 
currently the most efficient approach, yes?


— Ken

> 2018-02-19 0:10 GMT+01:00 Ken Krugler <kkrugler_li...@transpac.com 
> <mailto:kkrugler_li...@transpac.com>>:
> Hi there,
> I’ve got a MapState where I need to iterate over the entries.
> This currently isn’t supported (at least for Rocks DB), AFAIK, though there 
> is an issue/PR <https://issues.apache.org/jira/browse/FLINK-8297> to improve 
> this.
> The best solution I’ve seen is what Fabian proposed, which involves keeping a 
> ValueState with a count of entries, and then having the key for the MapState 
> be the index.
>> I cannot comment on the internal design, but you could put the data into a
>> RocksDBStateBackend MapState<Integer, X> where the value X is your data
>> type and the key is the list index. You would need another ValueState for
>> the current number of elements that you put into the MapState.
>> A MapState allows to fetch and traverse the key, value, or entry set of the
>> Map without loading it completely into memory.
>> The sets are traversed in sort order of the key, so should be in insertion
>> order (given that you properly increment the list index).
> This effectively lets you iterate over all of the map entries for a given 
> (keyed) state - though it doesn’t solve the “I have to iterate over _every_ 
> entry” situation.
> Is this currently the best option?
> Thanks,
> — Ken
> --------------------------------------------
> http://about.me/kkrugler <http://about.me/kkrugler>
> +1 530-210-6378 <tel:(530)%20210-6378>

+1 530-210-6378

Reply via email to