[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Yan updated APEXMALHAR-2191:
----------------------------------
    Assignee:     (was: David Yan)

> Implement a way to get all keys with or without prefix from 
> SpillableByteMapImpl
> --------------------------------------------------------------------------------
>
>                 Key: APEXMALHAR-2191
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2191
>             Project: Apache Apex Malhar
>          Issue Type: Sub-task
>            Reporter: David Yan
>            Priority: Critical
>
> WindowedKeyedStorage is basically a Map<Window, Map<K, V>>, and we need the 
> capability of getting all K's given a window, and getting V given a window 
> and K.
> Currently, the spillable implementation of WindowedKeyedStorage uses two 
> spillable data structures -- SpillableByteMapImpl<Pair<Window, K>, V> and 
> SpillableArrayListMultimapImpl<Window, K>.  This will not work because we 
> need to be able to remove a key from a window, which SpillableArrayList does 
> not support, and having two separate spillable data structures if it can be 
> achieved by just one should be avoided in general. 
> We will implement the solution that supports prefix scanning (given a window, 
> return all keys), which requires the keys to be stored in order in managed 
> state.
> Traversing all keys in the spillable map is needed by WindowStateMap. The 
> WindowedOperator needs a way to traverse all windows in its state when firing 
> a trigger. However, this is less urgent since the Window meta info is small 
> and should fit in memory even with millions of windows. This is more for the 
> checkpointing efficiency.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to