[ https://issues.apache.org/jira/browse/APEXMALHAR-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
David Yan closed APEXMALHAR-2191. --------------------------------- > Implement a way to get all keys with or without prefix from > SpillableByteMapImpl > -------------------------------------------------------------------------------- > > Key: APEXMALHAR-2191 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2191 > Project: Apache Apex Malhar > Issue Type: Sub-task > Reporter: David Yan > Assignee: David Yan > Priority: Critical > > WindowedKeyedStorage is basically a Map<Window, Map<K, V>>, and we need the > capability of getting all K's given a window, and getting V given a window > and K. > Currently, the spillable implementation of WindowedKeyedStorage uses two > spillable data structures -- SpillableByteMapImpl<Pair<Window, K>, V> and > SpillableArrayListMultimapImpl<Window, K>. This will not work because we > need to be able to remove a key from a window, which SpillableArrayList does > not support, and having two separate spillable data structures if it can be > achieved by just one should be avoided in general. > We will implement the solution that supports prefix scanning (given a window, > return all keys), which requires the keys to be stored in order in managed > state. > Traversing all keys in the spillable map is needed by WindowStateMap. The > WindowedOperator needs a way to traverse all windows in its state when firing > a trigger. However, this is less urgent since the Window meta info is small > and should fit in memory even with millions of windows. This is more for the > checkpointing efficiency. -- This message was sent by Atlassian JIRA (v6.3.4#6332)