[
https://issues.apache.org/jira/browse/APEXMALHAR-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Yan closed APEXMALHAR-2191.
---------------------------------
> Implement a way to get all keys with or without prefix from
> SpillableByteMapImpl
> --------------------------------------------------------------------------------
>
> Key: APEXMALHAR-2191
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2191
> Project: Apache Apex Malhar
> Issue Type: Sub-task
> Reporter: David Yan
> Assignee: David Yan
> Priority: Critical
>
> WindowedKeyedStorage is basically a Map<Window, Map<K, V>>, and we need the
> capability of getting all K's given a window, and getting V given a window
> and K.
> Currently, the spillable implementation of WindowedKeyedStorage uses two
> spillable data structures -- SpillableByteMapImpl<Pair<Window, K>, V> and
> SpillableArrayListMultimapImpl<Window, K>. This will not work because we
> need to be able to remove a key from a window, which SpillableArrayList does
> not support, and having two separate spillable data structures if it can be
> achieved by just one should be avoided in general.
> We will implement the solution that supports prefix scanning (given a window,
> return all keys), which requires the keys to be stored in order in managed
> state.
> Traversing all keys in the spillable map is needed by WindowStateMap. The
> WindowedOperator needs a way to traverse all windows in its state when firing
> a trigger. However, this is less urgent since the Window meta info is small
> and should fit in memory even with millions of windows. This is more for the
> checkpointing efficiency.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
