[
https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15447700#comment-15447700
]
Timothy Farkas commented on APEXMALHAR-2130:
--------------------------------------------
[~davidyan] As far I can tell what you have implemented ontop of spillable
datastructures is 95% of the way there. I'm not sure what has changed.
1. SpillableByteMap already supports deleting a key by setting the value for a
key to be an empty byte array. So that's there. This can easily be extended to
delete all the values in SpillableArrayListMultimap.
2. The only missing piece is the ability to iterate over the set of keys in a
map. This can be done with another SpillableByteMap let's call this map the
linkedListMap. The key of the linkedListMap represents the current node, the
value represents the next node. You then keep track of your "head" key and
iterate by taking your current node, and getting the value. The value then
becomes the current node and so on. When the value for the current node is null
you are done traversing the list. You can take this and wrap it in an iterator.
> implement scalable windowed storage
> -----------------------------------
>
> Key: APEXMALHAR-2130
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2130
> Project: Apache Apex Malhar
> Issue Type: Task
> Reporter: bright chen
> Assignee: David Yan
>
> This feature is used for supporting windowing.
> The storage needs to have the following features:
> 1. Spillable key value storage (integrate with APEXMALHAR-2026)
> 2. Upon checkpoint, it saves a snapshot for the entire data set with the
> checkpointing window id. This should be done incrementally (ManagedState) to
> avoid wasting space with unchanged data
> 3. When recovering, it takes the recovery window id and restores to that
> snapshot
> 4. When a window is committed, all windows with a lower ID should be purged
> from the store.
> 5. It should implement the WindowedStorage and WindowedKeyedStorage
> interfaces, and because of 2 and 3, we may want to add methods to the
> WindowedStorage interface so that the implementation of WindowedOperator can
> notify the storage of checkpointing, recovering and committing of a window.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
