[
https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15481280#comment-15481280
]
ASF GitHub Bot commented on APEXMALHAR-2130:
--------------------------------------------
GitHub user davidyan74 opened a pull request:
https://github.com/apache/apex-malhar/pull/407
APEXMALHAR-2130 WindowStorage interface changes in preparation of inc…
…orporating spillable data structures
@siyuanh please review and merge
@tweise @ShunxinLu FYI
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davidyan74/apex-malhar
windowedStorageInterfaceChange
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/apex-malhar/pull/407.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #407
----
commit 103f7e39d4d855a720d1f5e9b76152cfe9b79d8d
Author: David Yan <[email protected]>
Date: 2016-09-11T07:30:20Z
APEXMALHAR-2130 WindowStorage interface changes in preparation of
incorporating spillable data structures
----
> implement scalable windowed storage
> -----------------------------------
>
> Key: APEXMALHAR-2130
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2130
> Project: Apache Apex Malhar
> Issue Type: Task
> Reporter: bright chen
> Assignee: David Yan
>
> This feature is used for supporting windowing.
> The storage needs to have the following features:
> 1. Spillable key value storage (integrate with APEXMALHAR-2026)
> 2. Upon checkpoint, it saves a snapshot for the entire data set with the
> checkpointing window id. This should be done incrementally (ManagedState) to
> avoid wasting space with unchanged data
> 3. When recovering, it takes the recovery window id and restores to that
> snapshot
> 4. When a window is committed, all windows with a lower ID should be purged
> from the store.
> 5. It should implement the WindowedStorage and WindowedKeyedStorage
> interfaces, and because of 2 and 3, we may want to add methods to the
> WindowedStorage interface so that the implementation of WindowedOperator can
> notify the storage of checkpointing, recovering and committing of a window.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
