[ https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475007#comment-15475007 ]
ASF GitHub Bot commented on APEXMALHAR-2130: -------------------------------------------- GitHub user davidyan74 opened a pull request: https://github.com/apache/apex-malhar/pull/405 APEXMALHAR-2130 #resolve Added a spillable map that takes two keys with support of iterating through all entries with a given first key @siyuanh @tweise @ilooner Please review. This is mostly for the scalable implementation of a WindowedKeyedStorage. You can merge this pull request into a Git repository by running: $ git pull https://github.com/davidyan74/apex-malhar APEXMALHAR-2130 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/405.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #405 ---- commit 183c9ff01e3f197c9088cdcf460827023d414ce1 Author: David Yan <da...@datatorrent.com> Date: 2016-09-08T20:59:50Z APEXMALHAR-2130 #resolve Added a spillable map that takes two keys with support of iterating through all entries with a given first key ---- > implement scalable windowed storage > ----------------------------------- > > Key: APEXMALHAR-2130 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2130 > Project: Apache Apex Malhar > Issue Type: Task > Reporter: bright chen > Assignee: David Yan > > This feature is used for supporting windowing. > The storage needs to have the following features: > 1. Spillable key value storage (integrate with APEXMALHAR-2026) > 2. Upon checkpoint, it saves a snapshot for the entire data set with the > checkpointing window id. This should be done incrementally (ManagedState) to > avoid wasting space with unchanged data > 3. When recovering, it takes the recovery window id and restores to that > snapshot > 4. When a window is committed, all windows with a lower ID should be purged > from the store. > 5. It should implement the WindowedStorage and WindowedKeyedStorage > interfaces, and because of 2 and 3, we may want to add methods to the > WindowedStorage interface so that the implementation of WindowedOperator can > notify the storage of checkpointing, recovering and committing of a window. -- This message was sent by Atlassian JIRA (v6.3.4#6332)