ASF GitHub Bot commented on APEXMALHAR-2130:

GitHub user davidyan74 opened a pull request:


    APEXMALHAR-2130 Spillable implementation for WindowedOperator

    @tweise @brightchen @siyuanh Please review and merge

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/davidyan74/apex-malhar windowedSpillable-PR

Alternatively you can review and apply these changes as the patch at:


To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #424
commit ede7f056919ba11c607b06d302153a7498e5e3e6
Author: David Yan <da...@datatorrent.com>
Date:   2016-08-15T21:19:08Z

    APEXMALHAR-2130 Spillable implementation for WindowedOperator


> Scalable windowed storage
> -------------------------
>                 Key: APEXMALHAR-2130
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2130
>             Project: Apache Apex Malhar
>          Issue Type: Task
>            Reporter: bright chen
>            Assignee: David Yan
>              Labels: roadmap
> This feature is used for supporting windowing.
> The storage needs to have the following features:
> 1. Spillable key value storage (integrate with APEXMALHAR-2026)
> 2. Upon checkpoint, it saves a snapshot for the entire data set with the 
> checkpointing window id.  This should be done incrementally (ManagedState) to 
> avoid wasting space with unchanged data
> 3. When recovering, it takes the recovery window id and restores to that 
> snapshot
> 4. When a window is committed, all windows with a lower ID should be purged 
> from the store.
> 5. It should implement the WindowedStorage and WindowedKeyedStorage 
> interfaces, and because of 2 and 3, we may want to add methods to the 
> WindowedStorage interface so that the implementation of WindowedOperator can 
> notify the storage of checkpointing, recovering and committing of a window.

This message was sent by Atlassian JIRA

Reply via email to