[ 
https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402612#comment-15402612
 ] 

ASF GitHub Bot commented on APEXMALHAR-2130:
--------------------------------------------

GitHub user davidyan74 reopened a pull request:

    https://github.com/apache/apex-malhar/pull/345

    APEXMALHAR-2130 REVIEW ONLY (WindowedOperator): incorporating Spillable 
data structures

    

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/davidyan74/apex-malhar windowedSpillable

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/apex-malhar/pull/345.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #345
    
----
commit c2c3f0acfcdf033a0e3044967ab3f8048f719259
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-06-05T00:11:20Z

    - Intermediate commit.

commit 1bee1ed0308470ff35a739ab6f9e94d53debddb8
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-06-13T06:03:21Z

    Intermediate commit

commit 60acf68b96f2145af3d90b410c6b20613347f881
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-06-21T06:58:09Z

    Intermediate commit

commit 5b9ca5e7e1e8139cc5c8ab6dd96b8a60553e38fc
Author: David Yan <da...@datatorrent.com>
Date:   2016-07-13T18:57:57Z

    Merge branch 'APEXMALHAR-2048_pull' of 
github.com:ilooner/incubator-apex-malhar

commit b72ee18e6f7cf0a9d30b195439912d96acafb3b4
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-07-17T21:32:34Z

    Added implementations of SpillableList, SpillableMap, and 
SpillableArrayListMultimap

commit 184653a23f662f78e7e6a7e1d53ffb0efbdb7127
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-07-18T01:56:41Z

    Added SpillableComplexComponentImpl

commit 9f17b4ba9233e3f46746505941a378236193f719
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-07-18T03:29:07Z

    Added propagating callbacks to store

commit 9e637a899e6aedb4ca6495a676654bb636616267
Author: devtagare <devtag...@gmail.com>
Date:   2016-05-18T22:25:56Z

    APEXMALHAR-2066 JdbcPolling,idempotent,partitionable

commit 10fe7a14c7294c54c5d18e4d0e94882778266ac6
Author: sandeshh <sandesh.he...@gmail.com>
Date:   2016-05-25T15:56:56Z

    Kafka 0.9.0 output operators and unit tests.
    
    1. Abstract Base class
    2. Kafka Output operator
    3. Exactly Once output operator
         Key in the Kafka message is used by the operator to track the tuples 
written by it.

commit 8df0de73f58c69ac457e5a28d51a7b56b5549859
Author: Chaitanya <chaita...@datatorrent.com>
Date:   2016-07-13T10:16:13Z

    APEXMALHAR-2019 Implemented S3 Input Module

commit 2ed0a102d434ace34978c6e940e04c998133fd1e
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-07-18T01:56:41Z

    Added SpillableComplexComponentImpl

commit c60e76eee8ae5cf034d4d9984b840d3219f52594
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-07-18T03:29:07Z

    Added propagating callbacks to store

commit 452db2feee7cc6e186df3faf384147b41eef40ea
Author: David Yan <da...@datatorrent.com>
Date:   2016-07-14T00:04:20Z

    Split WindowedStorage interface into two interfaces for plain and key data

commit be96f667238cce47957324eac9979ebafc013fc6
Author: David Yan <da...@datatorrent.com>
Date:   2016-07-15T21:40:59Z

    check null for retractionStorage

commit b7471a5df5e883b23b5a6051b974d3521ce46151
Author: David Yan <da...@datatorrent.com>
Date:   2016-07-15T21:48:13Z

    Removed unused imports

commit 361d0e152f4709dd8fda864ce94727161807a98e
Author: David Yan <da...@datatorrent.com>
Date:   2016-07-16T01:00:20Z

    added first draft implementation for spillable data structures

commit ad65865ed9e5ba25d44aacaf76372e2338ef6377
Author: David Yan <da...@datatorrent.com>
Date:   2016-07-18T21:04:24Z

    Implemented some of Tim's suggestion

commit 32c2660409b3c85872002aba6b84c5c5e748f203
Author: David Yan <da...@datatorrent.com>
Date:   2016-07-20T01:17:43Z

    moved storage initialization to setup

commit 955f2d6c66be26f8f205a743eb9c49d4e8d2bdd6
Author: David Yan <da...@datatorrent.com>
Date:   2016-07-20T17:50:05Z

    removed identifier as per tim's suggestion

commit ec4e7508e2fd70030ecab256bba32be4cad8ad3a
Author: David Yan <da...@datatorrent.com>
Date:   2016-07-21T00:05:51Z

    choose a bucket automatically and added entrySet implementation for 
SpillableWindowedKeyedStorage

commit fe41f0c20235aac3ca57facc2491ecfba11d20a7
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-07-21T06:38:48Z

    Added checkpoint callbacks to spillable complex components
    Added some half completed tests

commit dfc84a7847b4e71ec12ca8e8e1e0127732d880c0
Author: David Yan <da...@datatorrent.com>
Date:   2016-07-21T18:07:47Z

    intermediate commit

commit fe209fb72e169ae9605c826e61adfabda670c74d
Author: David Yan <da...@datatorrent.com>
Date:   2016-07-21T18:08:15Z

    Merge branch 'APEXMALHAR-2048_pull' of 
github.com:ilooner/incubator-apex-malhar into windowedSpillable

commit 11fb1ddd24705ed215ecbe4b9da72a39a3e8a9e1
Author: David Yan <da...@datatorrent.com>
Date:   2016-07-21T19:59:24Z

    intermediate commit

commit e5cacbbc9e1ae18f5cf0938f13377f75a3a99cd2
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-07-24T04:09:54Z

    Finished unit test for SpillableArrayListMultimap

commit dc258b8900688264f349307737b59b096dbc3d2b
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-07-24T05:19:37Z

    Added unit test which uses managed state

commit ed9924b810a76c39404701da81f4753ab68af5a5
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-07-24T06:55:18Z

    Finished adding managed state tests for SpillableByteMap

commit 43da17d9633dc2405b596b25697b6b4b0baef69f
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-07-24T16:31:53Z

    Added ManagedStateTests For SpillableArrayList

commit 1343ccf4ccc5099d7abcf7f99ab2ed648baeef08
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-07-24T16:49:00Z

    Added managed state tests for SpillableArrayListMultimap

commit 57c4e5e3c3e019613f31753be5052c32ba762e53
Author: Timothy Farkas <t...@datatorrent.com>
Date:   2016-07-24T16:57:35Z

    Added ManagedStateTest for SpillableComplexComponent

----


> implement scalable windowed storage
> -----------------------------------
>
>                 Key: APEXMALHAR-2130
>                 URL: https://issues.apache.org/jira/browse/APEXMALHAR-2130
>             Project: Apache Apex Malhar
>          Issue Type: Task
>            Reporter: bright chen
>            Assignee: David Yan
>
> This feature is used for supporting windowing.
> The storage needs to have the following features:
> 1. Spillable key value storage (integrate with APEXMALHAR-2026)
> 2. Upon checkpoint, it saves a snapshot for the entire data set with the 
> checkpointing window id.  This should be done incrementally (ManagedState) to 
> avoid wasting space with unchanged data
> 3. When recovering, it takes the recovery window id and restores to that 
> snapshot
> 4. When a window is committed, all windows with a lower ID should be purged 
> from the store.
> 5. It should implement the WindowedStorage and WindowedKeyedStorage 
> interfaces, and because of 2 and 3, we may want to add methods to the 
> WindowedStorage interface so that the implementation of WindowedOperator can 
> notify the storage of checkpointing, recovering and committing of a window.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to