[ https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402612#comment-15402612 ]
ASF GitHub Bot commented on APEXMALHAR-2130: -------------------------------------------- GitHub user davidyan74 reopened a pull request: https://github.com/apache/apex-malhar/pull/345 APEXMALHAR-2130 REVIEW ONLY (WindowedOperator): incorporating Spillable data structures You can merge this pull request into a Git repository by running: $ git pull https://github.com/davidyan74/apex-malhar windowedSpillable Alternatively you can review and apply these changes as the patch at: https://github.com/apache/apex-malhar/pull/345.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #345 ---- commit c2c3f0acfcdf033a0e3044967ab3f8048f719259 Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-06-05T00:11:20Z - Intermediate commit. commit 1bee1ed0308470ff35a739ab6f9e94d53debddb8 Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-06-13T06:03:21Z Intermediate commit commit 60acf68b96f2145af3d90b410c6b20613347f881 Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-06-21T06:58:09Z Intermediate commit commit 5b9ca5e7e1e8139cc5c8ab6dd96b8a60553e38fc Author: David Yan <da...@datatorrent.com> Date: 2016-07-13T18:57:57Z Merge branch 'APEXMALHAR-2048_pull' of github.com:ilooner/incubator-apex-malhar commit b72ee18e6f7cf0a9d30b195439912d96acafb3b4 Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-07-17T21:32:34Z Added implementations of SpillableList, SpillableMap, and SpillableArrayListMultimap commit 184653a23f662f78e7e6a7e1d53ffb0efbdb7127 Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-07-18T01:56:41Z Added SpillableComplexComponentImpl commit 9f17b4ba9233e3f46746505941a378236193f719 Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-07-18T03:29:07Z Added propagating callbacks to store commit 9e637a899e6aedb4ca6495a676654bb636616267 Author: devtagare <devtag...@gmail.com> Date: 2016-05-18T22:25:56Z APEXMALHAR-2066 JdbcPolling,idempotent,partitionable commit 10fe7a14c7294c54c5d18e4d0e94882778266ac6 Author: sandeshh <sandesh.he...@gmail.com> Date: 2016-05-25T15:56:56Z Kafka 0.9.0 output operators and unit tests. 1. Abstract Base class 2. Kafka Output operator 3. Exactly Once output operator Key in the Kafka message is used by the operator to track the tuples written by it. commit 8df0de73f58c69ac457e5a28d51a7b56b5549859 Author: Chaitanya <chaita...@datatorrent.com> Date: 2016-07-13T10:16:13Z APEXMALHAR-2019 Implemented S3 Input Module commit 2ed0a102d434ace34978c6e940e04c998133fd1e Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-07-18T01:56:41Z Added SpillableComplexComponentImpl commit c60e76eee8ae5cf034d4d9984b840d3219f52594 Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-07-18T03:29:07Z Added propagating callbacks to store commit 452db2feee7cc6e186df3faf384147b41eef40ea Author: David Yan <da...@datatorrent.com> Date: 2016-07-14T00:04:20Z Split WindowedStorage interface into two interfaces for plain and key data commit be96f667238cce47957324eac9979ebafc013fc6 Author: David Yan <da...@datatorrent.com> Date: 2016-07-15T21:40:59Z check null for retractionStorage commit b7471a5df5e883b23b5a6051b974d3521ce46151 Author: David Yan <da...@datatorrent.com> Date: 2016-07-15T21:48:13Z Removed unused imports commit 361d0e152f4709dd8fda864ce94727161807a98e Author: David Yan <da...@datatorrent.com> Date: 2016-07-16T01:00:20Z added first draft implementation for spillable data structures commit ad65865ed9e5ba25d44aacaf76372e2338ef6377 Author: David Yan <da...@datatorrent.com> Date: 2016-07-18T21:04:24Z Implemented some of Tim's suggestion commit 32c2660409b3c85872002aba6b84c5c5e748f203 Author: David Yan <da...@datatorrent.com> Date: 2016-07-20T01:17:43Z moved storage initialization to setup commit 955f2d6c66be26f8f205a743eb9c49d4e8d2bdd6 Author: David Yan <da...@datatorrent.com> Date: 2016-07-20T17:50:05Z removed identifier as per tim's suggestion commit ec4e7508e2fd70030ecab256bba32be4cad8ad3a Author: David Yan <da...@datatorrent.com> Date: 2016-07-21T00:05:51Z choose a bucket automatically and added entrySet implementation for SpillableWindowedKeyedStorage commit fe41f0c20235aac3ca57facc2491ecfba11d20a7 Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-07-21T06:38:48Z Added checkpoint callbacks to spillable complex components Added some half completed tests commit dfc84a7847b4e71ec12ca8e8e1e0127732d880c0 Author: David Yan <da...@datatorrent.com> Date: 2016-07-21T18:07:47Z intermediate commit commit fe209fb72e169ae9605c826e61adfabda670c74d Author: David Yan <da...@datatorrent.com> Date: 2016-07-21T18:08:15Z Merge branch 'APEXMALHAR-2048_pull' of github.com:ilooner/incubator-apex-malhar into windowedSpillable commit 11fb1ddd24705ed215ecbe4b9da72a39a3e8a9e1 Author: David Yan <da...@datatorrent.com> Date: 2016-07-21T19:59:24Z intermediate commit commit e5cacbbc9e1ae18f5cf0938f13377f75a3a99cd2 Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-07-24T04:09:54Z Finished unit test for SpillableArrayListMultimap commit dc258b8900688264f349307737b59b096dbc3d2b Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-07-24T05:19:37Z Added unit test which uses managed state commit ed9924b810a76c39404701da81f4753ab68af5a5 Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-07-24T06:55:18Z Finished adding managed state tests for SpillableByteMap commit 43da17d9633dc2405b596b25697b6b4b0baef69f Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-07-24T16:31:53Z Added ManagedStateTests For SpillableArrayList commit 1343ccf4ccc5099d7abcf7f99ab2ed648baeef08 Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-07-24T16:49:00Z Added managed state tests for SpillableArrayListMultimap commit 57c4e5e3c3e019613f31753be5052c32ba762e53 Author: Timothy Farkas <t...@datatorrent.com> Date: 2016-07-24T16:57:35Z Added ManagedStateTest for SpillableComplexComponent ---- > implement scalable windowed storage > ----------------------------------- > > Key: APEXMALHAR-2130 > URL: https://issues.apache.org/jira/browse/APEXMALHAR-2130 > Project: Apache Apex Malhar > Issue Type: Task > Reporter: bright chen > Assignee: David Yan > > This feature is used for supporting windowing. > The storage needs to have the following features: > 1. Spillable key value storage (integrate with APEXMALHAR-2026) > 2. Upon checkpoint, it saves a snapshot for the entire data set with the > checkpointing window id. This should be done incrementally (ManagedState) to > avoid wasting space with unchanged data > 3. When recovering, it takes the recovery window id and restores to that > snapshot > 4. When a window is committed, all windows with a lower ID should be purged > from the store. > 5. It should implement the WindowedStorage and WindowedKeyedStorage > interfaces, and because of 2 and 3, we may want to add methods to the > WindowedStorage interface so that the implementation of WindowedOperator can > notify the storage of checkpointing, recovering and committing of a window. -- This message was sent by Atlassian JIRA (v6.3.4#6332)