[
https://issues.apache.org/jira/browse/APEXMALHAR-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15402612#comment-15402612
]
ASF GitHub Bot commented on APEXMALHAR-2130:
--------------------------------------------
GitHub user davidyan74 reopened a pull request:
https://github.com/apache/apex-malhar/pull/345
APEXMALHAR-2130 REVIEW ONLY (WindowedOperator): incorporating Spillable
data structures
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/davidyan74/apex-malhar windowedSpillable
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/apex-malhar/pull/345.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #345
----
commit c2c3f0acfcdf033a0e3044967ab3f8048f719259
Author: Timothy Farkas <[email protected]>
Date: 2016-06-05T00:11:20Z
- Intermediate commit.
commit 1bee1ed0308470ff35a739ab6f9e94d53debddb8
Author: Timothy Farkas <[email protected]>
Date: 2016-06-13T06:03:21Z
Intermediate commit
commit 60acf68b96f2145af3d90b410c6b20613347f881
Author: Timothy Farkas <[email protected]>
Date: 2016-06-21T06:58:09Z
Intermediate commit
commit 5b9ca5e7e1e8139cc5c8ab6dd96b8a60553e38fc
Author: David Yan <[email protected]>
Date: 2016-07-13T18:57:57Z
Merge branch 'APEXMALHAR-2048_pull' of
github.com:ilooner/incubator-apex-malhar
commit b72ee18e6f7cf0a9d30b195439912d96acafb3b4
Author: Timothy Farkas <[email protected]>
Date: 2016-07-17T21:32:34Z
Added implementations of SpillableList, SpillableMap, and
SpillableArrayListMultimap
commit 184653a23f662f78e7e6a7e1d53ffb0efbdb7127
Author: Timothy Farkas <[email protected]>
Date: 2016-07-18T01:56:41Z
Added SpillableComplexComponentImpl
commit 9f17b4ba9233e3f46746505941a378236193f719
Author: Timothy Farkas <[email protected]>
Date: 2016-07-18T03:29:07Z
Added propagating callbacks to store
commit 9e637a899e6aedb4ca6495a676654bb636616267
Author: devtagare <[email protected]>
Date: 2016-05-18T22:25:56Z
APEXMALHAR-2066 JdbcPolling,idempotent,partitionable
commit 10fe7a14c7294c54c5d18e4d0e94882778266ac6
Author: sandeshh <[email protected]>
Date: 2016-05-25T15:56:56Z
Kafka 0.9.0 output operators and unit tests.
1. Abstract Base class
2. Kafka Output operator
3. Exactly Once output operator
Key in the Kafka message is used by the operator to track the tuples
written by it.
commit 8df0de73f58c69ac457e5a28d51a7b56b5549859
Author: Chaitanya <[email protected]>
Date: 2016-07-13T10:16:13Z
APEXMALHAR-2019 Implemented S3 Input Module
commit 2ed0a102d434ace34978c6e940e04c998133fd1e
Author: Timothy Farkas <[email protected]>
Date: 2016-07-18T01:56:41Z
Added SpillableComplexComponentImpl
commit c60e76eee8ae5cf034d4d9984b840d3219f52594
Author: Timothy Farkas <[email protected]>
Date: 2016-07-18T03:29:07Z
Added propagating callbacks to store
commit 452db2feee7cc6e186df3faf384147b41eef40ea
Author: David Yan <[email protected]>
Date: 2016-07-14T00:04:20Z
Split WindowedStorage interface into two interfaces for plain and key data
commit be96f667238cce47957324eac9979ebafc013fc6
Author: David Yan <[email protected]>
Date: 2016-07-15T21:40:59Z
check null for retractionStorage
commit b7471a5df5e883b23b5a6051b974d3521ce46151
Author: David Yan <[email protected]>
Date: 2016-07-15T21:48:13Z
Removed unused imports
commit 361d0e152f4709dd8fda864ce94727161807a98e
Author: David Yan <[email protected]>
Date: 2016-07-16T01:00:20Z
added first draft implementation for spillable data structures
commit ad65865ed9e5ba25d44aacaf76372e2338ef6377
Author: David Yan <[email protected]>
Date: 2016-07-18T21:04:24Z
Implemented some of Tim's suggestion
commit 32c2660409b3c85872002aba6b84c5c5e748f203
Author: David Yan <[email protected]>
Date: 2016-07-20T01:17:43Z
moved storage initialization to setup
commit 955f2d6c66be26f8f205a743eb9c49d4e8d2bdd6
Author: David Yan <[email protected]>
Date: 2016-07-20T17:50:05Z
removed identifier as per tim's suggestion
commit ec4e7508e2fd70030ecab256bba32be4cad8ad3a
Author: David Yan <[email protected]>
Date: 2016-07-21T00:05:51Z
choose a bucket automatically and added entrySet implementation for
SpillableWindowedKeyedStorage
commit fe41f0c20235aac3ca57facc2491ecfba11d20a7
Author: Timothy Farkas <[email protected]>
Date: 2016-07-21T06:38:48Z
Added checkpoint callbacks to spillable complex components
Added some half completed tests
commit dfc84a7847b4e71ec12ca8e8e1e0127732d880c0
Author: David Yan <[email protected]>
Date: 2016-07-21T18:07:47Z
intermediate commit
commit fe209fb72e169ae9605c826e61adfabda670c74d
Author: David Yan <[email protected]>
Date: 2016-07-21T18:08:15Z
Merge branch 'APEXMALHAR-2048_pull' of
github.com:ilooner/incubator-apex-malhar into windowedSpillable
commit 11fb1ddd24705ed215ecbe4b9da72a39a3e8a9e1
Author: David Yan <[email protected]>
Date: 2016-07-21T19:59:24Z
intermediate commit
commit e5cacbbc9e1ae18f5cf0938f13377f75a3a99cd2
Author: Timothy Farkas <[email protected]>
Date: 2016-07-24T04:09:54Z
Finished unit test for SpillableArrayListMultimap
commit dc258b8900688264f349307737b59b096dbc3d2b
Author: Timothy Farkas <[email protected]>
Date: 2016-07-24T05:19:37Z
Added unit test which uses managed state
commit ed9924b810a76c39404701da81f4753ab68af5a5
Author: Timothy Farkas <[email protected]>
Date: 2016-07-24T06:55:18Z
Finished adding managed state tests for SpillableByteMap
commit 43da17d9633dc2405b596b25697b6b4b0baef69f
Author: Timothy Farkas <[email protected]>
Date: 2016-07-24T16:31:53Z
Added ManagedStateTests For SpillableArrayList
commit 1343ccf4ccc5099d7abcf7f99ab2ed648baeef08
Author: Timothy Farkas <[email protected]>
Date: 2016-07-24T16:49:00Z
Added managed state tests for SpillableArrayListMultimap
commit 57c4e5e3c3e019613f31753be5052c32ba762e53
Author: Timothy Farkas <[email protected]>
Date: 2016-07-24T16:57:35Z
Added ManagedStateTest for SpillableComplexComponent
----
> implement scalable windowed storage
> -----------------------------------
>
> Key: APEXMALHAR-2130
> URL: https://issues.apache.org/jira/browse/APEXMALHAR-2130
> Project: Apache Apex Malhar
> Issue Type: Task
> Reporter: bright chen
> Assignee: David Yan
>
> This feature is used for supporting windowing.
> The storage needs to have the following features:
> 1. Spillable key value storage (integrate with APEXMALHAR-2026)
> 2. Upon checkpoint, it saves a snapshot for the entire data set with the
> checkpointing window id. This should be done incrementally (ManagedState) to
> avoid wasting space with unchanged data
> 3. When recovering, it takes the recovery window id and restores to that
> snapshot
> 4. When a window is committed, all windows with a lower ID should be purged
> from the store.
> 5. It should implement the WindowedStorage and WindowedKeyedStorage
> interfaces, and because of 2 and 3, we may want to add methods to the
> WindowedStorage interface so that the implementation of WindowedOperator can
> notify the storage of checkpointing, recovering and committing of a window.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
