GitHub user xinyuiscool opened a pull request:
https://github.com/apache/samza/pull/236
SAMZA-1321: Propagate end-of-stream and watermark messages
The patch completes the end-of-stream work flow across multi-stage
pipeline. It also contains initial commit for supporting watermarks. For
watermark, there are issues raised in the review feedback and will be addressed
by further prs. The main logic this patch adds:
- EndOfStreamManager aggregates the end-of-stream control messages,
propagate the result to to downstream intermediate topics based on the topology
of the IO in the StreamGraph.
- WatermarkManager aggregates the watermark control messages from the
upstage tasks, pass it through the operators, and propagate it to downstream.
In operator impl, I implemented similar watermark logic as Beam for
watermark propagation:
* InputWatermark(op) = min { OutputWatermark(op') | op1 is upstream of op}
* OutputWatermark(op) = min { InputWatermark(op), OldestWorkTime(op) }
Add quite a few unit tests and integration test. The code is 100% covered
as reported by Intellij. Both control messages work as expected.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xinyuiscool/samza SAMZA-1321
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/samza/pull/236.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #236
----
commit f0bb67f26204fc5698f5d9b5fbdac8b90c70fabe
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-05-26T22:01:05Z
Add control message types and serde
commit 2d79ed7a8a6f4698904232899ef485c4f3a1e627
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-05-30T22:41:12Z
Clean up the ControlMessage class and some enhancements
commit f0e32d517942fb119253e2d4f06fc921c34ec0ba
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-05-31T00:52:49Z
Merge branch 'master' into SAMZA-1312
commit 80f6255a57715501c667450606af8b24595f7df5
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-05-31T01:05:15Z
remove unused file
commit 319cffdaca86e422c0a6c8f022cf931ad67404c7
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-05-31T17:01:54Z
Fix checkstyle issues
commit 276410188069ab84b78f3cf3328aa3c64a10976e
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-01T18:30:04Z
Updates to address Jagadish's feedback
commit 12c686ad42cd84a8bd87731319cf193fc0f7ab55
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-07T00:59:39Z
progagate end-of-stream
commit bdaa5126ff6bee4335253fd9d357e5bcd026b1ba
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-07T01:28:57Z
Add end-of-stream message to initial envelope from the consumer
commit 948fd925ca0889c4fe28f38573630363307789a4
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-07T20:54:59Z
Add dispatcher
commit a797372ab815b61475793ae708d8b8f568fb6d78
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-07T20:55:32Z
Merge branch 'master' into SAMZA-1321
commit 9cc5e10ba86334d1f77de90a3327d3ce7b3fcb06
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-08T17:33:17Z
Add logic to handle and dispatch eos messages
commit c691d0a81f403a8c303e15e41dd16dc8fb7cd19d
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-08T18:42:26Z
Wire up the eos processing inside the runloop
commit fd05bc721c9314fc37967bede0f41d96e01aa8c8
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-08T20:43:51Z
Clean up
commit 224c7d78d0a7d8bcf821280c2bcf3dbc40ef935b
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-13T00:18:33Z
Fix bugs found in the integration tests
commit 898aadccce48744c26998f3875d8f4dc2be63cbb
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-13T16:44:04Z
Fix check style errors
commit 9937d2429be3b3215a8d75763c9cfd7ebed79bee
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-13T16:49:31Z
Merge branch 'master' into SAMZA-1321
commit d6fa550fa0703ec0dff40e08bef53def43aa41fb
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-14T01:27:41Z
Add unit tests
commit dcc6d79d415102c009cf3ccffc98c46323bca6c7
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-14T19:04:37Z
Add more tests
commit a37d62b4f354d907d6d5b4d6f15439428f8a0597
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-14T23:32:53Z
Fix the bug where the EndOfStreamManager needs the container model to
figure out the tasks publishing to a int stream
commit 4e3063b579b388ff981c19369ada77b8b614f092
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-19T17:38:28Z
Merge branch 'master' into SAMZA-1321
commit 18add5a0f65d6e2876c53adc3094d7a23b847ebe
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-22T01:08:18Z
Support watermarks!
commit 390ad484bde82a38614524db946439446de046d1
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-22T23:07:40Z
Add unit tests and integration tests
commit 146eabe8951ba94e81342d732ac8733eb30584f7
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-24T01:16:59Z
Address Yi's feedback
commit a4314384be3a3459a0aa9392557c8a93352e063c
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-24T01:25:03Z
Fix style check error
commit 5a8898fc9ac18b5473f20cad9d13db003436735d
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-27T01:24:47Z
Clean up based on Yi's feedback
commit a21627b28ec7b444dd7e1262d3266237920fe547
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-28T19:51:19Z
Address Yi and Chris's feedback
commit e937cde3c815d0b241413693ffe4024a5933392b
Author: Xinyu Liu <xi...@xiliu-ld.linkedin.biz>
Date: 2017-06-28T20:50:39Z
Merge branch 'master' into SAMZA-1321
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
