GitHub user joseph-torres opened a pull request:
https://github.com/apache/spark/pull/19239
[SPARK-22017] Take minimum of all watermark execs in StreamExecution.
## What changes were proposed in this pull request?
Take the minimum of all watermark exec nodes as the "real" watermark in
StreamExecution, rather than picking one arbitrarily.
## How was this patch tested?
new unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/joseph-torres/spark SPARK-22017
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19239.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19239
----
commit 4a7f53fdab1e5e640e156a4a3d2ba27837195195
Author: Jose Torres <[email protected]>
Date: 2017-09-13T21:49:23Z
Implement multiple watermark StreamExecution support.
commit 9b9cd19106fae9a2de268eb2b559ca1bf159e9c2
Author: Jose Torres <[email protected]>
Date: 2017-09-14T18:30:40Z
partially fix test
commit 6a4c80b696f42a445c7f846fada3f823e04bd3ab
Author: Jose Torres <[email protected]>
Date: 2017-09-14T21:52:16Z
Finish rewriting test
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
