GitHub user joseph-torres reopened a pull request:
https://github.com/apache/spark/pull/19239
[SPARK-22017] Take minimum of all watermark execs in StreamExecution.
## What changes were proposed in this pull request?
Take the minimum of all watermark exec nodes as the "real" watermark in
StreamExecution, rather than picking one arbitrarily.
## How was this patch tested?
new unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/joseph-torres/spark SPARK-22017
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/19239.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #19239
----
commit 4a7f53fdab1e5e640e156a4a3d2ba27837195195
Author: Jose Torres <[email protected]>
Date: 2017-09-13T21:49:23Z
Implement multiple watermark StreamExecution support.
commit 9b9cd19106fae9a2de268eb2b559ca1bf159e9c2
Author: Jose Torres <[email protected]>
Date: 2017-09-14T18:30:40Z
partially fix test
commit 6a4c80b696f42a445c7f846fada3f823e04bd3ab
Author: Jose Torres <[email protected]>
Date: 2017-09-14T21:52:16Z
Finish rewriting test
commit 484940e5eb4d1eac1c5ec81f475681c9241bbab2
Author: Jose Torres <[email protected]>
Date: 2017-09-14T22:24:36Z
make IncrementalExecution.offsetSeqMetadata non-private
commit 032f55503c8d424390da1ff85054e3a01e7489eb
Author: Jose Torres <[email protected]>
Date: 2017-09-14T23:22:22Z
properly name test dataframes
commit d7f5f60c6be5bf228c960c3549eb81ed869f0227
Author: Jose Torres <[email protected]>
Date: 2017-09-14T23:39:22Z
Combine test helper functions.
commit 2f07f90423d87985322975f8ad5aef8f70f28066
Author: Jose Torres <[email protected]>
Date: 2017-09-15T01:18:12Z
Key watermarks by relative position rather than attribute.
commit 8b605384d77fdeb63b28feabee74284a5ab1409a
Author: Jose Torres <[email protected]>
Date: 2017-09-15T02:05:14Z
Address test comments.
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
