GitHub user jose-torres opened a pull request:
https://github.com/apache/spark/pull/20859
[SPARK-23702][SS] Forbid watermarks on both sides of stateful streaming
operators.
## What changes were proposed in this pull request?
Forbid watermarks on both sides of stateful streaming operators.
Multiple sequential watermarks are in general not supported by the
execution engine; support is only in parallel, e.g. on both sides of a join. We
can normally resolve this by simply picking the topmost watermark operator and
ignoring the rest, but this is not semantically valid when there's a stateful
operator in between.
## How was this patch tested?
new unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jose-torres/spark triggerDisallow
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/20859.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #20859
----
commit 051c85afe727f39ba9d505e00e162620f69c808f
Author: Jose Torres <torres.joseph.f+github@...>
Date: 2018-03-19T18:48:11Z
disallow watermarks on both sides of stateful
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
