Aviem Zur created BEAM-2847:
-------------------------------
Summary: Stop streaming pipeline when global watermark reaches
+INF
Key: BEAM-2847
URL: https://issues.apache.org/jira/browse/BEAM-2847
Project: Beam
Issue Type: Improvement
Components: runner-spark
Reporter: Aviem Zur
Assignee: Amit Sela
Stop streaming pipeline when global watermark reaches +INF as suggested in
BEAM-849 comments.
We can do this similarly to the solution in
[TestSparkRunner#awaitWatermarksOrTimeout|https://github.com/apache/beam/blob/v2.1.0/runners/spark/src/main/java/org/apache/beam/runners/spark/TestSparkRunner.java#L158-L179]
by polling {code}
SparkTimerInternals.global(GlobalWatermarkHolder.get(...))
{code}
We can also remove this code from {{TestSparkRunner}} if we make sure all
sources in test behave in a way which shifts the watermark to +INF at the end
of the test.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
