[
https://issues.apache.org/jira/browse/BEAM-604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450084#comment-15450084
]
Pei He commented on BEAM-604:
-----------------------------
Consider implement it in DataflowPipelineJob.waitUntilFinish()
https://github.com/apache/incubator-beam/blob/master/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineJob.java#L208
And, TestDataflowRunner can then use it.
But, you need to think about how it will interact with waitForAssertions.
> Use Watermark Check Streaming Job Finish in TestDataflowRunner
> ---------------------------------------------------------------
>
> Key: BEAM-604
> URL: https://issues.apache.org/jira/browse/BEAM-604
> Project: Beam
> Issue Type: Improvement
> Reporter: Mark Liu
> Assignee: Mark Liu
> Priority: Minor
>
> Currently, streaming job with bounded input can't be terminated automatically
> and TestDataflowRunner can't handle this case. Need to update
> TestDataflowRunner so that streaming integration test such as
> WindowedWordCountIT can run with it.
> Implementation:
> Query watermark of each step and wait until all watermarks set to MAX then
> cancel the job.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
