[
https://issues.apache.org/jira/browse/BEAM-604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Davor Bonaci updated BEAM-604:
------------------------------
Component/s: runner-dataflow
> Use Watermark Check Streaming Job Finish in TestDataflowRunner
> --------------------------------------------------------------
>
> Key: BEAM-604
> URL: https://issues.apache.org/jira/browse/BEAM-604
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow
> Reporter: Mark Liu
> Assignee: Mark Liu
> Priority: Minor
> Fix For: 0.3.0-incubating
>
>
> Currently, streaming job with bounded input can't be terminated automatically
> and TestDataflowRunner can't handle this case. Need to update
> TestDataflowRunner so that streaming integration test such as
> WindowedWordCountIT can run with it.
> Implementation:
> Query watermark of each step and wait until all watermarks set to MAX then
> cancel the job.
> Update:
> Suggesting by [~pei...@gmail.com], implement checkMaxWatermark in
> DataflowPipelineJob#waitUntilFinish. Thus, all dataflow streaming jobs with
> bounded input will take advantage of this change and are canceled
> automatically when watermarks reach to max value. Also Dataflow runners can
> keep simple and free from handling batch and streaming two cases.
> Update:
> Pipeline author should have control on whether or not canceling streaming job
> and when. Test framework is a better place to auto-cancel streaming test job
> when curtain conditions meet, rather than in waitUntilFinish().
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
