[
https://issues.apache.org/jira/browse/BEAM-9900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-9900:
----------------------------------
Status: Open (was: Triage Needed)
> Remove the need for shutdownSourcesOnFinalWatermark flag
> --------------------------------------------------------
>
> Key: BEAM-9900
> URL: https://issues.apache.org/jira/browse/BEAM-9900
> Project: Beam
> Issue Type: Improvement
> Components: runner-flink
> Reporter: Maximilian Michels
> Assignee: Maximilian Michels
> Priority: Major
>
> The {{shutdownSourcesOnFinalWatermark}} has caused some confusion in the
> past. It is generally used for testing pipelines to ensure that the pipeline
> and the testing cluster shuts down at the end of the job. Without it, the
> pipeline will run forever in streaming mode, regardless of whether the input
> is finite or not.
> We didn't want to enable the flag by default because shutting down any
> operators including sources in Flink will prevent checkpointing from working.
> If we have side input, for example, that may be the case even for
> long-running pipelines. However, we can detect whether checkpointing is
> enabled and set the flag automatically.
> The only situation where we may want the flag to be disabled is when users do
> not have checkpointing enabled but want to take a savepoint. This should be
> rare and users can mitigate by setting the flag to false to prevent operators
> from shutting down.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
