David Cavazos created BEAM-12455:
------------------------------------
Summary: Have a --wait_until_finish=[True|False] pipeline option
Key: BEAM-12455
URL: https://issues.apache.org/jira/browse/BEAM-12455
Project: Beam
Issue Type: Improvement
Components: sdk-py-core
Reporter: David Cavazos
Currently, if we use the `with` statement to run a pipeline, it waits until
finish by default with no way to configure it. The only way to run without
waiting is by calling `pipeline.run()` explicitly without the `with` statement
context. This is not consistent and users now have to be aware of two different
ways of calling.
Ideally, we would like to define pipelines with the `with` statement always to
have a "single way to do things". Running without waiting is useful, for
example, to run a pipeline in Cloud Build, Cloud Run, templates, or other
serverless frameworks.
Right now the only way to do it is by changing the code to remove the `with`
statement and calling `pipeline.run()` manually, which might not be always
possible. With a pipeline option we can just pass that from the command line
arguments, or even when testing.
I'm opening this for Python since many Python samples would greatly benefit
from this, but I can see this being useful in Java and Go as well.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
