mik-laj commented on a change in pull request #6386: [AIRFLOW-5716][part of
AIRFLOW-5697][depends on AIRFLOW-5711] Simplify DataflowJobsController logic
URL: https://github.com/apache/airflow/pull/6386#discussion_r344175358
##########
File path: airflow/gcp/hooks/dataflow.py
##########
@@ -334,15 +357,16 @@ def _start_dataflow(
name: str,
command_prefix: List[str],
label_formatter: Callable[[Dict], List[str]],
- multiple_jobs: bool = False
+ project_id: str,
+ multiple_jobs: bool = False,
) -> None:
variables = self._set_variables(variables)
- cmd = command_prefix + self._build_cmd(variables, label_formatter)
+ cmd = command_prefix + self._build_cmd(variables, label_formatter,
project_id)
runner = _DataflowRunner(cmd)
job_id = runner.wait_for_done()
Review comment:
We can't split this one method into two methods because a process is being
run that supervisor the task. Unfortunately, this is a limitation of Apache
Beam, which does not have the option of forcing external supervision. In any
case, we must wait until the Apache Beam system process is completed to be sure
of completing the job.
This operator can also be used to initiate streaming jobs, but we lack the
operator to stop the task if we want to handle your process fully.
https://github.com/apache/airflow/blob/master/tests/gcp/hooks/test_dataflow.py#L428-L458
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
