[
https://issues.apache.org/jira/browse/AIRFLOW-2774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16714830#comment-16714830
]
Evgeny Podlepaev commented on AIRFLOW-2774:
-------------------------------------------
[~xnuinside] great, there can be a choice of different engines. My motivation
was to facilitate debugging/smoke testing of an Airflow pipeline end to end. It
just takes unnecessary time to wait for Dataflow to start remotely. In many
cases, local execution would be enough and there should be an option for that.
> DataFlowPythonOperator needs to support DirectRunner to speed up end-to-end
> testing of Airflow dag
> --------------------------------------------------------------------------------------------------
>
> Key: AIRFLOW-2774
> URL: https://issues.apache.org/jira/browse/AIRFLOW-2774
> Project: Apache Airflow
> Issue Type: Improvement
> Components: Dataflow
> Affects Versions: 1.9.0
> Reporter: Evgeny Podlepaev
> Priority: Minor
>
> **DataFlowPythonOperator needs to support DirectRunner as a runner option to
> facilitate local end-to-end testing of the entire Airflow dag. Right now if
> DirectRunner is set via job options, the DataFlowHook will wait infinitely
> trying to get status of the remote job which does not exist:
> _DataflowJob(self.get_conn(), variables['project'], name,
> variables['region'], self.poll_sleep).wait_for_done()
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
