mik-laj commented on a change in pull request #11726:
URL: https://github.com/apache/airflow/pull/11726#discussion_r509662589
##########
File path: airflow/providers/google/cloud/hooks/dataflow.py
##########
@@ -268,9 +271,8 @@ def _check_dataflow_job_state(self, job) -> bool:
raise Exception("Google Cloud Dataflow job {} has
failed.".format(job['name']))
elif DataflowJobStatus.JOB_STATE_CANCELLED == job['currentState']:
raise Exception("Google Cloud Dataflow job {} was
cancelled.".format(job['name']))
- elif (
- DataflowJobStatus.JOB_STATE_RUNNING == job['currentState']
- and DataflowJobType.JOB_TYPE_STREAMING == job['type']
+ elif DataflowJobStatus.JOB_STATE_RUNNING == job['currentState'] and (
Review comment:
I've thought about it a bit longer, and I think we can provide a little
more flexibility here if we don't set a single default for wait_until_finish.
What do you think about implementing the following rules?
- If the user has passed the wait_until_finish parameter, this value is used.
- If the wait_until_finish parameter is empty and the job is of the
streaming type, NOT wait for the finish.
- If the wait_until_finish parameter is empty and the job is of the batch
type, wait for the finish.
```python
if self._wait_until_finish is None:
wait_until_finish = DataflowJobType.JOB_TYPE_STREAMING != job['type']
else:
wait_until_finish = self._wait_until_finish
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
