akspow opened a new issue #10030:
URL: https://github.com/apache/airflow/issues/10030
I am using the CloudDataFusionStartPipelineOperator in Google cloud composer
and success_states = ["COMPLETED"] parameter to wait for the pipeline to be
successful. It works fine, but sometimes I receive this error:
```
[2020-07-28 08:16:46,390] {taskinstance.py:1059} ERROR - Retrieving a
pipeline state failed with code 404
Traceback (most recent call last)
File "/usr/local/lib/airflow/airflow/models/taskinstance.py", line 930, in
_run_raw_tas
result = task_copy.execute(context=context
File
"/usr/local/lib/airflow/airflow/providers/google/cloud/operators/datafusion.py",
line 700, in execut
runtime_args=self.runtime_args
File
"/usr/local/lib/airflow/airflow/providers/google/cloud/hooks/datafusion.py",
line 498, in start_pipelin
instance_url=instance_url
File
"/usr/local/lib/airflow/airflow/providers/google/cloud/hooks/datafusion.py",
line 114, in wait_for_pipeline_stat
namespace=namespace
File
"/usr/local/lib/airflow/airflow/providers/google/cloud/hooks/datafusion.py",
line 442, in _get_workflow_stat
f"Retrieving a pipeline state failed with code {response.status}
airflow.exceptions.AirflowException: Retrieving a pipeline state failed with
code 404
```
But the Datafusion pipeline gets completed successfully.
It would be helpful if someone resolves this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
