nathadfield opened a new pull request, #34208:
URL: https://github.com/apache/airflow/pull/34208
closes: #34162
As mentioned on the issue, currently if a BigQuery job running in deferrable
mode ends unsuccessfully during execution - perhaps by cancellation in the
BigQuery UI or if the job runs out of resources - then the only message that is
returned to logs is the rather unhelpful
```
[2023-09-08, 11:43:22 UTC] {taskinstance.py:1939} ERROR - Task failed with
exception
Traceback (most recent call last):
File "/opt/airflow/airflow/models/baseoperator.py", line 1613, in
resume_execution
return execute_callable(context)
File "/opt/airflow/airflow/providers/google/cloud/operators/bigquery.py",
line 2881, in execute_complete
raise AirflowException(event["message"])
KeyError: 'message'
```
This is due to the fact that the async function `get_job_status` only
returns the string `error` and the actual error message is left behind.
The change involved here is to return a dictionary object that returns both
`status` and a `message` which, when a BQ job is in a `DONE` state but contains
a `errorResult`, then the specific error is carried through.
This then results in the following error output when the job is cancelled
from the UI.
```
[2023-09-08, 08:21:20 UTC] {taskinstance.py:1939} ERROR - Task failed with
exception
Traceback (most recent call last):
File "/opt/airflow/airflow/models/baseoperator.py", line 1613, in
resume_execution
return execute_callable(context)
File "/opt/airflow/airflow/providers/google/cloud/operators/bigquery.py",
line 2881, in execute_complete
raise AirflowException(event["message"])
airflow.exceptions.AirflowException: Job execution was cancelled: User
requested cancellation
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
