eladkal commented on code in PR #44279:
URL: https://github.com/apache/airflow/pull/44279#discussion_r1853912328
##########
providers/src/airflow/providers/google/cloud/operators/bigquery.py:
##########
@@ -2592,8 +2592,15 @@ def _submit_job(
nowait=True,
)
- @staticmethod
- def _handle_job_error(job: BigQueryJob | UnknownJob) -> None:
+ def _handle_job_error(self, job: BigQueryJob | UnknownJob) -> None:
+ self.log.info("Job %s is completed. Checking the job status",
self.job_id)
+ # I've noticed that sometimes BigQuery jobs transiently report the
wrong status, causing the Airflow job to be incorrectly marked as successful.
+ # To avoid this, we refresh the job properties before checking the
final state and handling any errors.
+ while job.state != "DONE":
Review Comment:
I don't follow why we should do this.
Is there even a bug report to BigQuery?
I am very against such permanent workarounds. Airflow should not be the
fixer for upstream library bugs. Its OK to do this a temporary workaround when
upstream library is engaged and working on a fix but other not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
