kandharvishnu opened a new issue, #40839: URL: https://github.com/apache/airflow/issues/40839
### Apache Airflow Provider(s) google ### Versions of Apache Airflow Providers apache-airflow-providers-google==10.12.0 ### Apache Airflow version 2.7.3 ### Operating System debian ### Deployment Astronomer ### Deployment details _No response_ ### What happened When the [BigQueryInsertJobOperator](https://github.com/apache/airflow/blob/4298c433bc8b3136b8131ebe006dcf10c601e9c1/airflow/providers/google/cloud/operators/bigquery.py#L2654) submits a job to BigQuery, there are instances where the TaskInstance in Airflow is marked as success rarely before the task actually completes in BigQuery. When the execute method of a TaskInstance’s Operator completes without raising an exception, the TaskInstance is marked as success. Since these TaskInstances are being marked as success, it implies that the BigQueryInsertJobOperator’s execute method is completing without any exceptions. When [QueryJob.result](https://github.com/googleapis/python-bigquery/blob/49bfd124cc76a719acf9257610181612d1452e56/google/cloud/bigquery/job/query.py#L1463) returns something, allowing the execute method to finish. ### What you think should happen instead Need to add an additional check [here](https://github.com/apache/airflow/blob/4298c433bc8b3136b8131ebe006dcf10c601e9c1/airflow/providers/google/cloud/operators/bigquery.py#L2866) to verify if the Job state is DONE in BigQuery, ensuring that Airflow only marks the TaskInstance as success when the job is indeed complete in BigQuery. ### How to reproduce I haven't been able to replicate the issue. It happens infrequently. ### Anything else _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
