betaedu27 opened a new issue, #30253: URL: https://github.com/apache/airflow/issues/30253
### Apache Airflow Provider(s) google ### Versions of Apache Airflow Providers Composer version composer-2.1.9-airflow-2.4.3 apache-airflow-providers-common-sql==1.3.3 apache-airflow-providers-dbt-cloud==3.0.0 apache-airflow-providers-ftp==3.3.1 apache-airflow-providers-google==8.9.0 apache-airflow-providers-hashicorp==3.2.0 apache-airflow-providers-http==4.1.1 apache-airflow-providers-imap==3.1.1 apache-airflow-providers-mysql==4.0.1 apache-airflow-providers-postgres==5.4.0 apache-airflow-providers-sendgrid==3.1.0 apache-airflow-providers-sqlite==3.3.1 apache-airflow-providers-ssh==3.4.0 ### Apache Airflow version airflow-2.4.3 ### Operating System composer-2.1.9 ### Deployment Google Cloud Composer ### Deployment details _No response_ ### What happened When running DataprocSubmitJobOperator, the job is successfully submitted to dataproc and **runs without any issues.** However, Airflow does not detect that the job has completed successfully, neither a error is generated in logs. This issue does not always occur. I am not sure when it occurs, but it appears more frequently when running several different jobs simultaneously. Airflow log: [airflow_log.txt](https://github.com/apache/airflow/files/11051523/airflow_log.txt) Dataproc log: [dataproc_log.txt](https://github.com/apache/airflow/files/11051596/dataproc_log.txt) Dataproc job rest api: [rest_api_dataproc_job.txt](https://github.com/apache/airflow/files/11051646/rest_api_dataproc_job.txt) ### What you think should happen instead _No response_ ### How to reproduce I am not sure when it occurs, but it appears more frequently when running several different jobs simultaneously. All jobs have a 1 - 2 mins of duration ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
