pmcquighan-camus opened a new issue, #77269: URL: https://github.com/apache/airflow/issues/77269
### Under which category would you file this issue? Providers ### Apache Airflow version 3.4.0 ### What happened and how to reproduce it? On job creation, the operator will call get pods [here](https://github.com/apache/airflow/blob/providers-cncf-kubernetes/11.0.0/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/job.py#L225), which has support for retries, but does not sleep between these retries [here](https://github.com/apache/airflow/blob/providers-cncf-kubernetes/11.0.0/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/job.py#L636-L646). What happened in my scenario is that the job was created successfully, but then for whatever reason GKE was slow about getting a pod created, and so no pod existed for about 5 seconds. Airflow tried listing the pod 3 times (default retry count) within about 1 second, didn't see the pod and then failed with `No pods running with labels ...` ### What you think should happen instead? Ideally with a retry mechanism such as this there should be exponential backoff (or at least some amount of waiting before retries). For example, in addition to having [discover_pods_retry_number](https://github.com/apache/airflow/blob/providers-cncf-kubernetes/11.0.0/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/job.py#L127) there could also be a `discover_pods_backoff_seconds` parameter, with a wait happening between failed attempts. I don't know if this can be handled async ### Operating System _No response_ ### Deployment Official Apache Airflow Helm Chart ### Apache Airflow Provider(s) cncf-kubernetes ### Versions of Apache Airflow Providers 10.25.0 ### Official Helm Chart version _No response_ ### Kubernetes Version _No response_ ### Helm Chart configuration _No response_ ### Docker Image customizations _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
