mo-rieger opened a new issue, #70585:
URL: https://github.com/apache/airflow/issues/70585
### Under which category would you file this issue?
Providers
### Apache Airflow version
2.11.2
### What happened and how to reproduce it?
`get_pods()` issues `discover_pods_retry_number + 1` (default 4)
`list_namespaced_pod` calls back to back with **no delay**, then raises `No
pods running with labels ...` if the pods are not visible yet. Raising
`discover_pods_retry_number` only adds more immediate GETs — there is no way to
wait.
Production trace for one occurrence:
| time (UTC) | event |
|---|---|
| `00:00:23.223` | `job.py` — `Building job job-<task>-<suffix>` |
| `00:00:23.360` | task failed — `AirflowException: No pods running with
labels ...` |
| `00:00:23` | Job controller — `SuccessfulCreate: Created pod
job-<task>-<suffix>-<id>` |
| `00:01:28` | Job controller — `Job completed` |
`kube_job_created` and `kube_pod_created` are both `00:00:23`: the pod
existed in the same second the operator gave up, 137 ms after the Job was
built. The workload then ran to success unmonitored. ~20 occurrences in 5 days
across 8+ unrelated DAGs.
Consequences:
1. The task is marked failed while its Job succeeds. Nothing waits on,
monitors or cleans up the Job; `on_kill()` does not fire.
2. `self.pods` is never assigned, so `_cleanup_monitoring_pods()` in the
`finally` block no-ops.
3. A retry or manual clear starts a **second** Job while the first is still
running or has already committed its writes — silent double execution for
non-idempotent tasks.
The message is also inaccurate: the pods are not "not running", they do not
exist yet.
### What you think should happen instead?
Wait for the pods, bounded by a timeout. No new parameters needed —
`KubernetesJobOperator` extends `KubernetesPodOperator`, so
`schedule_timeout_seconds` (deadline, default 120 via
`startup_timeout_seconds`) and `startup_check_interval_seconds` (interval,
default 5) are already inherited and documented. The first probe stays
immediate, so tasks whose pods appear at once are unaffected.
Also: break on `>=` parallelism rather than `==`, and deprecate
`discover_pods_retry_number` as a no-op.
### Operating System
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
cncf provider at 10.13.0 but related code is unchanged in current main
### Official Helm Chart version
1.19.0
### Kubernetes Version
1.35.6
### Helm Chart configuration
Not Applicable
### Docker Image customizations
Not Applicable
### Anything else?
The issue is Inherited by `KubernetesStartKueueJobOperator`,
`GKEStartJobOperator`, `GKEStartKueueJobOperator` — i.e. exactly the Kueue case
where pod creation is slowest.
The issue is known and was raised in
https://github.com/apache/airflow/pull/53368 but rejected as out of scope for
this PR.
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
