shahar1 opened a new issue, #75013:
URL: https://github.com/apache/airflow/issues/75013
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.2.0 (constraints pin `apache-airflow-providers-cncf-kubernetes==10.14.0`).
The affected code path is unchanged on `main`.
### What happened and how to reproduce it?
Since `apache-airflow-providers-cncf-kubernetes` 10.6.0 (#50192), the
synchronous `KubernetesPodOperator` starts a background event watcher while it
waits for the pod to leave `Pending`. `PodManager.watch_pod_events` calls
`list_namespaced_event(namespace, field_selector="involvedObject.name=<pod>")`
every `startup_check_interval_seconds` (default 5 s) for every running task.
Before 10.6.0 (for example 8.3.3, pinned by Airflow 2.9.3) the startup loop
only issued `read_namespaced_pod`, a cheap GET by name.
Two problems follow from this on a large, volatile cluster (reported at ~10
000 concurrent Airflow + Spark pods on EKS 1.36 with the official Helm chart):
1. **The event LIST is expensive and scales with the namespace, not with the
pod.** kube-apiserver has no field index for `Event` objects
(`pkg/registry/core/event/storage/storage.go` registers only
`Matcher`/`GetAttrs`, no `Indexers`/`TriggerFunc`), so every call scans every
event in the namespace and filters afterwards. With thousands of pods, a
one-hour event TTL and ~5–8 events per pod, each call scans hundreds of
thousands of objects, multiplied by every starting task every 5 s. The
apiserver request timeout trips and the client sees:
```
WARNING - Retrying
airflow.providers.cncf.kubernetes.utils.pod_manager.PodManager.read_pod_events
in 8 seconds as it raised ApiException: (504)
Reason: Gateway Timeout
HTTP response body: {"metadata":{},"status":"Failure","message":"Timeout:
request did not complete within the allotted
timeout","reason":"Timeout","details":{},"code":504}
```
On 10.14.0 the first poll (and every poll until the first event item
arrives, see #66470, fixed in 10.17.1 by #66471) is additionally sent without
`resourceVersion`, which forces a quorum read instead of a watch-cache read.
2. **A failure of the event watcher fails the task even though the pod is
fine.** `KubernetesPodOperator.await_pod_start` runs the watcher as an
`asyncio` task and, in its `finally` block, does `events_task.cancel()`
followed by `await events_task` with only `CancelledError` suppressed. When the
watcher has already died with `ApiException` after the `[workers] api_retries`
attempts, that `await` re-raises it, `asyncio.run` propagates it, and the task
fails. Logging events during startup is a convenience; it should never decide
the task outcome. The one-shot read behind `log_events_on_failure` already uses
`_optionally_suppress(reraise=False)`; the startup watcher does not.
There is no configuration or operator parameter that disables the startup
event watcher. `log_events_on_failure` only controls the one-shot read after a
failure. `SparkKubernetesOperator` inherits the same behaviour.
Reproduction without a 10 000-pod cluster: make `list_namespaced_event`
raise `ApiException(status=504)` (for example with a mock, or an RBAC/admission
rule that rejects event lists) and run any `KubernetesPodOperator` task with
`deferrable=False`. The pod starts and completes, yet the task fails with the
`ApiException` once the retries are exhausted.
Workaround until this is fixed (the loop in `PodManager.watch_pod_events`
checks `stop_watching_events` before every poll):
```python
class QuietKubernetesPodOperator(KubernetesPodOperator):
def await_pod_start(self, pod):
self.pod_manager.stop_watching_events = True
super().await_pod_start(pod)
```
Other mitigations: raise `startup_check_interval_seconds`, keep Spark
executor pods in a separate namespace from operator pods (the scan cost is per
namespace), or use `deferrable=True` (the triggerer uses a watch stream instead
of polling lists).
### What you think should happen instead?
1. The startup event watcher must be best-effort: an API error after retries
should log one warning and stop watching, while pod start tracking continues
and the task outcome stays unaffected.
2. `KubernetesPodOperator` (and `KubernetesPodTrigger`) should expose an
opt-out for the startup event watcher, mirroring `log_events_on_failure`, so
operators on large clusters can disable it without subclassing.
3. The first poll of the watcher should be served from the apiserver watch
cache (`resourceVersion="0"`) instead of forcing a quorum read; later polls
already use `resourceVersionMatch=NotOlderThan`.
### Operating System
Linux (EKS nodes)
### Deployment
Official Apache Airflow Helm Chart
### Apache Airflow Provider(s)
cncf-kubernetes
### Versions of Apache Airflow Providers
`apache-airflow-providers-cncf-kubernetes>=10.6.0` (introduced by #50192).
Verified on 10.14.0 and on `main`.
### Kubernetes Version
1.36 (EKS). The missing field index for `Event` applies to every Kubernetes
version.
### Anything else?
Related work that improved but did not remove the cost: #59080 (adds
`resourceVersion` / `NotOlderThan`, motivated by the same "many pods started in
parallel" load), #60532 (async watch reconnects), #66470 / #66471 (seed
`resourceVersion` after an empty list, released in 10.17.1), #58736 (403 on
event list, fixed by chart RBAC in #56872). None of them adds an opt-out or
makes the watcher best-effort.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
---
Drafted-by: Claude Code (Fable 5.1) (no human review before posting)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
