shahar1 opened a new pull request, #75016: URL: https://github.com/apache/airflow/pull/75016
closes: #75013 <details><summary>AI Summary></summary> Since cncf.kubernetes 10.6.0 the synchronous `KubernetesPodOperator` polls the Kubernetes events of the pod while it waits for the pod to start. On a large cluster each of those LISTs makes the API server read every event of the namespace from etcd (kube-apiserver keeps no watch cache for `Event` objects), the request times out with 504, and once the client retries were exhausted the failed watcher task was re-raised from `await_pod_start` and failed the task although the pod had started normally. There was no way to switch the watcher off. This PR: - makes the sync and async startup event watchers best-effort: an API error that survives the retries logs one warning and stops the watch, while pod start tracking and the task outcome stay unaffected; - adds `log_events_on_startup` (default `True`) to `KubernetesPodOperator`, `KubernetesPodTrigger` and the `@task.kubernetes` / `@task.kubernetes_cmd` stubs so the watcher can be disabled without subclassing. The trigger serializes the key only when it is `False`, so trigger subclasses with a fixed `__init__` signature (the Amazon `EksPodTrigger`) and triggerers on an older provider keep working; - documents the behaviour, its cost on the API server and the opt-out. `GKEStartPodOperator` and `EksPodOperator` honour the flag in synchronous mode only. Forwarding it to their own deferrable triggers needs cncf version floors in the google and amazon providers and is tracked in #75014. Verified against a local Kubernetes 1.36.1 cluster (docker-desktop): with the events API raising 504 the pod start is still tracked and the task continues, and with the flag off no event LIST is issued. Checks run locally: `prek run --from-ref upstream/main --stage pre-commit` and `--stage manual` (UI and skill-eval hooks skipped) pass, including `mypy-providers`; the full `providers/cncf/kubernetes/tests/unit` suite passes on sqlite except four pre-existing timezone-dependent `since_time` tests that also fail on `main` unless `TZ=UTC`; the Amazon `EksPodTrigger` tests (4) and the google GKE trigger/operator tests (142) pass. </details> --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Fable 5.1) Generated-by: Claude Code (Fable 5.1) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
