henry3260 opened a new pull request, #76276: URL: https://github.com/apache/airflow/pull/76276
## Why `AsyncKubernetesHook.watch_pod_events` is an async generator, and `@generic_api_retry` on it never retried anything. tenacity treats an async generator function as a plain sync callable, so it wrapped the construction of the generator instead of its iteration — and construction cannot fail. So the first 5xx from the events endpoint ended pod event logging for the rest of that pod's life. A dropped socket was worse: the handler only caught `ApiException`, so `aiohttp.ClientConnectionError` was not caught at all, even though that is how a watch stream held open for hours usually ends. The task still succeeds, because the watch is best-effort. What you lose is the pod events in the task log — the thing you read when a pod fails to start. The retry has to be inside the generator: resuming a watch needs the last `resourceVersion`, which only the reconnect loop can see, and retrying from outside would re-deliver events that were already yielded. ## What `hooks/kubernetes.py` — drop the dead decorator and reconnect inside the existing loop instead, for the errors `_should_retry_api` already accepts. Budget is `API_RETRIES - 1` reconnects with the same backoff bounds the decorator used, and it resets whenever an event arrives so unrelated blips hours apart don't accumulate. The handler now also catches `TRANSIENT_CONNECTION_ERRORS`. 403, 404 and 410 behave as before, and a non-transient status still fails fast. `kubernetes_helper_functions.py` — `generic_api_retry` now raises `TypeError` on an async generator function, so the same mistake fails at import time. `watch_pod_events` was its only async generator. `utils/pod_manager.py` — the dead decorator had hidden the generator's signature from mypy, which was masking a float `startup_check_interval` being passed as the watch's integer `timeoutSeconds`. Now sends whole seconds, floored at 1 since 0 means no timeout. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
