wolfdn opened a new pull request, #59080:
URL: https://github.com/apache/airflow/pull/59080
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
## Description
This PR optimizes how the `KubernetesPodOperator` interacts with the
Kubernetes API when retrieving events.
Previously, the operator did **not** pass the `resourceVersion` parameter
when listing events for a pod. This forced Kubernetes to perform a quorum read
for every request—an expensive operation. Combined with frequent polling for
new events, this created significant load on the Kubernetes API and etcd,
especially when many pods were started in parallel.
Best practice is for clients to store the `resourceVersion` from each
response and provide it in subsequent requests. This allows Kubernetes to serve
the event list far more efficiently. As stated in the Kubernetes documentation:
> Unless you have strong consistency requirements, using
resourceVersionMatch=NotOlderThan and a known resourceVersion is preferable
since it can achieve better performance and scalability of your cluster than
leaving resourceVersion and resourceVersionMatch unset, which requires quorum
read to be served.
Reference:
[https://kubernetes.io/docs/reference/using-api/api-concepts/#semantics-for-get-and-list](https://kubernetes.io/docs/reference/using-api/api-concepts/#semantics-for-get-and-list)
With this change, the operator performs one initial event listing without a
`resourceVersion`, and all subsequent requests include the last known
`resourceVersion`.
Additionally, this PR introduces usage of the Kubernetes **watch** API in
deferred (asynchronous) mode. Instead of polling every few seconds, the
operator can now watch for new events. This provides two major benefits:
* New events become visible almost immediately.
* The number of requests sent to the Kubernetes API is reduced because the
watch connection remains active for a longer period.
We implemented this change after observing a high number of HTTP `429`
(rate-limited) responses from our cluster’s API server. One contributing factor
was the large volume of `GET` requests for event listings, which placed heavy
load on etcd. After deploying a patched version of the operator with these
improvements, the number of `429` responses dropped from several thousand per
minute to nearly zero.
<img width="1015" height="313" alt="image"
src="https://github.com/user-attachments/assets/bff05d82-fa27-4645-9c79-a9d1d75592f2";
/>
## Changes
- Remember `resourceVersion` when retrieving events from K8s API
- Use Kubernetes `watch` API to watch events when running in deferred
(asynchronous) mode
- There is also a fallback to poll events in deferred mode in case that
the Airflow triggerer does not have the permission to watch events (to stay
compatible with older versions of Helm chart)
- Improve mechanism to avoid printing duplicate events (now remembers seen
event UIDs instead of counting events)
- Add `watch` verb for `events` in pod launcher role in Helm chart (required
so that Airflow triggerer has permission to watch events)
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request
Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)**
for more information.
In case of fundamental code changes, an Airflow Improvement Proposal
([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals))
is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party
License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a
newsfragment file, named `{pr_number}.significant.rst` or
`{issue_number}.significant.rst`, in
[airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
