bcanvural opened a new issue, #74584:
URL: https://github.com/apache/airflow/issues/74584
### Under which category would you file this issue?
Helm chart
### Apache Airflow version
3.3.0
### What happened and how to reproduce it?
With an executor string that contains both `KubernetesExecutor` and
`CeleryExecutor`, the chart's headless worker Service (`<release>-worker`, port
8793 `worker-logs`) selects every KubernetesExecutor task pod as well as the
Celery worker pods.
The Service selector (`templates/workers/worker-service.yaml`) for the
default worker set is:
```yaml
selector:
tier: airflow
component: worker
release: <release>
```
The KubernetesExecutor pod template
(`files/pod-template-file.kubernetes-helm-yaml`) gives every task pod exactly
those labels:
```yaml
labels:
tier: airflow
component: worker
release: <release>
```
The Celery worker pods carry the same three labels and nothing else that
distinguishes them, so the two can't be told apart. Named worker sets
(`workers.celery.sets`) don't have the problem, because their Service selector
adds `worker-set: <name>`, which task pods don't carry. Only the default set is
affected.
Values to reproduce:
```yaml
executor: "KubernetesExecutor,CeleryExecutor"
```
1. `helm install airflow apache-airflow/airflow --version 1.22.0 -f
values.yaml`
2. Run any DAG whose tasks use the KubernetesExecutor (the default executor,
listed first).
3. `kubectl get endpointslices -l kubernetes.io/service-name=airflow-worker
-o yaml`
The EndpointSlice lists the task pods. In one of our deployments it held 1
Celery worker pod and 11 task pods at the moment we looked. Because task pods
come and go with every task, the slice has been updated about 3.35 million
times since it was created about 6 months ago, roughly once every 4–5 seconds.
Why it matters: anything that watches Endpoints/EndpointSlices sees that
churn.
- **Istio:** every endpoint change of a headless Service triggers a full xDS
push (listeners and routes) to every sidecar that can see the Service. By
default, that's every sidecar in the mesh.
- In one cluster with ~550 sidecars (Istio 1.26), 716 of 777 full pushes
in a 15-minute window were for this Service, and about 99% of the bytes istiod
sent were those pushes: several TB per day of control-plane traffic, mostly
crossing availability zones.
- A second cluster with ~230 sidecars (Istio 1.30) showed 843 of 846.
- None of those sidecars ever talks to the worker log server.
- **DNS:** the Service's DNS name also resolves to task pod IPs, which don't
run the Celery log server on 8793.
### What you think should happen instead?
The default worker Service should select only Celery worker pods, the way
named worker sets already do.
A non-breaking way to do it:
- Add a distinguishing label to the Celery worker pod template only (for
example `worker-set: default`, mirroring named sets).
- Add the same label to the Service selector.
- Leave the Deployment/StatefulSet `spec.selector` unchanged, since it is
immutable.
Service selectors can be changed in place, so existing installs upgrade
without recreating anything. Changing `component: worker` on task pods would
also work, but it would break anything users already select on that label
(NetworkPolicies, monitoring).
### Operating System
Official Airflow image (Debian-based), not customized for this path
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
Amazon EKS, Kubernetes 1.36. Istio sidecar mesh in the cluster. The Airflow
namespace itself has no sidecars.
### Official Helm Chart version
1.22.0 (latest released). Also checked `main`: same selector and pod
template labels.
### Kubernetes Version
1.36
### Helm Chart configuration
```yaml
executor: "KubernetesExecutor,CeleryExecutor"
# podTemplate not set (chart default pod template)
```
### Docker Image customizations
None relevant (extra Python dependencies only).
### Anything else?
We're working around it by hiding the Service from the mesh with the Istio
annotation `networking.istio.io/exportTo: "."`. The chart has no annotations
value for the worker Service (unlike `apiServer.service.annotations`), so we
apply it with `kubectl annotate` after each deploy. A
`workers.celery.service.annotations` value would be a useful addition, but it
only hides the symptom; the selector is the actual bug.
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's Code of Conduct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
