seanghaeli opened a new pull request, #75251:
URL: https://github.com/apache/airflow/pull/75251
In multi-team mode, a team's KubernetesExecutor already reads its namespace,
pod template and the other pod settings from the team's config, but it still
connects with the global `[kubernetes_executor]` settings. So every team ends
up on the same cluster. A team can set its own `client_factory`, but the
factory is called with no arguments, so it can't tell which team it's building
a client for.
This has a second problem for subclasses that bring their own client, like
the AwsEksExecutor in #74399. The only way for a subclass to install its
factory is the `[kubernetes_executor] client_factory` setting. A plain
KubernetesExecutor in the same scope (both global, or the same team) reads that
same setting, so it ends up sending its pods to the subclass's cluster.
This PR:
- Reads `in_cluster`, `config_file`, `cluster_context`, `verify_ssl`,
`ssl_ca_cert` and `api_client_retry_configuration` through the team's config,
for the sync client, the async client and the pod watcher (which already gets
the team). They don't fall back to the global section, same as all other team
config. `enable_tcp_keepalive` and `tcp_keep_*` stay global, since they set
socket options for the whole process.
- Calls a `client_factory` / `async_client_factory` with `team_name` if it
takes that argument. Factories with no arguments are still called with none, so
the 10.24.0 contract still works.
- Lets a subclass set `client_factory` and `async_client_factory` as class
attributes (import paths, so the watcher process can resolve them). They take
precedence over the config settings and are called with `team_name`, so a
KubernetesExecutor running next to the subclass keeps its own client.
One thing worth calling out: `verify_ssl = False` changes the process-wide
default `Configuration`. Now that teams can set it differently, a team's client
sets it on that client only. So one team turning off TLS verification doesn't
turn it off for the other teams' clients in the same scheduler, and a team
doesn't pick it up from a global executor that turned it off. The no-team path
is unchanged.
Behavior change for multi-team users: a team that relied on the global
connection settings now needs to set them for the team. Multi-team is still
experimental, and I added a note at the top of the changelog.
closes: #74844
### Testing
Unit tests in Breeze (`test_kube_client.py`, `test_client.py`,
`executors/test_kubernetes_executor.py`): 287 passed, 1 skipped. I also
reverted each of the 26 pieces of the change one at a time and checked that a
test fails for each one (26/26), and with all the source reverted to main, the
new tests fail (24 failed).
E2E on two real EKS clusters (A and B) with a real scheduler, Dag processor
and API server, and one Dag per team. Each task prints the OIDC issuer of the
cluster its pod runs on (from its service account token).
Two teams, both KubernetesExecutor, with team-scoped
`config_file`/`cluster_context` pointing at cluster A and cluster B. The global
section points at cluster A.
Before (this change reverted):
```
01:23:19 RUN ii_tc_team_a state=success t=44s
01:23:26 RUN ii_tc_team_b state=success t=51s
--- kubectl get pods -n tc-e2e (cluster a: eks-systest-lane)
NAME STATUS NODE
RUN_ID
tc-team-a-where-am-i-r851z2sy Completed
ip-192-168-69-92.us-west-2.compute.internal ii_tc_team_a
tc-team-b-where-am-i-xoyo4rf2 Completed
ip-192-168-55-77.us-west-2.compute.internal ii_tc_team_b
pod/tc-team-a-where-am-i-r851z2sy: TC_E2E dag=tc_team_a
pod=tc-team-a-where-am-i-r851z2sy namespace=tc-e2e
issuer=.../id/FCCA5D040C7E38D31ACD37CD4CC74F77
pod/tc-team-b-where-am-i-xoyo4rf2: TC_E2E dag=tc_team_b
pod=tc-team-b-where-am-i-xoyo4rf2 namespace=tc-e2e
issuer=.../id/FCCA5D040C7E38D31ACD37CD4CC74F77
--- kubectl get pods -n tc-e2e (cluster b: eks-team-b-lane)
No resources found in tc-e2e namespace.
```
After:
```
01:14:12 RUN ii_tc_team_a state=success t=45s
01:14:18 RUN ii_tc_team_b state=success t=51s
--- kubectl get pods -n tc-e2e (cluster a: eks-systest-lane)
NAME STATUS NODE
RUN_ID
tc-team-a-where-am-i-bjz65ezy Completed
ip-192-168-69-92.us-west-2.compute.internal ii_tc_team_a
pod/tc-team-a-where-am-i-bjz65ezy: TC_E2E dag=tc_team_a
pod=tc-team-a-where-am-i-bjz65ezy namespace=tc-e2e
issuer=.../id/FCCA5D040C7E38D31ACD37CD4CC74F77
--- kubectl get pods -n tc-e2e (cluster b: eks-team-b-lane)
NAME STATUS NODE
RUN_ID
tc-team-b-where-am-i-zwc4tcfy Completed
ip-192-168-66-143.us-west-2.compute.internal ii_tc_team_b
pod/tc-team-b-where-am-i-zwc4tcfy: TC_E2E dag=tc_team_b
pod=tc-team-b-where-am-i-zwc4tcfy namespace=tc-e2e
issuer=.../id/CDA69AD28888CDCE3DE924E826AFBCDD
```
The pod watcher builds its client from the team's settings too. The
kubeconfig exec plugin logs which process asked for a token for which cluster:
the scheduler and one watcher process per team, plus the async client team_b
uses for pod creation:
```
01:13:38 EXEC cluster=eks-systest-lane ppid=514 parent='airflow scheduler'
01:13:40 EXEC cluster=eks-systest-lane ppid=596 parent='airflow scheduler'
01:13:40 EXEC cluster=eks-team-b-lane ppid=514 parent='airflow scheduler'
01:13:42 EXEC cluster=eks-team-b-lane ppid=656 parent='airflow scheduler'
01:14:02 EXEC cluster=eks-team-b-lane ppid=514 parent='airflow scheduler'
```
A team's `client_factory` that takes `team_name` is told its team (this one
picks cluster B for team_a), and a no-argument factory (the 10.24.0 contract)
still works. The scheduler and the watcher process (its child) each call the
factory:
```
01:20:42 RUN v3_tc_team_a state=success t=48s
01:20:02 FACTORY build_client_for_team(team_name='team_a') called pid=1089
ppid=1079
01:20:04 FACTORY build_client_for_team(team_name='team_a') called pid=1169
ppid=1089
pod/tc-team-a-where-am-i-sm9o2230: TC_E2E dag=tc_team_a
pod=tc-team-a-where-am-i-sm9o2230 namespace=tc-e2e
issuer=.../id/CDA69AD28888CDCE3DE924E826AFBCDD
01:19:29 RUN v2_tc_side_by_side state=success t=39s
01:18:57 FACTORY build_client_no_args() called pid=790 ppid=780
01:19:00 FACTORY build_client_no_args() called pid=834 ppid=790
pod/tc-side-by-side-on-kubernetesexecutor-ywul9ns4: TC_E2E
dag=tc_side_by_side pod=tc-side-by-side-on-kubernetesexecutor-ywul9ns4
namespace=tc-e2e issuer=.../id/CDA69AD28888CDCE3DE924E826AFBCDD
```
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: Claude Code (Opus 5.5) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
