nailo2c opened a new pull request, #69459: URL: https://github.com/apache/airflow/pull/69459
The K8s test Kind clusters have used a control-plane + worker topology since the original Kind migration in 2019 (#5837), and it was never revisited. Nothing uses the second node: workload pods can never schedule on the control-plane (`NoSchedule` taint), yet `kind load` copies the ~1.8GB Airflow image and the pinned test images into **every** node — the control-plane copy is pure overhead. No test depends on the topology. This switches the cluster to a single control-plane node (Kind's own default), which runs the workloads too — Kind removes the taint on single-node clusters. ## Changes - `kind-cluster-conf.yaml`: drop the worker node, move `extraPortMappings` to the control-plane - `get_kubernetes_port_numbers()`: the forwarded port was read from the hard-coded `nodes[1]`, which raises `IndexError` on the new config; it now looks the mapping up by `containerPort`, so rendered configs of clusters created **before** this change keep working too - unit tests for the port extraction (single-node / legacy two-node / missing mapping) - doc sample outputs and workflow comments updated to match ## Local benchmark Single run, Apple-silicon macOS + Docker Desktop, same image on both sides, KubernetesExecutor, python 3.10 / K8s v1.30.13: | metric | two-node (main) | single-node (this PR) | delta | |---|---|---|---| | `upload-k8s-image` | 114.1s | 66.4s | **−42%** | | containerd disk, all nodes | 15.1 GB | 7.8 GB | **−48%** | | memory after deploy, all nodes | ~3.39 GiB | ~2.65 GiB | **−22%** | | `kind create cluster` ¹ | 50.5s | 16.8s | **−67%** | | `deploy-airflow` | 97.0s | 83.9s | −14% | | test results | 57 passed / 3 skipped | 57 passed / 3 skipped | identical | ¹ measured with plain `kind create cluster` and the node image cached on both sides, isolating the topology effect — the worker join is the expensive part. ## Verification - KubernetesExecutor suite on the single-node cluster: 57 passed / 3 skipped — identical to the two-node baseline - CeleryExecutor (largest pod count: redis + celery workers): 53 passed / 7 skipped, including the two Celery-specific tests that are skipped under KubernetesExecutor Every job in the CI K8s matrix (36 `K8S System` jobs per canary run, ~500 job-minutes) pays the double image load today. I'll post a per-job duration comparison against recent canary runs once CI has run on this PR. ## Validation - `uv run --project dev/breeze pytest dev/breeze/tests/test_kubernetes_utils.py -xvs` - `prek run --from-ref main --stage pre-commit` - full manual `breeze k8s` flow (create / configure / upload / deploy / tests / delete) for both executors above --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Fable 5) Generated-by: Claude Code (Fable 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
