This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new 95bbf6ace25 Pin Docker Hub test images against K8s system-test
rate-limit flakes (#66423)
95bbf6ace25 is described below
commit 95bbf6ace25c013a67962c078ea7be75f7ac2190
Author: Jarek Potiuk <[email protected]>
AuthorDate: Thu May 7 04:10:15 2026 +0200
Pin Docker Hub test images against K8s system-test rate-limit flakes
(#66423)
* Pin Docker Hub test images against rate-limit flakes
The scheduled K8s system-test job has been intermittently red because
multiple test pods pull the unpinned `alpine:latest` (xcom sidecar) and
`busybox:latest` / `ubuntu:latest` (test pods) from Docker Hub
anonymously and trip its 100-pulls-per-6h limit
(https://github.com/apache/airflow/actions/runs/25365187430/job/74380551079).
Without a tag, kubelet defaults `imagePullPolicy` to `Always`, so even
nodes that already cached the image re-pull every run.
Changes
-------
1. **Production default**: `xcom_sidecar.PodDefaults.SIDECAR_CONTAINER`
now uses `alpine:3.23` via a new module-level `XCOM_SIDECAR_IMAGE`
constant. Tagged → `imagePullPolicy: IfNotPresent` by default →
nodes with the image cached do not re-pull.
2. **System / kubernetes-tests pin**: every bare `image="ubuntu"` /
`"busybox"` / `"alpine"` in `kubernetes-tests/...` and the
`cncf/kubernetes` system / unit tests is now pinned (ubuntu:24.04,
busybox:1.37, alpine:3.23). Test assertions in
`test_pod.py` updated to match the new sidecar default.
3. **Pre-load into kind**: a new `_preload_test_images_to_kind()` helper
in `breeze k8s` runs after `_upload_k8s_image()` in
`_run_complete_tests`. It pulls each image on the runner with
exponential-backoff retries on Docker Hub 429s, then `kind load
docker-image` puts it on every node — so kubelet never has to reach
out to the registry once the cluster is ready.
4. **Auto-tracker**: `scripts/ci/prek/upgrade_important_versions.py`
gains `UPGRADE_ALPINE` / `UPGRADE_BUSYBOX` flags, fetchers using the
existing Docker Hub `get_latest_image_version()`, regex patterns for
`alpine:` / `busybox:` literals plus chart `ALPINE_VERSION` ARGs, and
the relevant call-sites added to `FILES_TO_UPDATE`. The next "Upgrade
important CI environment" run will keep these pins fresh
automatically. Ubuntu is intentionally not auto-tracked: the tracker
would prefer the highest semver, which can be an interim
(non-LTS) release — system tests want LTS.
Drive-by
--------
`# type: ignore[no-redef]` on the standard `import tomli as tomllib`
fallback in `dev/registry/extract_{metadata,versions}.py` so `mypy-dev`
passes on edits to anything else under `dev/`. Identical fix lives in
PR #66314 — whichever lands first, the other becomes a no-op rebase.
* Fix expected_pod fixtures + changelog formatting + spelling
Three follow-ups to the original commit, surfaced by CI on
#66423:
1. The dict-literal `image` keys in `expected_pod` fixtures inside
`kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py`
still pointed at the bare names (`"ubuntu"`, `"alpine"`) — only
the kwarg-style `image=` references were caught by the original
sed. Pinned them to match the new defaults. Without this, every
pod-spec equality assertion against `self.expected_pod` failed
on Python 3.10 K8s system tests.
2. The cncf.kubernetes changelog note used a level-3 `~~~` heading
directly under `Changelog ---`, which (a) shifted the entire
version-section hierarchy and produced ~700 cascading docs-build
errors, and (b) was 1 char short of the title length triggering
a `Title underline too short` warning. Replaced the heading with
a bold-led paragraph — same content, no hierarchy disruption.
3. `kubelet` was missing from `docs/spelling_wordlist.txt`, so the
sphinx spellcheck flagged it in the new note.
---
.../airflow_breeze/commands/kubernetes_commands.py | 101 +++++++++++++++++++++
dev/registry/extract_metadata.py | 2 +-
dev/registry/extract_versions.py | 2 +-
docs/spelling_wordlist.txt | 1 +
.../test_kubernetes_pod_operator.py | 90 +++++++++---------
providers/cncf/kubernetes/docs/changelog.rst | 14 +++
.../cncf/kubernetes/utils/xcom_sidecar.py | 10 +-
.../system/cncf/kubernetes/example_kubernetes.py | 2 +-
.../cncf/kubernetes/example_kubernetes_async.py | 4 +-
.../unit/cncf/kubernetes/operators/test_pod.py | 4 +-
scripts/ci/prek/upgrade_important_versions.py | 90 ++++++++++++++++++
11 files changed, 267 insertions(+), 53 deletions(-)
diff --git a/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
b/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
index 093cbeb960e..2645e811919 100644
--- a/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
+++ b/dev/breeze/src/airflow_breeze/commands/kubernetes_commands.py
@@ -707,6 +707,97 @@ def _upload_k8s_image(python: str, kubernetes_version:
str, output: Output | Non
return kind_load_result.returncode, f"Uploaded K8S image to {cluster_name}"
+# Test-suite container images that Airflow's K8s system tests pull from Docker
+# Hub. Tagged (not `:latest`) so kubelet's default imagePullPolicy is
+# IfNotPresent — combined with `kind load` below, this means kubelet uses the
+# already-loaded image and never reaches out to Docker Hub. The pin protects
+# CI runs from Docker Hub anonymous-pull rate limits, which intermittently
+# turn the scheduled K8s test job red. Auto-bumped by
+# scripts/ci/prek/upgrade_important_versions.py.
+K8S_TEST_IMAGES_TO_PRELOAD: tuple[str, ...] = (
+ "alpine:3.23", # xcom_sidecar default in providers/cncf/kubernetes
+ "busybox:1.37", # busybox-based system tests in kubernetes-tests/
+ "ubuntu:24.04", # ubuntu-based system tests in kubernetes-tests/
+)
+
+
+def _docker_pull_with_429_retry(image: str, output: Output | None,
max_attempts: int = 5) -> int:
+ """Run `docker pull <image>` retrying with exponential backoff on Docker
Hub 429s.
+
+ Returns the final docker exit code (0 on success). Non-429 failures fail
+ fast — only the rate-limit pattern is retried, since for everything else
+ retrying would just amplify a real error.
+ """
+ import time
+
+ delay = 5
+ for attempt in range(1, max_attempts + 1):
+ result = run_command(
+ ["docker", "pull", image],
+ check=False,
+ output=output,
+ capture_output=True,
+ text=True,
+ )
+ if result.returncode == 0:
+ return 0
+ stderr = (result.stderr or "") + (result.stdout or "")
+ rate_limited = "429" in stderr or "Too Many Requests" in stderr or
"toomanyrequests" in stderr
+ if not rate_limited:
+ get_console(output=output).print(
+ f"[error]docker pull {image} failed (non-rate-limit):
{stderr.strip()[:500]}"
+ )
+ return result.returncode
+ if attempt == max_attempts:
+ get_console(output=output).print(
+ f"[error]docker pull {image} hit Docker Hub 429 on every
{max_attempts} attempts; giving up."
+ )
+ return result.returncode
+ get_console(output=output).print(
+ f"[warning]docker pull {image} hit Docker Hub 429 "
+ f"(attempt {attempt}/{max_attempts}); sleeping {delay}s before
retry."
+ )
+ time.sleep(delay)
+ delay *= 2
+ return 1
+
+
+def _preload_test_images_to_kind(
+ python: str,
+ kubernetes_version: str,
+ output: Output | None,
+) -> tuple[int, str]:
+ """Pre-pull and `kind load` the pinned test-suite images.
+
+ See K8S_TEST_IMAGES_TO_PRELOAD for the list and rationale. Each image is
+ pulled once on the host (with retry-on-429), then loaded into every kind
+ node. Pods that reference these images then start without kubelet ever
+ reaching out to Docker Hub.
+ """
+ cluster_name = get_kind_cluster_name(python=python,
kubernetes_version=kubernetes_version)
+ for image in K8S_TEST_IMAGES_TO_PRELOAD:
+ get_console(output=output).print(
+ f"[info]Pre-pulling test image {image} for kind cluster
{cluster_name}"
+ )
+ pull_rc = _docker_pull_with_429_retry(image, output=output)
+ if pull_rc != 0:
+ return pull_rc, f"docker pull {image} failed"
+ get_console(output=output).print(f"[info]Loading {image} into kind
cluster {cluster_name}")
+ kind_load_result = run_command_with_k8s_env(
+ ["kind", "load", "docker-image", "--name", cluster_name, image],
+ python=python,
+ output=output,
+ kubernetes_version=kubernetes_version,
+ check=False,
+ )
+ if kind_load_result.returncode != 0:
+ get_console(output=output).print(
+ f"[error]kind load docker-image {image} into {cluster_name}
failed."
+ )
+ return kind_load_result.returncode, f"kind load {image} failed"
+ return 0, f"Pre-loaded {len(K8S_TEST_IMAGES_TO_PRELOAD)} test images into
{cluster_name}"
+
+
@kubernetes_group.command(
name="build-k8s-image",
help="Build k8s-ready airflow image (optionally all images in parallel).",
@@ -2043,6 +2134,16 @@ def _run_complete_tests(
returncode, message = _upload_k8s_image(
python=python, kubernetes_version=kubernetes_version, output=output
)
+ if returncode != 0:
+ _logs(python=python, kubernetes_version=kubernetes_version)
+ return returncode, message
+ get_console(output=output).print(
+ f"\n[info]Pre-loading pinned test images into kind cluster for "
+ f"Python {python}, Kubernetes {kubernetes_version}\n"
+ )
+ returncode, message = _preload_test_images_to_kind(
+ python=python, kubernetes_version=kubernetes_version, output=output
+ )
if returncode != 0:
_logs(python=python, kubernetes_version=kubernetes_version)
return returncode, message
diff --git a/dev/registry/extract_metadata.py b/dev/registry/extract_metadata.py
index 463a9c40808..5d9f635f74e 100644
--- a/dev/registry/extract_metadata.py
+++ b/dev/registry/extract_metadata.py
@@ -46,7 +46,7 @@ from typing import Any
try:
import tomllib # Python 3.11+ stdlib
except ModuleNotFoundError: # pragma: no cover -- Python 3.10 fallback
- import tomli as tomllib
+ import tomli as tomllib # type: ignore[no-redef]
import yaml
from registry_contract_models import validate_providers_catalog
diff --git a/dev/registry/extract_versions.py b/dev/registry/extract_versions.py
index d9dc4e166dc..2908b22b32e 100644
--- a/dev/registry/extract_versions.py
+++ b/dev/registry/extract_versions.py
@@ -49,7 +49,7 @@ from typing import Any
try:
import tomllib # Python 3.11+ stdlib
except ModuleNotFoundError: # pragma: no cover -- Python 3.10 fallback
- import tomli as tomllib
+ import tomli as tomllib # type: ignore[no-redef]
from registry_contract_models import validate_provider_version_metadata
try:
diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt
index a0d9bea9e02..8d6413a2885 100644
--- a/docs/spelling_wordlist.txt
+++ b/docs/spelling_wordlist.txt
@@ -927,6 +927,7 @@ krb
Kube
kube
kubeconfig
+kubelet
Kubernetes
kubernetes
KubernetesPodOperator
diff --git
a/kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py
b/kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py
index 9cf94c98ac6..66825830715 100644
--- a/kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py
+++ b/kubernetes-tests/tests/kubernetes_tests/test_kubernetes_pod_operator.py
@@ -137,7 +137,7 @@ class TestKubernetesPodOperatorSystem:
"affinity": {},
"containers": [
{
- "image": "ubuntu",
+ "image": "ubuntu:24.04",
"args": ["echo 10"],
"command": ["bash", "-cx"],
"env": [],
@@ -182,7 +182,7 @@ class TestKubernetesPodOperatorSystem:
shutil.copy(kubeconfig_path, new_config_path)
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -199,7 +199,7 @@ class TestKubernetesPodOperatorSystem:
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -218,7 +218,7 @@ class TestKubernetesPodOperatorSystem:
def test_working_pod(self):
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -235,7 +235,7 @@ class TestKubernetesPodOperatorSystem:
def test_skip_cleanup(self):
k = KubernetesPodOperator(
namespace="unknown",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -250,7 +250,7 @@ class TestKubernetesPodOperatorSystem:
def test_delete_operator_pod(self):
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -268,7 +268,7 @@ class TestKubernetesPodOperatorSystem:
def test_skip_on_specified_exit_code(self):
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["exit 42"],
task_id=str(uuid4()),
@@ -288,7 +288,7 @@ class TestKubernetesPodOperatorSystem:
"""
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -310,7 +310,7 @@ class TestKubernetesPodOperatorSystem:
"""
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["lalala"],
labels=self.labels,
@@ -331,7 +331,7 @@ class TestKubernetesPodOperatorSystem:
def test_pod_hostnetwork(self):
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -351,7 +351,7 @@ class TestKubernetesPodOperatorSystem:
dns_policy = "ClusterFirstWithHostNet"
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -373,7 +373,7 @@ class TestKubernetesPodOperatorSystem:
scheduler_name = "default-scheduler"
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -392,7 +392,7 @@ class TestKubernetesPodOperatorSystem:
node_selector = {"beta.kubernetes.io/os": "linux"}
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -414,7 +414,7 @@ class TestKubernetesPodOperatorSystem:
)
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -493,7 +493,7 @@ class TestKubernetesPodOperatorSystem:
}
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -516,7 +516,7 @@ class TestKubernetesPodOperatorSystem:
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -547,7 +547,7 @@ class TestKubernetesPodOperatorSystem:
]
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=args,
labels=self.labels,
@@ -577,7 +577,7 @@ class TestKubernetesPodOperatorSystem:
name = str(uuid4())
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
task_id=name,
@@ -602,7 +602,7 @@ class TestKubernetesPodOperatorSystem:
name = str(uuid4())
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
task_id=name,
@@ -626,7 +626,7 @@ class TestKubernetesPodOperatorSystem:
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -666,7 +666,7 @@ class TestKubernetesPodOperatorSystem:
def test_faulty_service_account(self):
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -688,7 +688,7 @@ class TestKubernetesPodOperatorSystem:
bad_internal_command = ["foobar 10 "]
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=bad_internal_command,
labels=self.labels,
@@ -708,7 +708,7 @@ class TestKubernetesPodOperatorSystem:
args = [f"echo '{json.dumps(expected)}' > /airflow/xcom/return.json"]
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=args,
labels=self.labels,
@@ -733,7 +733,7 @@ class TestKubernetesPodOperatorSystem:
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
env_vars=env_vars,
@@ -852,7 +852,7 @@ class TestKubernetesPodOperatorSystem:
containers=[
k8s.V1Container(
name="base",
- image="ubuntu",
+ image="ubuntu:24.04",
command=["/bin/bash"],
args=["-c", 'echo {\\"hello\\" : \\"world\\"} | cat >
/airflow/xcom/return.json'],
env=[k8s.V1EnvVar(name="env_name", value="value")],
@@ -901,7 +901,7 @@ class TestKubernetesPodOperatorSystem:
init_container = k8s.V1Container(
name="init-container",
- image="ubuntu",
+ image="ubuntu:24.04",
env=init_environments,
volume_mounts=volume_mounts,
command=["bash", "-cx"],
@@ -914,7 +914,7 @@ class TestKubernetesPodOperatorSystem:
)
expected_init_container = {
"name": "init-container",
- "image": "ubuntu",
+ "image": "ubuntu:24.04",
"command": ["bash", "-cx"],
"args": ["echo 10"],
"env": [{"name": "key1", "value": "value1"}, {"name": "key2",
"value": "value2"}],
@@ -923,7 +923,7 @@ class TestKubernetesPodOperatorSystem:
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -1037,7 +1037,7 @@ class TestKubernetesPodOperatorSystem:
},
{
"command": ["sh", "-c", 'trap "exit 0" INT; while
true; do sleep 1; done;'],
- "image": "alpine",
+ "image": "alpine:3.23",
"name": "airflow-xcom-sidecar",
"resources": {
"requests": {"cpu": "1m", "memory": "10Mi"},
@@ -1077,7 +1077,7 @@ class TestKubernetesPodOperatorSystem:
priority_class_name = "medium-test"
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -1100,7 +1100,7 @@ class TestKubernetesPodOperatorSystem:
pod_name_too_long = "a" * 221
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -1122,7 +1122,7 @@ class TestKubernetesPodOperatorSystem:
namespace = "default"
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["sleep 1000"],
labels=self.labels,
@@ -1164,7 +1164,7 @@ class TestKubernetesPodOperatorSystem:
def get_op():
return KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["exit 1"],
labels=self.labels,
@@ -1225,7 +1225,7 @@ class TestKubernetesPodOperatorSystem:
def test_changing_base_container_name_with_get_logs(self):
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -1255,7 +1255,7 @@ class TestKubernetesPodOperatorSystem:
"""
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["echo 10"],
labels=self.labels,
@@ -1285,7 +1285,7 @@ class TestKubernetesPodOperatorSystem:
"""
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["bash", "-cx"],
arguments=["sleep 3"],
labels=self.labels,
@@ -1311,7 +1311,7 @@ class TestKubernetesPodOperatorSystem:
def test_changing_base_container_name_failure(self):
k = KubernetesPodOperator(
namespace="default",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=["exit"],
arguments=["1"],
labels=self.labels,
@@ -1360,13 +1360,13 @@ class TestKubernetesPodOperatorSystem:
callback = MagicMock()
init_container = k8s.V1Container(
name="init-container",
- image="busybox",
+ image="busybox:1.37",
command=["sh", "-cx"],
args=[f"echo {marker_from_init_container}"],
)
k = KubernetesPodOperator(
namespace="default",
- image="busybox",
+ image="busybox:1.37",
cmds=["sh", "-cx"],
arguments=[f"echo {marker_from_main_container}"],
labels=self.labels,
@@ -1393,25 +1393,25 @@ class TestKubernetesPodOperatorSystem:
callback = MagicMock()
init_container_to_log_1 = k8s.V1Container(
name="init-container-to-log-1",
- image="busybox",
+ image="busybox:1.37",
command=["sh", "-cx"],
args=[f"echo {marker_from_init_container_to_log_1}"],
)
init_container_to_log_2 = k8s.V1Container(
name="init-container-to-log-2",
- image="busybox",
+ image="busybox:1.37",
command=["sh", "-cx"],
args=[f"echo {marker_from_init_container_to_log_2}"],
)
init_container_to_ignore = k8s.V1Container(
name="init-container-to-ignore",
- image="busybox",
+ image="busybox:1.37",
command=["sh", "-cx"],
args=[f"echo {marker_from_init_container_to_ignore}"],
)
k = KubernetesPodOperator(
namespace="default",
- image="busybox",
+ image="busybox:1.37",
cmds=["sh", "-cx"],
arguments=[f"echo {marker_from_main_container}"],
labels=self.labels,
@@ -1477,7 +1477,7 @@ class TestKubernetesPodOperatorSystem:
marker = f"test_log_{uuid4()}"
k = KubernetesPodOperator(
namespace="default",
- image="busybox",
+ image="busybox:1.37",
cmds=["sh", "-cx"],
arguments=[f"echo {marker}"],
labels={"test_label": "test"},
@@ -1559,7 +1559,7 @@ class TestKubernetesPodOperator(BaseK8STest):
k = KubernetesPodOperator(
task_id=f"test_task_{active_deadline_seconds}",
active_deadline_seconds=active_deadline_seconds,
- image="busybox",
+ image="busybox:1.37",
cmds=["sh", "-c", echo],
namespace=ns,
on_finish_action="keep_pod",
diff --git a/providers/cncf/kubernetes/docs/changelog.rst
b/providers/cncf/kubernetes/docs/changelog.rst
index d5672698561..73400e734e2 100644
--- a/providers/cncf/kubernetes/docs/changelog.rst
+++ b/providers/cncf/kubernetes/docs/changelog.rst
@@ -27,6 +27,20 @@
Changelog
---------
+**Default xcom-sidecar image is now pinned to** ``alpine:3.23``.
+The default container image for the xcom sidecar (used by
``KubernetesPodOperator``
+when ``do_xcom_push=True``) has changed from the unpinned ``alpine`` (which
resolves
+to ``alpine:latest``) to the pinned ``alpine:3.23``. The pin makes the
kubelet's
+default ``imagePullPolicy`` ``IfNotPresent`` instead of ``Always``, so a node
with
+the image cached does not re-pull on every task — protecting deployments and CI
+from Docker Hub anonymous-pull rate limits.
+
+Deployments that override the image via ``xcom_sidecar_container_image`` (or
the
+``[kubernetes] xcom_sidecar_container_image`` config) are unaffected.
Deployments
+that relied on the unpinned default will now be pinned to ``alpine:3.23`` until
+the next Airflow upgrade. Set ``xcom_sidecar_container_image`` explicitly if
you
+need a different alpine version, a private mirror, or another base image.
+
10.17.0
.......
diff --git
a/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/utils/xcom_sidecar.py
b/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/utils/xcom_sidecar.py
index 6cdc9febb02..0931df7e04e 100644
---
a/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/utils/xcom_sidecar.py
+++
b/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/utils/xcom_sidecar.py
@@ -22,6 +22,14 @@ import copy
from kubernetes.client import models as k8s
+# Pinned alpine version for the xcom sidecar default. Pinning (rather than
+# using the implicit `:latest`) makes kubelet's default imagePullPolicy
+# `IfNotPresent` instead of `Always`, so a node that has the image cached
+# does not re-pull on every task — protecting CI and disconnected
+# deployments from Docker Hub anonymous-pull rate limits. Tracked by
+# scripts/ci/prek/upgrade_important_versions.py.
+XCOM_SIDECAR_IMAGE = "alpine:3.23"
+
class PodDefaults:
"""Static defaults for Pods."""
@@ -34,7 +42,7 @@ class PodDefaults:
SIDECAR_CONTAINER = k8s.V1Container(
name=SIDECAR_CONTAINER_NAME,
command=["sh", "-c", XCOM_CMD],
- image="alpine",
+ image=XCOM_SIDECAR_IMAGE,
volume_mounts=[VOLUME_MOUNT],
resources=k8s.V1ResourceRequirements(
requests={
diff --git
a/providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes.py
b/providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes.py
index 732831e0e64..060d03fcb86 100644
---
a/providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes.py
+++
b/providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes.py
@@ -149,7 +149,7 @@ with DAG(
# [START howto_operator_k8s_write_xcom]
write_xcom = KubernetesPodOperator(
namespace="default",
- image="alpine",
+ image="alpine:3.23",
cmds=["sh", "-c", "mkdir -p /airflow/xcom/;echo '[1,2,3,4]' >
/airflow/xcom/return.json"],
name="write-xcom",
do_xcom_push=True,
diff --git
a/providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_async.py
b/providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_async.py
index e245314386c..3754c98ccf1 100644
---
a/providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_async.py
+++
b/providers/cncf/kubernetes/tests/system/cncf/kubernetes/example_kubernetes_async.py
@@ -137,7 +137,7 @@ with DAG(
namespace="kubernetes_task_async_log",
in_cluster=False,
name="astro_k8s_test_pod",
- image="ubuntu",
+ image="ubuntu:24.04",
cmds=[
"bash",
"-cx",
@@ -180,7 +180,7 @@ with DAG(
write_xcom_async = KubernetesPodOperator(
task_id="kubernetes_write_xcom_task_async",
namespace="default",
- image="alpine",
+ image="alpine:3.23",
cmds=["sh", "-c", "mkdir -p /airflow/xcom/;echo '[1,2,3,4]' >
/airflow/xcom/return.json"],
name="write-xcom",
do_xcom_push=True,
diff --git
a/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py
b/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py
index e81f248bb50..e7b1a27e37c 100644
--- a/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py
+++ b/providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py
@@ -892,7 +892,7 @@ class TestKubernetesPodOperator:
do_xcom_push=True,
)
pod = k.build_pod_request_obj(create_context(k))
- assert pod.spec.containers[1].image == "alpine"
+ assert pod.spec.containers[1].image == "alpine:3.23"
def test_xcom_sidecar_container_resources_default(self):
k = KubernetesPodOperator(
@@ -2741,7 +2741,7 @@ class TestKubernetesPodOperatorAsync:
deferrable=True,
)
pod = k.build_pod_request_obj(create_context(k))
- assert pod.spec.containers[1].image == "alpine"
+ assert pod.spec.containers[1].image == "alpine:3.23"
def
test_async_xcom_sidecar_container_resources_default_should_execute_successfully(self):
k = KubernetesPodOperator(
diff --git a/scripts/ci/prek/upgrade_important_versions.py
b/scripts/ci/prek/upgrade_important_versions.py
index 8b751e7d77f..64c3151cb4f 100755
--- a/scripts/ci/prek/upgrade_important_versions.py
+++ b/scripts/ci/prek/upgrade_important_versions.py
@@ -89,6 +89,79 @@ FILES_TO_UPDATE: list[tuple[Path, bool]] = [
(AIRFLOW_ROOT_PATH / "dev" / "provider_db_inventory.py", False),
(AIRFLOW_ROOT_PATH / "dev" / "pyproject.toml", False),
(AIRFLOW_ROOT_PATH / "go-sdk" / ".pre-commit-config.yaml", False),
+ # Files that pin Docker Hub `alpine:` / `busybox:` tags and should be
+ # auto-bumped alongside the rest of the "important versions". Adding new
+ # call sites? Add them here too — the regex in SIMPLE_VERSION_PATTERNS
+ # only mutates files in this list.
+ (
+ AIRFLOW_ROOT_PATH
+ / "providers"
+ / "cncf"
+ / "kubernetes"
+ / "src"
+ / "airflow"
+ / "providers"
+ / "cncf"
+ / "kubernetes"
+ / "utils"
+ / "xcom_sidecar.py",
+ False,
+ ),
+ (
+ AIRFLOW_ROOT_PATH
+ / "providers"
+ / "cncf"
+ / "kubernetes"
+ / "tests"
+ / "system"
+ / "cncf"
+ / "kubernetes"
+ / "example_kubernetes.py",
+ False,
+ ),
+ (
+ AIRFLOW_ROOT_PATH
+ / "providers"
+ / "cncf"
+ / "kubernetes"
+ / "tests"
+ / "system"
+ / "cncf"
+ / "kubernetes"
+ / "example_kubernetes_async.py",
+ False,
+ ),
+ (
+ AIRFLOW_ROOT_PATH
+ / "providers"
+ / "cncf"
+ / "kubernetes"
+ / "tests"
+ / "unit"
+ / "cncf"
+ / "kubernetes"
+ / "operators"
+ / "test_pod.py",
+ False,
+ ),
+ (
+ AIRFLOW_ROOT_PATH
+ / "kubernetes-tests"
+ / "tests"
+ / "kubernetes_tests"
+ / "test_kubernetes_pod_operator.py",
+ False,
+ ),
+ (
+ AIRFLOW_ROOT_PATH
+ / "dev"
+ / "breeze"
+ / "src"
+ / "airflow_breeze"
+ / "commands"
+ / "kubernetes_commands.py",
+ False,
+ ),
]
for file in DOCKER_IMAGES_EXAMPLE_DIR_PATH.rglob("*.sh"):
FILES_TO_UPDATE.append((file, False))
@@ -477,6 +550,8 @@ if UPGRADE_ALL_BY_DEFAULT and VERBOSE:
console.print("[bright_blue]Upgrading all important versions")
# Package upgrade flags
+UPGRADE_ALPINE: bool = get_env_bool("UPGRADE_ALPINE")
+UPGRADE_BUSYBOX: bool = get_env_bool("UPGRADE_BUSYBOX")
UPGRADE_FLIT_CORE: bool = get_env_bool("UPGRADE_FLIT_CORE")
UPGRADE_GITPYTHON: bool = get_env_bool("UPGRADE_GITPYTHON")
UPGRADE_GOLANG: bool = get_env_bool("UPGRADE_GOLANG")
@@ -604,6 +679,19 @@ SIMPLE_VERSION_PATTERNS: dict[str, list[tuple[str, str]]]
= {
"openapi_generator": [
(r"(OPENAPI_GENERATOR_CLI_VER = )(\"[0-9.]+\")",
'OPENAPI_GENERATOR_CLI_VER = "{version}"'),
],
+ # Pinning Docker Hub base-image tags used by Airflow's K8s system tests
+ # protects CI from anonymous-pull rate limits — the kind cluster
+ # `kind load`s the pre-pulled image so kubelet (default
+ # imagePullPolicy=IfNotPresent for tagged images) never reaches Docker
+ # Hub. Pattern matches both `alpine:X.Y[.Z]` literals in code and
+ # `ARG ALPINE_VERSION="X.Y[.Z]"` in chart Dockerfiles.
+ "alpine": [
+ (r"(alpine:)([0-9]+\.[0-9]+(?:\.[0-9]+)?)", "alpine:{version}"),
+ (r'(ALPINE_VERSION=")([0-9.]+)(")', 'ALPINE_VERSION="{version}"'),
+ ],
+ "busybox": [
+ (r"(busybox:)([0-9]+\.[0-9]+(?:\.[0-9]+)?)", "busybox:{version}"),
+ ],
"sphinx_airflow_theme": [
(
r"(sphinx-airflow-theme@https://airflow\.apache\.org/sphinx-airflow-theme/sphinx_airflow_theme-)([0-9.]+)(-py3-none-any\.whl)",
@@ -640,6 +728,8 @@ def fetch_all_package_versions() -> dict[str, str]:
"mypy": get_latest_pypi_version("mypy", UPGRADE_MYPY),
"node_lts": get_latest_lts_node_version() if UPGRADE_NODE_LTS else "",
"protoc": get_latest_image_version("rvolosatovs/protoc") if
UPGRADE_PROTOC else "",
+ "alpine": get_latest_image_version("alpine") if UPGRADE_ALPINE else "",
+ "busybox": get_latest_image_version("busybox") if UPGRADE_BUSYBOX else
"",
"mprocs": get_latest_github_release_version("pvolok/mprocs") if
UPGRADE_MPROCS else "",
"openapi_generator": get_latest_openapi_generator_version() if
UPGRADE_OPENAPI_GENERATOR else "",
"sphinx_airflow_theme": get_latest_sphinx_airflow_theme_version()
