1fanwang opened a new pull request, #77204:
URL: https://github.com/apache/airflow/pull/77204
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
When a KubernetesExecutor worker pod is deleted while its task runs, for
example by a forced delete, the executor's pod watch normally reports it and
the task fails within seconds. After 30 seconds without pod changes, the watch
read times out (the client timeout from
https://github.com/apache/airflow/pull/39551). The watcher logs "Kubernetes
watch timed out waiting for events. Restarting watch." and a second later
starts a new watch from the resource version the old one started at, usually 0.
A watch from 0 lists only existing pods, so a pod deleted during that second is
never reported. Its task stays running and holds an executor slot until
"Detected a task instance without a heartbeat", five minutes later by default.
The watcher now keeps the resource version of the last change event it
handled, so the next watch after a read timeout starts there and replays the
deletion. Below, the task fails two seconds after the deletion. The watcher
skips the ADDED events that open a watch from 0, since each carries its pod's
own, possibly expired, version. If the server reports that version as too old
(410), the watcher restarts from the current state in place instead of
crashing, which is how the provider's async pod event watch already handles it.
## Testing Done
Both runs use the reproducer below, which runs a native scheduler, Dag
processor and execution API on macOS with SQLite, starts KubernetesExecutor
pods in a kind cluster running Kubernetes 1.37.0 from one worker image built at
3b35135afe9549e99088c4d557c62e5deafb125b, and force-deletes the running task
pod as soon as the watcher logs its read timeout.
Current main:
```bash
"$MAIN_TREE/.venv/bin/python" watch_gap_e2e.py --label main --tree
"$MAIN_TREE" --image aip97-watch-worker:3b35135a \
--kubeconfig "$KUBECONFIG" --context kind-aip97-watch --output
raw/main-2 > raw/main-2.driver.log 2>&1; echo "exit=$?" >> raw/main-2.driver.log
sed -n '/TI_RUNNING/,$p' raw/main-2.driver.log | sed -E 's/ \(See
https:[^)]*\)//; s/ Learn more: https:[^ ]*//'
```
```text
{"utc": "2026-11-17T14:57:30.954547+00:00", "event": "TI_RUNNING", "ti":
{"state": "running", "start_date": "2026-11-17 14:57:28.435273", "end_date":
null, "last_heartbeat_at": "2026-11-17 14:57:30.669467"}}
{"utc": "2026-11-17T14:57:55.839606+00:00", "event":
"POD_FORCE_DELETED_AFTER_WATCH_TIMEOUT", "pod":
"watch-gap-long-running-p1kk98ot", "timeout_line_seen_at":
"2026-11-17T14:57:55.830244+00:00", "phase": "Running", "pod_resource_version":
"2057"}
{"utc": "2026-11-17T15:03:00.736384+00:00", "event": "TI_FINISHED", "ti":
{"state": "failed", "start_date": "2026-11-17 14:57:28.435273", "end_date":
"2026-11-17 15:03:00.117391", "last_heartbeat_at": "2026-11-17
14:57:55.964167"}}
SCHEDULER_LOG (watcher and task state lines)
2026-11-17T14:57:21.394058Z [info ] Event: and now my watch begins
starting at resource_version: 0
2026-11-17T14:57:25.665453Z [info ] Received executor event with state
queued for task instance 01a1ea03-41e1-7a12-8801-18f716bbd659
(coordinates=TaskInstanceKey(dag_id='watch_gap', task_id='long_running',
run_id='watch-gap-main', try_number=1, map_index=-1))
2026-11-17T14:57:25.832225Z [info ] Event:
watch-gap-long-running-p1kk98ot is Running, annotations: <omitted>
2026-11-17T14:57:55.829606Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
2026-11-17T14:57:56.836865Z [info ] Event: and now my watch begins
starting at resource_version: 0
2026-11-17T14:58:26.841850Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
2026-11-17T14:58:27.844106Z [info ] Event: and now my watch begins
starting at resource_version: 0
2026-11-17T14:58:57.849079Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
2026-11-17T14:58:58.850544Z [info ] Event: and now my watch begins
starting at resource_version: 0
2026-11-17T14:59:28.854866Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
2026-11-17T14:59:29.855997Z [info ] Event: and now my watch begins
starting at resource_version: 0
2026-11-17T14:59:59.859803Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
2026-11-17T15:00:00.869966Z [info ] Event: and now my watch begins
starting at resource_version: 0
2026-11-17T15:00:30.875219Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
2026-11-17T15:00:31.878657Z [info ] Event: and now my watch begins
starting at resource_version: 0
2026-11-17T15:01:01.881422Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
2026-11-17T15:01:02.884357Z [info ] Event: and now my watch begins
starting at resource_version: 0
2026-11-17T15:01:32.889459Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
2026-11-17T15:01:33.890294Z [info ] Event: and now my watch begins
starting at resource_version: 0
2026-11-17T15:02:03.894720Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
2026-11-17T15:02:04.897976Z [info ] Event: and now my watch begins
starting at resource_version: 0
2026-11-17T15:02:34.902935Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
2026-11-17T15:02:35.911784Z [info ] Event: and now my watch begins
starting at resource_version: 0
2026-11-17T15:03:00.116303Z [error ] Detected a task instance without a
heartbeat: <TaskInstance: watch_gap.long_running watch-gap-main [running]
ti_id=01a1ea03-41e1-7a12-8801-18f716bbd659>
2026-11-17T15:03:00.117901Z [info ] Marking task as FAILED.
dag_id=watch_gap, task_id=long_running, run_id=watch-gap-main, logical_date=,
start_date=20261117T145728, end_date=20261117T150300
2026-11-17T15:03:00.656800Z [info ] Received executor event with state
failed for task instance 01a1ea03-41e1-7a12-8801-18f716bbd659
(coordinates=TaskInstanceKey(dag_id='watch_gap', task_id='long_running',
run_id='watch-gap-main', try_number=1, map_index=-1))
2026-11-17T15:03:05.917141Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
2026-11-17T15:03:06.922587Z [info ] Event: and now my watch begins
starting at resource_version: 0
EXECUTOR_GAUGES (value changes)
2026-11-17T14:57:19.457262+00:00 airflow.executor.open_slots=4
2026-11-17T14:57:19.457329+00:00 airflow.executor.running_tasks=0
2026-11-17T14:57:25.687757+00:00 airflow.executor.open_slots=3
2026-11-17T14:57:25.687804+00:00 airflow.executor.running_tasks=1
2026-11-17T15:03:00.656705+00:00 airflow.executor.open_slots=4
2026-11-17T15:03:00.656964+00:00 airflow.executor.running_tasks=0
exit=0
```
This branch, same scenario:
```bash
"$FIX_TREE/.venv/bin/python" watch_gap_e2e.py --label fix --tree "$FIX_TREE"
--image aip97-watch-worker:3b35135a \
--kubeconfig "$KUBECONFIG" --context kind-aip97-watch --output raw/fix-1
> raw/fix-1.driver.log 2>&1; echo "exit=$?" >> raw/fix-1.driver.log
sed -n '/TI_RUNNING/,$p' raw/fix-1.driver.log | sed -E 's/ \(See
https:[^)]*\)//; s/ Learn more: https:[^ ]*//'
```
```text
{"utc": "2026-11-17T14:55:48.578966+00:00", "event": "TI_RUNNING", "ti":
{"state": "running", "start_date": "2026-11-17 14:55:46.290364", "end_date":
null, "last_heartbeat_at": "2026-11-17 14:55:48.504518"}}
{"utc": "2026-11-17T14:56:13.592856+00:00", "event":
"POD_FORCE_DELETED_AFTER_WATCH_TIMEOUT", "pod":
"watch-gap-long-running-ax1lqh71", "timeout_line_seen_at":
"2026-11-17T14:56:13.583828+00:00", "phase": "Running", "pod_resource_version":
"1889"}
{"utc": "2026-11-17T14:56:16.603907+00:00", "event": "TI_FINISHED", "ti":
{"state": "failed", "start_date": "2026-11-17 14:55:46.290364", "end_date":
"2026-11-17 14:56:15.617775", "last_heartbeat_at": "2026-11-17
14:56:13.751881"}}
SCHEDULER_LOG (watcher and task state lines)
2026-11-17T14:55:39.975056Z [info ] Event: and now my watch begins
starting at resource_version: 0
2026-11-17T14:55:43.456271Z [info ] Received executor event with state
queued for task instance 01a1ea01-b50f-7f08-a3b6-74199847397b
(coordinates=TaskInstanceKey(dag_id='watch_gap', task_id='long_running',
run_id='watch-gap-fix', try_number=1, map_index=-1))
2026-11-17T14:55:43.585363Z [info ] Event:
watch-gap-long-running-ax1lqh71 is Running, annotations: <omitted>
2026-11-17T14:56:13.583721Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
2026-11-17T14:56:14.593859Z [info ] Event: and now my watch begins
starting at resource_version: 1845
2026-11-17T14:56:14.601954Z [info ] Event:
watch-gap-long-running-ax1lqh71 is Running, annotations: <omitted>
2026-11-17T14:56:14.602683Z [info ] Event: Pod
watch-gap-long-running-ax1lqh71 deleted before it could complete, annotations:
<omitted>
2026-11-17T14:56:15.614028Z [info ] Changing state of
KubernetesResults(key=TaskInstanceUuid(id=UUID('01a1ea01-b50f-7f08-a3b6-74199847397b')),
state=<TaskInstanceState.FAILED: 'failed'>,
pod_name='watch-gap-long-running-ax1lqh71', namespace='watch-gap-fix',
resource_version='1889', failure_details=None) to failed
2026-11-17T14:56:15.614698Z [info ] Received executor event with state
failed for task instance 01a1ea01-b50f-7f08-a3b6-74199847397b
(coordinates=TaskInstanceKey(dag_id='watch_gap', task_id='long_running',
run_id='watch-gap-fix', try_number=1, map_index=-1))
2026-11-17T14:56:15.616890Z [error ] Executor
KubernetesExecutor(parallelism=4) reported that the task instance
<TaskInstance: watch_gap.long_running watch-gap-fix [running]
ti_id=01a1ea01-b50f-7f08-a3b6-74199847397b> finished with state failed, but the
task instance's state attribute is running.
2026-11-17T14:56:15.618150Z [info ] Marking task as FAILED.
dag_id=watch_gap, task_id=long_running, run_id=watch-gap-fix, logical_date=,
start_date=20261117T145546, end_date=20261117T145615
EXECUTOR_GAUGES (value changes)
2026-11-17T14:55:38.285914+00:00 airflow.executor.open_slots=4
2026-11-17T14:55:38.285955+00:00 airflow.executor.running_tasks=0
2026-11-17T14:55:43.483620+00:00 airflow.executor.open_slots=3
2026-11-17T14:55:43.483664+00:00 airflow.executor.running_tasks=1
2026-11-17T14:56:16.637923+00:00 airflow.executor.open_slots=4
2026-11-17T14:56:16.637974+00:00 airflow.executor.running_tasks=0
exit=0
```
This branch with the saved version expired. Here the watcher runs on its own
against the cluster while the driver patches an unrelated pod five times a
second, so the server drops the old version from its watch history.
```bash
"$FIX_TREE/.venv/bin/python" watch_expire_e2e.py --label fix --kubeconfig
"$KUBECONFIG" --context kind-aip97-watch \
--log raw/expire-fix-2.log > raw/expire-fix-2.log 2>&1; echo "exit=$?"
>> raw/expire-fix-2.log
cat raw/expire-fix-2.log
```
```text
2026-11-17T15:11:39.567094Z [info ] Event: and now my watch begins
starting at resource_version: 0
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:170
2026-11-17T15:11:39.771307Z [info ] Event: task-pod Pending,
annotations: <omitted>
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:340
2026-11-17T15:11:39.772770Z [info ] Event: task-pod Pending,
annotations: <omitted>
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:340
2026-11-17T15:11:39.778963Z [info ] Event: task-pod Pending,
annotations: <omitted>
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:340
2026-11-17T15:11:40.173488Z [info ] Event: task-pod Pending,
annotations: <omitted>
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:340
2026-11-17T15:11:40.227411Z [info ] Event: task-pod is Running,
annotations: <omitted>
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:392
DRIVER: task-pod is Running; patching pod churn in watch-expire-churn 5
times a second
2026-11-17T15:12:10.224521Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:126
2026-11-17T15:12:11.230316Z [info ] Event: and now my watch begins
starting at resource_version: 4706
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:170
2026-11-17T15:12:41.236580Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:126
2026-11-17T15:12:42.239313Z [info ] Event: and now my watch begins
starting at resource_version: 4706
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:170
2026-11-17T15:13:12.244990Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:126
2026-11-17T15:13:13.249424Z [info ] Event: and now my watch begins
starting at resource_version: 4706
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:170
2026-11-17T15:13:13.267228Z [info ] Kubernetes resource version 4706 is
too old. Restarting watch from the current state.
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:130
2026-11-17T15:13:13.267476Z [info ] Event: and now my watch begins
starting at resource_version: 0
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:170
2026-11-17T15:13:13.275348Z [info ] Event: task-pod is Running,
annotations: <omitted>
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:392
DRIVER: 450 churn patches in 95s, outcome: is too old
2026-11-17T15:13:43.274413Z [info ] Kubernetes watch timed out waiting
for events. Restarting watch.
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:126
2026-11-17T15:13:44.284986Z [info ] Event: and now my watch begins
starting at resource_version: 0
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:170
2026-11-17T15:13:44.292173Z [info ] Event: task-pod is Running,
annotations: <omitted>
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:392
DRIVER: watcher process alive: True exitcode: None
2026-11-17T15:13:44.478040Z [info ] Event: task-pod is Running,
annotations: <omitted>
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:392
2026-11-17T15:13:44.481512Z [info ] Event: Pod task-pod deleted before
it could complete, annotations: <omitted>
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
loc=kubernetes_executor_utils.py:376
DRIVER: deleted task-pod
DRIVER: watcher reported task-pod failed
exit=0
```
<details>
<summary>Reproducer source: setup.sh</summary>
```bash
#!/usr/bin/env bash
# MAIN_TREE: checkout of apache/main at
3b35135afe9549e99088c4d557c62e5deafb125b; FIX_TREE: checkout of this branch.
set -euo pipefail
export DOCKER_CONTEXT=colima
export KUBECONFIG="$PWD/kind/kubeconfig"
mkdir -p kind
cat > kind/kind-config.yaml <<'YAML'
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
name: aip97-watch
networking:
apiServerAddress: "127.0.0.1"
apiServerPort: 18445
nodes:
- role: control-plane
image:
kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
YAML
kind create cluster --config kind/kind-config.yaml --kubeconfig
"$KUBECONFIG" --wait 180s
for tree in "$MAIN_TREE" "$FIX_TREE"; do
(cd "$tree" && uv sync --frozen && uv pip install --python
.venv/bin/python datadog==0.56.0)
done
./build_worker_image.sh "$MAIN_TREE"
3b35135afe9549e99088c4d557c62e5deafb125b image/context
aip97-watch-worker:3b35135a
kind load docker-image aip97-watch-worker:3b35135a --name aip97-watch
```
</details>
<details>
<summary>Reproducer source: watch_gap_e2e.py</summary>
```python
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import contextlib
import datetime
import json
import os
import re
import secrets
import shutil
import signal
import socket
import sqlite3
import subprocess
import threading
import time
from collections.abc import Callable
from pathlib import Path
from typing import Any
import yaml
from kubernetes import client, config
API_PORT: int = 18090
STATSD_PORT: int = 18095
HOST_ALIAS: str = "host.lima.internal"
DAG_ID: str = "watch_gap"
TIMEOUT_LINE: str = "Kubernetes watch timed out waiting for events.
Restarting watch."
def utc() -> str:
return datetime.datetime.now(datetime.timezone.utc).isoformat()
def emit(event: str, **values: Any) -> None:
print(json.dumps({"utc": utc(), "event": event, **values}, default=str),
flush=True)
def read_ti(database: Path) -> dict[str, Any] | None:
with contextlib.suppress(sqlite3.OperationalError):
with sqlite3.connect(f"file:{database}?mode=ro", uri=True,
timeout=5) as connection:
connection.row_factory = sqlite3.Row
row = connection.execute(
"SELECT state, start_date, end_date, last_heartbeat_at FROM
task_instance WHERE dag_id = ?",
(DAG_ID,),
).fetchone()
return dict(row) if row else None
return None
def wait_for(predicate: Callable[[], Any], description: str, timeout: float,
interval: float = 0.5) -> Any:
deadline: float = time.monotonic() + timeout
while time.monotonic() < deadline:
if value := predicate():
return value
time.sleep(interval)
raise TimeoutError(description)
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--label", required=True)
parser.add_argument("--tree", required=True, type=Path)
parser.add_argument("--image", required=True)
parser.add_argument("--kubeconfig", required=True, type=Path)
parser.add_argument("--context", required=True)
parser.add_argument("--output", required=True, type=Path)
arguments = parser.parse_args()
output: Path = arguments.output
home: Path = output / "home"
(home / "dags").mkdir(parents=True)
shutil.copyfile(src=Path(__file__).with_name("watch_gap_dag.py"),
dst=home / "dags" / "watch_gap_dag.py")
namespace: str = f"watch-gap-{arguments.label}"
repository, tag = arguments.image.rsplit(":", 1)
worker_env: dict[str, str] = {
"AIRFLOW__CORE__EXECUTOR": "LocalExecutor",
"AIRFLOW__CORE__LOAD_EXAMPLES": "False",
"AIRFLOW__CORE__DAGS_FOLDER": "/opt/watch-gap/dags",
"AIRFLOW__CORE__EXECUTION_API_SERVER_URL":
f"http://{HOST_ALIAS}:{API_PORT}/execution/",
}
template: dict[str, Any] = {
"apiVersion": "v1",
"kind": "Pod",
"metadata": {"name": "watch-gap-template"},
"spec": {
"restartPolicy": "Never",
"automountServiceAccountToken": False,
"containers": [
{
"name": "base",
"image": arguments.image,
"imagePullPolicy": "Never",
"env": [{"name": name, "value": value} for name, value
in worker_env.items()],
"resources": {"requests": {"cpu": "100m", "memory":
"256Mi"}},
}
],
},
}
(home / "pod-template.yaml").write_text(yaml.safe_dump(template))
database: Path = home / "airflow.db"
environment: dict[str, str] = {
**os.environ,
"PYTHONUNBUFFERED": "1",
"AIRFLOW_HOME": str(home),
"AIRFLOW__DATABASE__SQL_ALCHEMY_CONN": f"sqlite:///{database}",
"AIRFLOW__CORE__DAGS_FOLDER": str(home / "dags"),
"AIRFLOW__CORE__LOAD_EXAMPLES": "False",
"AIRFLOW__CORE__EXECUTOR": "KubernetesExecutor",
"AIRFLOW__CORE__PARALLELISM": "4",
"AIRFLOW__CORE__EXECUTION_API_SERVER_URL":
f"http://{HOST_ALIAS}:{API_PORT}/execution/",
"AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL": "1",
"AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC": "1",
"AIRFLOW__API_AUTH__JWT_SECRET": secrets.token_urlsafe(64),
"AIRFLOW__METRICS__STATSD_ON": "True",
"AIRFLOW__METRICS__STATSD_DATADOG_ENABLED": "True",
"AIRFLOW__METRICS__STATSD_HOST": "127.0.0.1",
"AIRFLOW__METRICS__STATSD_PORT": str(STATSD_PORT),
"KUBECONFIG": str(arguments.kubeconfig),
"AIRFLOW__KUBERNETES_EXECUTOR__CONFIG_FILE":
str(arguments.kubeconfig),
"AIRFLOW__KUBERNETES_EXECUTOR__CLUSTER_CONTEXT": arguments.context,
"AIRFLOW__KUBERNETES_EXECUTOR__IN_CLUSTER": "False",
"AIRFLOW__KUBERNETES_EXECUTOR__NAMESPACE": namespace,
"AIRFLOW__KUBERNETES_EXECUTOR__POD_TEMPLATE_FILE": str(home /
"pod-template.yaml"),
"AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_REPOSITORY":
repository,
"AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_TAG": tag,
}
airflow: str = str(arguments.tree / ".venv" / "bin" / "airflow")
config.load_kube_config(config_file=str(arguments.kubeconfig),
context=arguments.context)
kube = client.CoreV1Api()
kube.create_namespace(body=client.V1Namespace(metadata=client.V1ObjectMeta(name=namespace)))
emit("SETUP", tree=str(arguments.tree), image=arguments.image,
namespace=namespace)
stop = threading.Event()
datagrams_path: Path = output / "statsd-datagrams.log"
receiver = socket.socket(family=socket.AF_INET, type=socket.SOCK_DGRAM)
receiver.bind(("127.0.0.1", STATSD_PORT))
receiver.settimeout(0.2)
def capture() -> None:
with datagrams_path.open(mode="a") as datagrams:
while not stop.is_set():
with contextlib.suppress(TimeoutError):
packet, _ = receiver.recvfrom(65535)
for line in packet.decode().splitlines():
datagrams.write(f"{utc()} {line}\n")
datagrams.flush()
capture_thread = threading.Thread(target=capture, daemon=True)
capture_thread.start()
processes: list[subprocess.Popen[bytes]] = []
scheduler_log: Path = output / "scheduler.log"
try:
subprocess.run(args=[airflow, "db", "migrate"], env=environment,
check=True, timeout=300,
stdout=subprocess.DEVNULL)
for name, command in (
("api", ["api-server", "--apps", "execution", "--host",
"127.0.0.1", "--port", str(API_PORT)]),
("dag-processor", ["dag-processor"]),
("scheduler", ["scheduler"]),
):
with (output / f"{name}.log").open(mode="wb") as log:
processes.append(
subprocess.Popen(
args=[airflow, *command], env=environment,
stdout=log, stderr=subprocess.STDOUT,
start_new_session=True,
)
)
wait_for(lambda: _dag_serialized(database), "Dag not serialized",
timeout=180)
wait_for(lambda: "now my watch begins" in
scheduler_log.read_text(errors="replace"),
"pod watcher never started", timeout=180)
subprocess.run(
args=[airflow, "dags", "trigger", DAG_ID, "--run-id",
f"watch-gap-{arguments.label}"],
env=environment, check=True, timeout=120,
stdout=subprocess.DEVNULL,
)
running: dict[str, Any] = wait_for(
lambda: (ti if (ti := read_ti(database)) and ti["state"] ==
"running" and ti["last_heartbeat_at"] else None),
"task instance never ran", timeout=300,
)
emit("TI_RUNNING", ti=running)
pod = wait_for(lambda:
next(iter(kube.list_namespaced_pod(namespace=namespace).items), None),
"no worker pod", timeout=30)
pod_name: str = pod.metadata.name
wait_for(lambda: f"Event: {pod_name} is Running" in
scheduler_log.read_text(errors="replace"),
"watcher never saw the pod running", timeout=60)
offset: int = scheduler_log.stat().st_size
def timeout_line() -> bool:
with scheduler_log.open(mode="rb") as log:
log.seek(offset)
return TIMEOUT_LINE.encode() in log.read()
wait_for(timeout_line, "watcher never timed out", timeout=120,
interval=0.02)
detected_at: str = utc()
deleted = kube.delete_namespaced_pod(name=pod_name,
namespace=namespace, grace_period_seconds=0)
emit("POD_FORCE_DELETED_AFTER_WATCH_TIMEOUT", pod=pod_name,
timeout_line_seen_at=detected_at,
phase=deleted.status.phase,
pod_resource_version=deleted.metadata.resource_version)
final: dict[str, Any] = wait_for(
lambda: (ti if (ti := read_ti(database)) and ti["state"] not in
("running", "queued", "scheduled") else None),
"task instance never finished", timeout=600, interval=1,
)
emit("TI_FINISHED", ti=final)
time.sleep(10)
finally:
for process in processes:
with contextlib.suppress(ProcessLookupError):
os.killpg(process.pid, signal.SIGTERM)
for process in processes:
try:
process.wait(timeout=20)
except subprocess.TimeoutExpired:
with contextlib.suppress(ProcessLookupError):
os.killpg(process.pid, signal.SIGKILL)
stop.set()
capture_thread.join(timeout=2)
receiver.close()
kube.delete_namespace(name=namespace)
summarize(output=output)
def _dag_serialized(database: Path) -> bool:
with contextlib.suppress(sqlite3.OperationalError):
with sqlite3.connect(f"file:{database}?mode=ro", uri=True,
timeout=5) as connection:
return connection.execute("SELECT 1 FROM serialized_dag WHERE
dag_id = ?", (DAG_ID,)).fetchone() is not None
return False
def summarize(output: Path) -> None:
keep = re.compile(
r"watch timed out|my watch begins|is Running|too old|deleted before
it could complete|Changing state of|"
r"Received executor event|without a heartbeat|Marking task as
FAILED|reported that the task instance"
)
print("\nSCHEDULER_LOG (watcher and task state lines)")
for line in (output /
"scheduler.log").read_text(errors="replace").splitlines():
if keep.search(line):
print(re.sub(r" \[airflow\.\S+\]| loc=\S+", "", line)[:400])
print("\nEXECUTOR_GAUGES (value changes)")
last: dict[str, str] = {}
for line in (output / "statsd-datagrams.log").read_text().splitlines():
if match :=
re.search(r"(airflow\.executor\.(?:running_tasks|open_slots)):([\d.]+)\|g",
line):
if last.get(match[1]) != match[2]:
last[match[1]] = match[2]
print(line.split(" ", 1)[0], f"{match[1]}={match[2]}")
if __name__ == "__main__":
main()
```
</details>
<details>
<summary>Reproducer source: watch_gap_dag.py</summary>
```python
from __future__ import annotations
import time
import pendulum
from airflow.sdk import DAG, task
@task
def long_running() -> None:
print("WATCH_GAP_TASK_STARTED", flush=True)
time.sleep(900)
with DAG(
dag_id="watch_gap",
schedule=None,
start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
catchup=False,
is_paused_upon_creation=False,
default_args={"retries": 0},
):
long_running()
```
</details>
<details>
<summary>Reproducer source: watch_expire_e2e.py</summary>
```python
#!/usr/bin/env python3
from __future__ import annotations
import argparse
import multiprocessing
import os
import time
from pathlib import Path
CHURN_NAMESPACE: str = "watch-expire-churn"
SCHEDULER_JOB_ID: str = "777"
def wait_for_line(log: Path, needles: tuple[str, ...], offset: int, timeout:
float) -> str | None:
deadline: float = time.monotonic() + timeout
while True:
with log.open(mode="rb") as handle:
handle.seek(offset)
text: str = handle.read().decode(errors="replace")
for needle in needles:
if needle in text:
return needle
if time.monotonic() >= deadline:
return None
time.sleep(0.2)
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--label", required=True)
parser.add_argument("--kubeconfig", required=True)
parser.add_argument("--context", required=True)
parser.add_argument("--log", required=True, type=Path)
arguments = parser.parse_args()
os.environ.update(
{
"AIRFLOW_HOME": str(arguments.log.with_suffix(".home")),
"KUBECONFIG": arguments.kubeconfig,
"AIRFLOW__KUBERNETES_EXECUTOR__CONFIG_FILE":
arguments.kubeconfig,
"AIRFLOW__KUBERNETES_EXECUTOR__CLUSTER_CONTEXT":
arguments.context,
"AIRFLOW__KUBERNETES_EXECUTOR__IN_CLUSTER": "False",
}
)
from kubernetes import client, config
from
airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils import
KubernetesJobWatcher
from airflow.providers.cncf.kubernetes.kube_config import KubeConfig
config.load_kube_config(config_file=arguments.kubeconfig,
context=arguments.context)
kube = client.CoreV1Api()
namespace: str = f"watch-expire-{arguments.label}"
for name in (namespace, CHURN_NAMESPACE):
kube.create_namespace(body=client.V1Namespace(metadata=client.V1ObjectMeta(name=name)))
kube.create_namespaced_pod(
namespace=CHURN_NAMESPACE,
body={
"apiVersion": "v1",
"kind": "Pod",
"metadata": {"name": "churn"},
"spec": {
"nodeSelector": {"watch-expire": "never"},
"containers": [{"name": "c", "image":
"aip97-watch-worker:3b35135a"}],
},
},
)
watcher = KubernetesJobWatcher(
namespace=namespace,
watcher_queue=multiprocessing.Queue(),
resource_version="0",
scheduler_job_id=SCHEDULER_JOB_ID,
kube_config=KubeConfig(),
)
watcher.start()
try:
if not wait_for_line(arguments.log, ("now my watch begins",),
offset=0, timeout=120):
raise TimeoutError("watcher did not start")
kube.create_namespaced_pod(
namespace=namespace,
body={
"apiVersion": "v1",
"kind": "Pod",
"metadata": {
"name": "task-pod",
"labels": {"airflow-worker": SCHEDULER_JOB_ID},
"annotations": {"dag_id": "d", "task_id": "t", "run_id":
"r", "try_number": "1"},
},
"spec": {
"terminationGracePeriodSeconds": 0,
"containers": [
{"name": "base", "image":
"aip97-watch-worker:3b35135a", "imagePullPolicy": "Never",
"command": ["sleep", "900"]}
],
},
},
)
if not wait_for_line(arguments.log, ("Event: task-pod is Running",),
offset=0, timeout=120):
raise TimeoutError("watcher never saw task-pod running")
print("DRIVER: task-pod is Running; patching pod churn in",
CHURN_NAMESPACE, "5 times a second", flush=True)
offset: int = arguments.log.stat().st_size
started: float = time.monotonic()
outcome: str | None = None
count: int = 0
while outcome is None and time.monotonic() - started < 300:
kube.patch_namespaced_pod(
name="churn", namespace=CHURN_NAMESPACE, body={"metadata":
{"annotations": {"n": str(count)}}}
)
count += 1
time.sleep(0.2)
if count % 25 == 0:
outcome = wait_for_line(
arguments.log, ("is too old", "Unknown error in
KubernetesJobWatcher"), offset=offset, timeout=0
)
print(f"DRIVER: {count} churn patches in {time.monotonic() -
started:.0f}s, outcome: {outcome}", flush=True)
offset = arguments.log.stat().st_size
if not wait_for_line(arguments.log, ("now my watch begins",),
offset=offset, timeout=120):
raise TimeoutError("watcher stopped watching")
print("DRIVER: watcher process alive:", watcher.is_alive(),
"exitcode:", watcher.exitcode, flush=True)
kube.delete_namespaced_pod(name="task-pod", namespace=namespace,
grace_period_seconds=0)
print("DRIVER: deleted task-pod", flush=True)
reported = watcher.watcher_queue.get(timeout=60)
print("DRIVER: watcher reported", reported.pod_name, reported.state,
flush=True)
finally:
watcher.terminate()
watcher.join(timeout=10)
for name in (namespace, CHURN_NAMESPACE):
kube.delete_namespace(name=name)
if __name__ == "__main__":
main()
```
</details>
<details>
<summary>Reproducer source: Dockerfile</summary>
```dockerfile
FROM
python:3.12-slim-bookworm@sha256:782412e85d0f0984994c290652577d4018aff08145c85b262bb63dc0c7522254
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1 \
PIP_NO_CACHE_DIR=1 \
AIRFLOW_HOME=/opt/watch-gap/home
WORKDIR /opt/watch-gap
RUN python -m pip install uv==0.13.8
COPY requirements.txt ./
RUN uv pip install --system --no-deps -r requirements.txt
COPY source ./source
RUN uv pip install --system --no-deps --no-sources \
-e ./source/airflow-core -e ./source/task-sdk \
./source/providers/standard ./source/providers/common/compat
./source/providers/cncf/kubernetes \
&& python -c "import airflow, airflow.sdk; print('airflow',
airflow.__version__)"
COPY source-commit.txt watch_gap_dag.py ./
RUN mkdir -p dags home && cp watch_gap_dag.py dags/
```
</details>
<details>
<summary>Reproducer source: build_worker_image.sh</summary>
```bash
#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Usage: build_worker_image.sh <airflow-worktree> <commit> <context-dir>
<image-tag>
set -euo pipefail
worktree="$1"
commit="$2"
context="$3"
tag="$4"
demo="$(cd "$(dirname "$0")" && pwd)"
rm -rf "$context"
mkdir -p "$context/source"
git -C "$worktree" archive --format=tar "$commit" \
airflow-core task-sdk shared providers/standard providers/common/compat
providers/cncf/kubernetes \
| tar -x -C "$context/source"
git -C "$worktree" rev-parse "$commit" > "$context/source-commit.txt"
(cd "$worktree" && uv export --frozen --no-hashes --no-dev
--no-emit-workspace --no-header \
--package apache-airflow-core --package apache-airflow-task-sdk \
--package apache-airflow-providers-cncf-kubernetes --package
apache-airflow-providers-standard \
--package apache-airflow-providers-common-compat -o
"$context/requirements.txt")
cp "$demo/Dockerfile" "$demo/watch_gap_dag.py" "$context/"
docker build --platform linux/arm64 --tag "$tag" "$context"
docker image inspect "$tag" --format '{{.Id}} {{.Architecture}} {{.Os}}'
```
</details>
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: GitHub Copilot CLI (gpt-6.1-sol, claude-opus-5.5) following
[the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the Pull Request Guidelines for more information:
https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines.
Note: commit author/co-author name and email in commits become permanently
public when merged.
* For fundamental code changes, an Airflow Improvement Proposal (AIP) is
needed:
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals.
* When adding dependency, check compliance with the ASF 3rd Party License
Policy: https://www.apache.org/legal/resolved.html#category-x.
* For significant user-facing changes create newsfragment:
{pr_number}.significant.rst, in airflow-core/newsfragments:
https://github.com/apache/airflow/tree/main/airflow-core/newsfragments. You can
add this file in a follow-up commit after the PR is created so you know the PR
number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]