GitHub user sdg9670f edited a discussion: Airflow 3: Scheduler liveness probe failure & performance drop with thousands of dynamic tasks (KubernetesExecutor)
> **TL;DR:** > Running Airflow 3 on K8s with `KubernetesExecutor` (2 scheduler replicas). > Processing thousands of dynamic tasks causes `_do_scheduling()` execution > time (~25s+) to exceed the heartbeat interval (5s). Because heartbeat updates > are synchronous in the main loop, delayed heartbeats trigger false-positive > K8s liveness probe failures and pod restarts (Related: Discussion #71584). > **We want to solve this without capping dynamic task counts or throttling > task concurrency.** Looking for tuning strategies for maximum parallel > throughput. Hi everyone, We are currently running Airflow 3 deployed via the official Helm chart in a Kubernetes environment using `KubernetesExecutor` with `2` scheduler replicas. Recently, we encountered severe scheduler instability when running DAGs with large-scale dynamic task mapping (generating several thousand tasks per run). The main symptom is that during heavy task expansion, the **scheduler pod's liveness probe repeatedly fails**, causing Kubernetes to restart the scheduler pods while they are actively processing tasks. --- ### βοΈ Environment & Configuration * **Deployment:** Airflow 3 via official Helm Chart * **Executor:** `KubernetesExecutor` (workers spawned as individual K8s pods) * **Scheduler Replicas:** 2 * **Resource Usage:** CPU and Memory utilization on scheduler pods look healthy with plenty of headroom (no OOM or CPU starvation observed). * **Key Configuration Snippet:** ```yaml scheduler: enable_health_check: 'True' max_tis_per_query: 128 task_queued_timeout: 1200 ``` --- ### π Deep Dive: Problem Mechanics & Root Cause After analyzing the Airflow execution loop (`job.py` & `scheduler_job_runner.py`) alongside discussions like [Discussion #71584], we identified the exact flow leading to these false-positive restarts: ``` Loop Duration (25s+) > heartbeat_sec (5s) x threshold (30s window) β βββΊ _do_scheduling() takes 20-30s+ to expand/queue thousands of dynamic tasks βββΊ Synchronous heartbeat skips sleeping (sleep_for = 0) & loops immediately βββΊ Time since latest_heartbeat exceeds health check window β βββΊ RESULT A: K8s Probe Check = False FAIL βββΊ Unnecessary Pod Restart βββΊ RESULT B: Actual issues (e.g., DB deadlocks) masked or missed ``` Because the main loop executes back-to-back without breathing room, there is almost no window for the health check to pass. The pod gets killed by Kubernetes even though it is performing heavy, valid scheduling work. --- ### β Key Question: How to Scale Without Throttling? Our primary goal is to **maintain high parallelism and scale**. We do **NOT** want to reduce the number of dynamic tasks or lower the task concurrency limitsβwe want Airflow to execute as many tasks simultaneously as possible. With that in mind, how can we resolve this bottleneck and probe failure while maximizing throughput? Any insights, recommended config settings, or architectural advice from high-volume production setups would be greatly appreciated! Thanks in advance! GitHub link: https://github.com/apache/airflow/discussions/71584 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
