GitHub user sdg9670f created a discussion: Airflow 3: Scheduler liveness probe failure & performance drop with thousands of dynamic tasks (Helm deployment)
Hi everyone, > TL;DR: We are running Airflow 3 on Kubernetes via Helm with KubernetesExecutor (2 scheduler replicas). When executing DAGs with thousands of dynamic tasks, the scheduler suffers severe latency, leading to liveness probe failures and pod restarts. Scheduler CPU/Memory usage remains normal, and we are using default configurations. Looking for advice on optimizing scheduler and KubernetesExecutor settings to handle high task volume smoothly. We are currently running Airflow 3 deployed via the official Helm chart in a Kubernetes environment. Our setup uses `2` scheduler replicas, and we are mostly relying on the default scheduler configurations provided by the chart. Recently, we started running DAGs that utilize dynamic task mapping at scale (generating several thousand tasks per run). When these tasks are expanded and the scheduler starts processing them, we encounter severe scheduler performance degradation. Eventually, the **liveness probe for the scheduler fails**, causing Kubernetes to restart the scheduler pods repeatedly. Here are a few additional details about our environment and observations: * **Deployment:** Airflow 3 via official Helm Chart * **Executor:** `KubernetesExecutor` (workers are spawned as individual K8s pods) * **Scheduler Replicas:** 2 * **Resource Usage:** CPU and Memory utilization on the scheduler pods look healthy with plenty of headroom (no OOM or CPU starvation observed). * **Configurations:** Mostly default `airflow.cfg` / Helm values for scheduler settings. It seems like the scheduler loop gets choked or severely delayed while handling the high volume of dynamic task instances and managing the corresponding worker pods via the KubernetesExecutor, leading to unresponsiveness for the liveness check. Since hardware resources don't appear to be the bottleneck, we're trying to figure out the best way to optimize Airflow to handle thousands of dynamic tasks smoothly in this architecture. A few questions we'd love advice on: 1. What are the key `[scheduler]` or `[kubernetes_executor]` configurations we should tune for large-scale dynamic tasks? (e.g., batching parameters, DB pool limits, worker pod creation rates, etc.) 2. Is it recommended to adjust the `livenessProbe` initial delays/timeouts, or should we strictly focus on fixing the underlying scheduling throughput? 3. Are there specific DB-side or Airflow 3 specific optimizations we should look into for heavy dynamic task mapping combined with KubernetesExecutor? Any insights, recommended config settings, or best practices from similar high-volume setups would be greatly appreciated! Thanks in advance! GitHub link: https://github.com/apache/airflow/discussions/71584 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
