GitHub user sdg9670f edited a discussion: Airflow 3: Scheduler liveness probe
failure & performance drop with thousands of dynamic tasks (KubernetesExecutor)
Hi everyone,
> TL;DR:
We are running Airflow 3 on Kubernetes via Helm with KubernetesExecutor (2
scheduler replicas). When executing DAGs with thousands of dynamic tasks, the
scheduler suffers severe latency, leading to liveness probe failures and pod
restarts. Scheduler CPU/Memory usage remains normal, and we are using default
configurations. Looking for advice on optimizing scheduler and
KubernetesExecutor settings to handle high task volume smoothly.
We are currently running Airflow 3 deployed via the official Helm chart in a
Kubernetes environment. Our setup uses `2` scheduler replicas, and we are
mostly relying on the default scheduler configurations provided by the chart.
Recently, we started running DAGs that utilize dynamic task mapping at scale
(generating several thousand tasks per run). When these tasks are expanded and
the scheduler starts processing them, we encounter severe scheduler performance
degradation. Eventually, the **liveness probe for the scheduler fails**,
causing Kubernetes to restart the scheduler pods repeatedly.
Here are a few additional details about our environment and observations:
* **Deployment:** Airflow 3 via official Helm Chart
* **Executor:** `KubernetesExecutor` (workers are spawned as individual K8s
pods)
* **Scheduler Replicas:** 2
* **Resource Usage:** CPU and Memory utilization on the scheduler pods look
healthy with plenty of headroom (no OOM or CPU starvation observed).
* ```yaml
scheduler:
enable_health_check: 'True'
max_tis_per_query: 128
task_queued_timeout: 1200
```
It seems like the scheduler loop gets choked or severely delayed while handling
the high volume of dynamic task instances and managing the corresponding worker
pods via the KubernetesExecutor, leading to unresponsiveness for the liveness
check.
Since hardware resources don't appear to be the bottleneck, we're trying to
figure out the best way to optimize Airflow to handle thousands of dynamic
tasks smoothly in this architecture.
A few questions we'd love advice on:
1. What are the key `[scheduler]` or `[kubernetes_executor]` configurations we
should tune for large-scale dynamic tasks? (e.g., batching parameters, DB pool
limits, worker pod creation rates, etc.)
2. Is it recommended to adjust the `livenessProbe` initial delays/timeouts, or
should we strictly focus on fixing the underlying scheduling throughput?
3. Are there specific DB-side or Airflow 3 specific optimizations we should
look into for heavy dynamic task mapping combined with KubernetesExecutor?
Any insights, recommended config settings, or best practices from similar
high-volume setups would be greatly appreciated!
Thanks in advance!
GitHub link: https://github.com/apache/airflow/discussions/71584
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
