GitHub user amerob added a comment to the discussion: Airflow 3: Scheduler
liveness probe failure & performance drop with thousands of dynamic tasks
(KubernetesExecutor)
Your loop analysis is right, but the arithmetic on the probe is off in a way
that changes the fix.
`is_alive()` for a SchedulerJob doesn't use the heartrate at all. From
`airflow/jobs/job.py`:
```python
def health_check_threshold(job_type: str, heartrate: int) -> int | float:
grace_multiplier = 2.1
if job_type == "SchedulerJob":
health_check_threshold_value = conf.getint("scheduler",
"scheduler_health_check_threshold")
...
else:
health_check_threshold_value = heartrate * grace_multiplier
```
The 2.1x grace multiplier other job types get is skipped for the scheduler. It
compares `latest_heartbeat` against `scheduler_health_check_threshold` flat,
default **30s**. So `scheduler_heartbeat_sec` (5s) never enters the liveness
calculation — it only controls how often the loop is *willing* to write a
heartbeat (`perform_heartbeat(..., only_if_necessary=True)`). Your 25s+
`_do_scheduling()` is running straight into that 30s wall, and that's the only
number that matters here.
Second: a single slow loop can't restart your pod. The chart's default is
```yaml
livenessProbe:
initialDelaySeconds: 10
timeoutSeconds: 20
failureThreshold: 5
periodSeconds: 60
```
That's 5 consecutive failures 60s apart — five straight minutes in which
*every* probe sees a >30s-stale heartbeat. That isn't one big expansion, it's
sustained back-to-back long loops. Worth confirming before you tune anything:
if pods are dying more often than once per ~5 min, you're chasing the wrong
thing.
One more that bites people — `enable_health_check: 'True'` is not what your
probe is using. That flag starts a small HTTP server on port 8974 serving
`/health`. The chart's scheduler liveness probe is an exec:
```
airflow jobs check --job-type SchedulerJob --local
```
Unless you've set `scheduler.livenessProbe.command` yourself, toggling
`enable_health_check` changes nothing about the restarts. Both paths land on
the same 30s threshold so it isn't the cause, but it's a dead end to tune.
For the fix, you don't have to throttle anything.
Raise the threshold so it reflects your worst-case loop rather than your
average:
```yaml
config:
scheduler:
scheduler_health_check_threshold: '300'
```
That's the legitimate fix for a false positive — you're telling the probe what
"hung" actually means for this workload. Keep it under the 5x60s window or
you've just moved the goalpost.
Then look hard at `max_tis_per_query: 128`. Default is 16, and this is what's
making your loops long. It bounds how many TIs are evaluated inside the
critical section — the part holding row locks. **Lowering it does not lower
throughput.** You schedule the same tasks in more, shorter passes; the loop
only sleeps when idle:
```python
idle_in_this_run = not num_queued_tis and not num_finished_events
if not is_unit_test and idle_in_this_run:
time.sleep(min(self._scheduler_idle_sleep_time, next_event or 0))
```
With work pending it comes straight back around, no sleep. What you gain is a
heartbeat write between chunks instead of only after all 128.
With 2 replicas that lever does double duty. A 128-TI critical section holds
locks the other scheduler is blocking on, so your two schedulers are partly
serializing against each other — likely the same DB contention you noted
getting masked. Try 32 and watch the `scheduler.scheduler_loop_duration` timer.
If p99 drops under 30s and your queued-tasks/sec holds, you've solved it
without capping task counts.
Last one, if your mapped DAGs are large: check
`max_dagruns_per_loop_to_schedule` (default 20). The `_do_scheduling` docstring
is explicit that raising it helps small DAGs and *hurts* throughput on
>500-task DAGs. If you've bumped it, put it back.
GitHub link:
https://github.com/apache/airflow/discussions/71584#discussioncomment-18013846
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
