potiuk commented on PR #67077:
URL: https://github.com/apache/airflow/pull/67077#issuecomment-5975882040
Thanks for the effort here, but I'm going to close this PR.
The scheduler loop is the most performance-sensitive code in Airflow:
everything in it runs on every iteration, often inside the critical section
that serializes all HA schedulers, so even small additions in logging, metrics
or queries multiply across every loop of every deployment. This PR has too many
issues of exactly that kind for a review round to fix:
- a new INFO log line on every loop (~86k lines/day per scheduler, even when
idle), next to timing that is deliberately kept at DEBUG;
- two new metrics that duplicate existing ones
(`scheduler.zombies.detected{reason=heartbeat_timeout}` and the per-executor
`executor.open_slots`), one of which never resets to 0;
- the query-count guard in `test_execute_queries_count_with_harvested_dags`
loosened from 21/148 to 40/211 although the PR adds no queries — which would
hide real query regressions in the loop;
- no tests for any of the new behaviour, and a broken
`triggerer.trigger_queue_delay` entry in `metrics_template.yaml`.
Changes to the scheduler loop need to start from a measured problem and come
with evidence of their impact — query counts, log volume and timing on a
realistic number of Dags and task instances — rather than just adding
diagnostics. If you'd like to pursue this, please take a deeper look at how the
loop works today, measure what your change costs, and reopen the PR (or open a
new one) with a much smaller, focused change and that evidence. The extra
heartbeat-timeout context (`pid` and the durations) is a good candidate for
such a focused change, added to the existing
`_generate_task_instance_heartbeat_timeout_message_details` dict.
---
Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
