kaxil opened a new pull request, #74832: URL: https://github.com/apache/airflow/pull/74832
On every pass, the Dag processor manager loop claimed callbacks, claimed priority parse requests and ran `DELETE FROM dag_warning ... is_stale`, each in its own transaction. The loop aims for one pass a second, but `_service_processor_sockets` returns as soon as any parsing process sends a message. So while files are being parsed, which is all the time in a deployment with many Dags, the loop runs many times a second and makes all three DB calls every time. This change polls for callbacks and priority parse requests at most once a second. The exception is a callback fetch that returned a full `max_callbacks_per_loop` batch: then the next pass polls again, so a callback backlog drains as fast as before. The stale-Dag warning purge now runs right after `deactivate_stale_dags`, on the existing `[scheduler] parsing_cleanup_interval` timer (default 60s). That is the step that marks Dags stale, and this purge had been running on every pass since it was added. Measured in breeze on Postgres with `load_examples=True`: 60s warm-up, then scan counts from `pg_stat_user_tables` over a 60s window, one run each: | Per second | Before | After | |---|---:|---:| | `callback` polls | 14.8 | 0.85 | | `dag_priority_parsing_request` polls | 14.8 | 0.83 | | `dag_warning` scans (purge plus parse-result writes) | 18.8 | 4.2 | | Committed transactions, whole DB (only the Dag processor running) | 103.6 | 17.7 | **Callbacks and "reparse" requests can now wait up to about a second longer while parsing is busy.** Before, the loop polled for them about 15 times a second in that state, but an idle loop already waited up to a second, so this is the latency the loop was designed for. The rate limit sits around the overridable `fetch_callbacks` and `claim_priority_files` seams, so managers that override them get the same reduction. **Why not slow the whole loop to one pass a second?** The early wake-up is what lets the manager collect a finished parse and start the next one right away. Limiting the loop would throttle parsing throughput, while limiting only the DB polls leaves parsing unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
