hussein-awala opened a new pull request, #76218: URL: https://github.com/apache/airflow/pull/76218
Once a second, the trigger runner's `cleanup_finished_triggers()` checks `task.done()` on every running trigger, awaiting `asyncio.sleep(0)` after each ([source](https://github.com/apache/airflow/blob/b95498cd56/airflow-core/src/airflow/jobs/triggerer_job_runner.py#L1580-L1626)). At 5,000 running triggers that costs about 77.6 ms per second of the triggers' event loop, even when nothing finished. Instead, each trigger task's done callback queues its id, and the cleanup pass handles only queued ids. The create, cancel and cleanup loops yield after 2 ms of work, not after every item. related: #76209 Cancelled triggers are reported as finished without a log line; exceptions, a returned `TriggerEvent` and exiting without an event are logged and reported, all as before. `create_triggers()`, the only creator of trigger tasks, attaches the callback; ids queued while cleanup yields wait for the next pass. `create_triggers()` still creates everything queued in one call, as on `main`; bounding it per call is a separate PR stacked on this one. No new configuration options. Two other PRs in #76209 overlap this code; whichever merges later is rebased. One, which stops repeated cancels from cutting `on_kill()` short, edits the same `cancel_triggers()` loop and forgets cancelled ids (`_cancel_delivered_at.pop()`) inside the `cleanup_finished_triggers()` scan replaced here; when the second of the two merges, that pop moves into the callback-driven cleanup. The other, planned to wake the runner on trigger activity, also adds a done callback per trigger task in `create_triggers()` and runs cleanup up to about 20 times per second, so it asks for this PR to land with or before it. In-process benchmarks (cleanup rows: per-pass means over 10 passes; all values: medians of 5 runs): | Scenario | Before | After | |---|---:|---:| | Cleanup pass, 5,000 running, none finished | 77.55 ms | < 0.01 ms | | Cleanup pass, 5,000 running + 100 finished | 70.58 ms | 0.01 ms | | Finished triggers collected over 10 passes | 100 | 100 | | `create_triggers()` for 500 workloads | 167.4 ms | 71.2 ms | | Control: supervisor `update_triggers()`, 10,000 running | 198.3 µs | 207.4 µs | The cleanup bench attaches an equivalent done callback; with 100 finished, the first pass collects all 100. The creation bench times one call for 500 `SuccessTrigger` workloads. Base and head alternated running first. Every head creation run (66.3–127.9 ms) beat the fastest base run (156.2 ms); the control, which this change does not touch, moved +4.6 % with overlapping ranges. These paths issue no SQL. Local macOS measurements, not end-to-end latency. `test_cleanup_finished_triggers_visits_only_finished_triggers` sets the yield interval to 0, so every visited trigger yields, and expects exactly one event-loop turn during a cleanup pass over 100 running triggers and one finished one; scanning every trigger, as on `main`, gives 101. `test_cleanup_finished_triggers_handles_only_ids_queued_when_it_starts` queues a finished trigger while a pass yields and expects the next pass to handle it. `test_create_cancel_and_cleanup_yield_by_time_not_per_trigger` sets the interval to an hour and expects no turns while creating, cancelling and cleaning up 100 triggers; yielding per item in each loop gives 200, 100 and 100. Each fails by assertion when what it covers is reverted: `main`'s full scan (both cleanup tests), a `_maybe_yield` that never yields, per-item yielding in any one of the three loops, or a pass that also drains ids queued while it yields. Validation: - Triggerer and trigger-model tests: 208 passed, 1 skipped on SQLite. - PostgreSQL and MySQL were not run locally; this change has no database code. - Regular and manual static checks: re-running on the final revision. - Selected core suites and the documentation build were not run locally; CI validation is pending. --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
