hussein-awala opened a new issue, #76209: URL: https://github.com/apache/airflow/issues/76209
### Goal Make the triggerer cheaper and more responsive: less metadata-database load per loop and per event, lower latency between a trigger firing and its task resuming, less CPU and memory per running trigger, and cheaper scheduler-side checks of deferred tasks. Fix the correctness problems found along the way. The triggerer's architecture stays as it is; public and overridable interfaces stay backward compatible where possible, and every addition or change to them is called out in the PR that makes it. The triggerer runs a tight loop. About once a second its supervisor assigns triggers, refreshes what its runner should run, persists events and failures, and cleans up. Meanwhile the runner's event loop creates, cancels and drains triggers. Small per-iteration costs there are multiplied by every triggerer in every deployment, and by the number of deferred tasks each one holds. ### Approach This work was done with AI tooling (Claude Code) under maintainer direction: 1. **Model the current behaviour.** Write down how the triggerer works on current `main`: the supervisor's database loop, trigger assignment and hydration, event and failure persistence, the runner's asyncio loop, and supervisor–runner communication. 2. **Generate and challenge ideas.** Derive improvement ideas from that model, then have each one challenged independently. Is the problem real on current `main`? Does the proposed fix preserve semantics, such as ordering, locking, HA behaviour and API compatibility? Only ideas that pass both checks go forward. 3. **Implement and review.** Implement each remaining idea as a small, self-contained change with tests, and review every change adversarially, on its own and against the other changes in the series, before submission. 4. **Measure.** Measure each change against the same `main` baseline, using whichever method fits it: statement counts and alternated base/change wall-time medians on SQLite and PostgreSQL for database paths, micro-benchmarks of the real code for runner and supervisor internals, and query plans or lock clauses for schema and locking changes. ### How the work arrives The changes arrive as a series of small, independently reviewable pull requests that reference this issue, opened in batches so the review load stays manageable. Each PR states the problem with links into `main`, the measured before/after effect and how it was measured, and its compatibility notes. PRs that add configuration, change a default, touch an overridable hook or touch the schema say so explicitly. A few PRs build on another one and are marked as stacked. Where two PRs change the same code, they say so, and the later one is rebased once the earlier one merges. This issue stays open while the series is under review and is closed when it is complete. --- Drafted-by: Claude Code (Opus 5.5); reviewed by @hussein-awala before posting -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
