hussein-awala opened a new pull request, #76006: URL: https://github.com/apache/airflow/pull/76006
When a Dag run finishes on time, pruning its unmissed deadlines can scan unrelated rows in the deadline table. Add an index on `dagrun_id` so pruning can find the run's deadlines directly. PostgreSQL uses a full index, which also supports run-specific deadline lookups. SQLite indexes only unmissed deadlines: this supports pruning while preserving the global listing plans in the measured fixtures. MySQL reuses its existing foreign-key index by renaming it. On MySQL, upgrade and downgrade preserve the foreign key and keep a single supporting index. Measurements used private databases created from Airflow's migration files, with 100,100 deadlines across 50,050 Dag runs. These are warm-cache synthetic measurements, with three runs per variant, alternating baseline/change, and thirteen recorded repetitions per run. Reported timings are medians of the three run medians. Pruning timings include the actual helper's callback deletion, flush, and Dag-run refresh; transaction rollback is excluded. | Operation | Before | After | | --- | ---: | ---: | | PostgreSQL 14.24: prune one run's deadlines | 6.78 ms | 2.33 ms | | SQLite 3.46.1: prune one run's deadlines | 11.15 ms | 1.29 ms | | SQLite: actual global deadline endpoint function and serialization | 106.49 ms | 107.63 ms | | PostgreSQL: fetch expired deadlines with eager relationships | 4.86 ms | 4.76 ms | | SQLite: fetch expired deadlines with eager relationships | 3.94 ms | 3.90 ms | Pruning in the 100-row fixture remained effectively unchanged. Returned rows and pruning outcomes matched throughout; the selective prune preserved the missed deadline and its callback. SQLite checks also covered the actual endpoint's permitted-Dag filter, scoped listings, and both missed/unmissed filters. Endpoint timings exclude HTTP middleware and authentication. MySQL 8.0.46 retains the same physical index, so this change does not claim a MySQL read speedup. The PostgreSQL index used 1.95 MiB for this fixture. Inserting 100 deadlines added about 0.07 ms; updating 100 missed flags added about 0.21 ms. Deletes remained neutral. SQLite's aggregate median write overhead was at most 3.5% in the large fixture; individual paired runs varied up to 5.2%. Validation passed on PostgreSQL, MySQL and SQLite, including migration upgrade/downgrade cycles, preserved rows and foreign keys, cascade behavior, and model/schema parity. The actual SQLite pruning query passes the index-plan and query-count regression; removing the new index makes that regression fail. Fresh MySQL schema creation also retains only one supporting index. The complete selected database and non-database test scopes passed, along with regular/manual static checks, the migration round-trip hook, and the core documentation build. Large groups ran in bounded batches to fit the local VM, with fresh-process file runs for existing test-order sensitivities. The combined result sets were checked against the full selected scope, with no omitted or duplicated tests. --- ##### Was generative AI tooling used to co-author this PR? - [x] Yes — Codex (GPT-6) Generated-by: Codex (GPT-6) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). --- Drafted-by: Codex (GPT-6) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
