steveahnahn opened a new pull request, #69518: URL: https://github.com/apache/airflow/pull/69518
`update_dags` resolves each Dag's run info individually — one latest-run SELECT plus one `DagRun.active_runs_of_dags([dag.dag_id])` call per Dag (a batch-shaped API invoked with a single-element list). Measured with `CountQueries`: exactly 2 queries per Dag, linear — a 10-Dag parse batch issues 20 run-info queries, a 100-Dag Dag-factory file issues ~200 against `dag_run` on every re-parse interval. #59376 deliberately moved this to per-Dag resolution, on the premise that "most of the time it's one dag per file / process anyway" — and this PR keeps that premise intact: **a batch with a single schedulable Dag issues exactly the same queries as today** (it delegates to the unchanged per-Dag path). Only multi-Dag batches — the Dag-factory case #59376 left unmeasured — switch to batched resolution: one latest-run query per timetable kind plus one active-run count query, so the 10-Dag batch drops from 20 run-info queries to 2. ### What changed - `_get_latest_runs_stmt_batch` / `_get_latest_runs_stmt_partitioned_batch`: batch equivalents of the existing single-Dag statements. The non-partitioned one selects per Dag the run(s) matching the max `logical_date` among automated runs (same tie and all-NULL semantics as the single statement); the partitioned one ranks with `row_number()` over the identical ordering — the same window-function pattern already used in `serialized_dag.py`. - `_RunInfo.calculate_many`: resolves a whole batch; single-schedulable-Dag batches delegate to the unchanged `calculate`; non-schedulable Dags keep costing zero queries. - `update_dags` resolves run info once for the batch instead of per Dag in the loop. ### Reviewer notes - On a `logical_date` tie (e.g. a manual run sharing the max), the batch path keeps the first row per Dag — the same arbitrary-row semantics as `session.scalar()` on the single-Dag statement. - Multi-Dag batches no longer emit the per-Dag "Getting latest run…" INFO lines (single-Dag batches keep them, unchanged). ### Tests - Query-count regression: run-info resolution for a 3-Dag batch is exactly 2 queries (fails on the previous code with 6). - Batch-vs-single parity per Dag for both statement kinds, including a Dag whose automated runs all lack a `logical_date` (absent from both) and partitioned ordering with NULL `partition_date`. - `calculate_many` equals `calculate` per Dag across plain / partitioned / run-less / non-schedulable Dags, and for the single-Dag delegation path. - Full `test_collection.py` passes; the new tests plus existing latest-runs tests also verified on PostgreSQL via Breeze. related: #59376 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes - Claude Code (Opus 4.8) Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
