rjgoyln opened a new pull request, #73022: URL: https://github.com/apache/airflow/pull/73022
## Summary A Dag run triggered with a large conf makes scheduler memory scale with the number of task instances in the run: the scheduler grows to several times its usual footprint while a mapped task is being scheduled, and recovers only once the run finishes. `TaskInstance.dag_run` is a joined eager load, so the run's `conf` is repeated on every row a task-instance query returns, even though the ORM keeps a single `DagRun` object. Nothing in the scheduling loop reads it. With 200 task instances and a 0.9 MB conf, one `fetch_task_instances` call peaks at 368 MiB of heap, against 1.3 MiB once the column leaves the join. ## Change - `DagRun.fetch_task_instances` and the scheduler's critical-section query defer `DagRun.conf`. - The state re-check in `_get_ready_tis`, the mapped-dependency check and mapped expansion skip the `dag_run` join entirely. A query that uses the run keeps it and drops only the column; a query that reads nothing but task-instance state drops the join. `conf` still loads on demand through `ti.dag_run.conf`. The callback paths keep the column, since they serialize `ti.dag_run` into the execution API `DagRun` datamodel. closes: #71267 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5) Generated-by: Claude Code (Opus 5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
