luc-pimentel opened a new pull request, #73907: URL: https://github.com/apache/airflow/pull/73907
A Dag run with a large conf made the scheduler's memory grow with the number of task instances in the run. `TaskInstance.dag_run` is a joined eager load, so every row of a task-instance query also carries the run's conf. On every loop the scheduler reads all task instances of each running Dag run through `DagRun.fetch_task_instances` in `airflow-core/src/airflow/models/dagrun.py`, and re-reads the ones still waiting on their upstreams in `_get_ready_tis`, so it decoded the conf once per task instance. `fetch_task_instances` now defers the conf, which still loads if something reads it, and the re-check, which only compares states, no longer joins the run. The other task-instance queries in the scheduling loop return a bounded number of rows (the critical section, executor events) or a few rows per task instance (the mapped upstream dependency check), so they are left as they are. #73022 changed these two queries and four others; it was closed under the open PR limit before it was reviewed. Reproduced with `airflow standalone` (SQLite, LocalExecutor): one run of a Dag with a single mapped task, triggered through the REST API. The 1 MB conf is about 8,400 small JSON records. Scheduler RSS, idle → peak: | Mapped task instances | Run conf | 3.3.2 | main | With this change | | --- | --- | --- | --- | --- | | 500 | empty | 150 → 160 MB | 206 → 216 MB | 206 → 216 MB | | 50 | 1 MB | 150 → 452 MB | 206 → 553 MB | 206 → 317 MB | | 500 | 1 MB | 150 → 3,234 MB | 206 → 3,422 MB | 206 → 321 MB | A task mapped over an upstream task's output behaves the same: 206 → 3,355 MB on main, 206 → 328 MB with this change. The remaining ~110 MB is the same for 50 and 500 task instances, and the 500-instance run with the 1 MB conf takes 196 s instead of 376 s. The two new tests in `airflow-core/tests/unit/models/test_dagrun.py` fail without this change; reverting either query's change fails at least one of them. Locally, the airflow-core `models`, `jobs`, `ti_deps` and execution API unit tests and the other tests that call these functions pass, apart from four that also fail without this change (two need `graphviz`, one `moto`, and `test_still_in_retry_period`). The prek pre-commit hooks, including `mypy-airflow-core`, and the manual stage pass. closes: #71267 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 5.5) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
