waterWang opened a new pull request, #71322: URL: https://github.com/apache/airflow/pull/71322
## Problem When a DAG with dynamic task mapping is triggered with a large run config (512KB–2MB JSON), the scheduler memory spikes by 5–6× during scheduling of the dynamic tasks. Once the dagrun completes, the memory returns to normal. Reported in #71267. ## Root Cause The scheduler's critical section query at `_executable_task_instances_to_queued` uses `joinedload(TI.dag_run)` (line 808), which eagerly loads the full dag_run row — including the `conf` column — for every task instance in the SQL result set. When 500+ mapped task instances all share the same dag_run, the joined SQL result set carries the 2MiB conf column 500+ times. This results in approximately 1 GiB of redundant conf data being transferred from the database and materialized in Python memory, causing the memory spike. ## Fix Add `.defer(DagRun.conf)` to the eager-load chain. The scheduler never reads `dag_run.conf` in the critical section, so deferring it eliminates the bloat while keeping all other columns and the secondary `selectinload` of `created_dag_version` available. ## Verification - SQLAlchemy standalone test confirms `defer` correctly excludes the `conf` column from the SELECT statement - The existing test `test_executable_task_instances_no_per_ti_queries` still passes (no N+1 regression) - No query count changes — the defer only affects which columns are loaded, not how many queries are issued Closes #71267 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
