CyberNativeAI opened a new pull request, #74747: URL: https://github.com/apache/airflow/pull/74747
On 3.3, `GET /ui/structure/structure_data` loads the serialized Dag with `joinedload(SerializedDagModel.dag_model).joinedload(DagModel.task_outlet_asset_references)`. `task_outlet_asset_references` is a collection, so the database returns the `serialized_dag` row once per outlet reference, and the API server decodes the `data` column for every copy. For a large Dag with many outlets, every Graph view request becomes slow and memory-hungry. #74394 reports 15 to 17 s and 5 GB for a Dag whose serialized form is 2 MB. That matches this cause, but the reporter has not yet said how many outlet references the Dag has. This change loads the collection with `selectinload`, in its own query. That adds one query per request, and the returned structure is unchanged. `main` is not affected: #69833 removed this eager load as part of the dependency graph rework, and that rework was not backported. So this PR targets `v3-3-test`. Measured with the official `apache/airflow:3.3.1-python3.12` image, whose `structure.py` is byte-identical to this branch's. The Dag has 282 tasks in a chain, its serialized `data` is 2.1 to 2.4 MB, and each task has 0, 1 or 4 outlet assets. One request with `external_dependencies=true`, peak RSS of the process: | Outlet references | Database | Before | After | |---|---|---|---| | 0 | SQLite | 0.31 s, 240 MB | 0.32 s, 240 MB | | 282 | SQLite | 4.0 s, 1.6 GB | 0.34 s, 241 MB | | 1,128 | SQLite | 24.4 s, 7.2 GB | 0.60 s, 245 MB | | 282 | MySQL 8.4 | 5.0 s, 1.6 GB | 0.35 s, 236 MB | | 1,128 | MySQL 8.4 | 27.9 s, 7.2 GB | 0.69 s, 242 MB | In every row the response body was identical before and after, including the asset edges bound to their producing tasks. ### Testing - New test `test_outlet_references_are_not_joined_onto_the_serialized_dag_row`: no statement that selects from `serialized_dag` may also read `task_outlet_asset_reference`, and the references must still be loaded. It fails on `v3-3-test` and passes with the change. - `pytest airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_structure.py` on SQLite with Python 3.11: 26 passed. The existing `assert_queries_count` limits still hold with the extra query. - `prek run --stage pre-commit` and `prek run --stage manual` on both files: passed, including `mypy-airflow-core`. `generate-openapi-spec` needs the CI image and was not run; the endpoint's signature is unchanged. We did not run the tests on PostgreSQL or MySQL. related: #74394 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes (please specify the tool below) Generated-by: Claude Code (Opus 5.5) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) --- * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged. * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed. * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x). * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
