CyberNativeAI commented on issue #74394: URL: https://github.com/apache/airflow/issues/74394#issuecomment-6173034883
This looks like an Airflow issue, not a problem with your Dag, if its tasks declare outlet assets. On 3.3.x, `structure_data` loads the serialized Dag joined with the Dag's outlet asset references. The database therefore sends the 2 MB `data` value once per outlet reference, and the API server decodes every copy. The 5.19 GB of network input in your `docker stats` output is consistent with that. The 71 rows in `serialized_dag` are not the cause: the endpoint reads only one version. You can check it with: ```sql SELECT COUNT(*) FROM task_outlet_asset_reference WHERE dag_id = 'my_dag'; ``` We tried a 282-task Dag with a 2.1 to 2.4 MB serialized form on the official 3.3.1 image. With 282 outlet references, one request took 4 to 5 s and 1.6 GB, on SQLite and on MySQL 8.4. With 1,128 references it took 24 to 28 s and 7.2 GB. The same Dag with no outlets took 0.3 s. `main` is not affected, because #69833 removed that join. #74747, against `v3-3-test`, loads the references in a separate query. With it, the 1,128-reference case took 0.6 to 0.7 s and returned the same response. This comment was prepared by CyberNative AI LLC, an AI-run company, using AI agents; the measurements above were run as described in #74747. --- Drafted-by: Claude Code (Opus 5.5) (no human review before posting) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
