kaxil commented on PR #74048: URL: https://github.com/apache/airflow/pull/74048#issuecomment-5939823187
Closing this. The task still needs `dag_run.conf` as a dict, so the task process builds the full object graph either way; this replaces msgspec's msgpack decoding with `json.loads` rather than avoiding it. I measured the encode/decode steps on a 12.8 MB conf shaped like the one in #74025 (11,000 nested records), including the Pydantic dump and validate: | | supervisor | task process | |---|---|---| | main | 41 ms, 46 MB peak | 25 ms, 54 MB peak | | this PR | 46 ms, 32 MB peak | 39 ms, 67 MB peak | The supervisor saves about 14 MB of transient memory and the task process spends about 13 MB more and roughly 55% more time, so there is no net win, and the change adds a permanent field to the versioned supervisor/task protocol. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
