deepak4babu opened a new issue, #71267: URL: https://github.com/apache/airflow/issues/71267
### Under which category would you file this issue? Airflow Core ### Apache Airflow version 3.2.0 ### What happened and how to reproduce it? When a dag with dynamic task is triggered with large run config, the scheduler memory increases by 5-6 times the regular usage when scheduling the dynamic tasks. Once the dagrun completes, the memory usage is back to normal. The run config is a json of size 512KB to 2MB To reproduce, create a dag with dynamic task that create 500+ tasks. Trigger the dag with a json of size 512KB to 1MB as run config ### What you think should happen instead? The scheduler memory should not spike when running dynamic task with large run config. ### Operating System Debian GNU/Linux 12 (bookworm) ### Deployment Other 3rd-party Helm chart ### Apache Airflow Provider(s) _No response_ ### Versions of Apache Airflow Providers apache-airflow-providers-celery==3.17.1 apache-airflow-providers-cncf-kubernetes==10.14.0 apache-airflow-providers-common-compat==1.14.1 apache-airflow-providers-common-io==1.7.1 apache-airflow-providers-common-sql==1.33.0 apache-airflow-providers-fab==3.6.1 apache-airflow-providers-smtp==2.4.3 apache-airflow-providers-standard==1.12.1 ### Official Helm Chart version Not Applicable ### Kubernetes Version Not Applicable ### Helm Chart configuration Not Applicable ### Docker Image customizations Not Applicable ### Anything else? Issue happens in dynamic dags, everytime, the dag is triggered with big config. Older issue for reference - https://github.com/apache/airflow/issues/49076 ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
