GitHub user prayaagmahajan created a discussion: [Memory Leak] Memory usage scales linearly and does not release after DAG execution in webserver(High), celery and scheduler
I am observing a consistent memory leak in our environment for components(Webserver, Scheduler, Celery). Memory consumption increases steadily over time and does not return to baseline levels, even when the system is idle or after the completion of heavy DAG runs. Airflow Version: 3.1.7 (recently migrated from 2.10.3) Executor: [CeleryExecutor] Python Version: [3.11.3] OS/Container Image: [Ubuntu] PS: I am looking a steep memory consumption highly in webserver. Is it by any chance due to some FastAPI configuration which is not releasing the memory? Please let me know if you need more information also one more thing I would like to add i.e how can I fine tune my celery and scheduler to work more efficiently and complete task execution quick and effective is there any configs I can try out related to this performance part? The reason I asked this is when I was using 3.1.3 my dag was getting executed in 9 mins and with 3.1.7 its taking around 14 mins of time Attaching some metrics for better understanding <img width="589" height="210" alt="image" src="https://github.com/user-attachments/assets/5183dcfa-eea7-43fa-921e-39c36bc2c078"; /> <img width="601" height="215" alt="image" src="https://github.com/user-attachments/assets/c15dfc15-b168-4d61-b17d-3c8ccd616a46"; /> <img width="597" height="206" alt="image" src="https://github.com/user-attachments/assets/dce3ad4b-899f-461a-b0ae-89123ac6a809"; /> GitHub link: https://github.com/apache/airflow/discussions/61684 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
