kaxil commented on issue #56641: URL: https://github.com/apache/airflow/issues/56641#issuecomment-3461512931
As promised in https://github.com/apache/airflow/issues/56641#issuecomment-3456893101 , here's our analysis on impact on Celery workers comparing 3.0.6 vs 3.1.1 (with fixes) with one of our deployments: In Airflow 3.0.6 various tasks running on Celery failed with OOMs as the memory leaks were significant. After applying changes in https://github.com/apache/airflow/pull/56695 to 3.0.6 or directly upgrading to 3.1.1, the memory stayed mostly-flat and there were 0 task failures. --- **Celery Worker** with 4 GB memory on **Airflow 3.0.6** <img width="1714" height="407" alt="Image" src="https://github.com/user-attachments/assets/ba5b96a6-df9b-45c3-b9c6-8019c8378c36"; /> **Celery Worker** with 4 GB memory on **Airflow 3.1.1** <img width="1718" height="321" alt="Image" src="https://github.com/user-attachments/assets/ad7b520b-6bb8-4f55-bbde-8fe98f895b0d"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
