LourTV opened a new issue, #55768: URL: https://github.com/apache/airflow/issues/55768
### Apache Airflow version 3.0.6 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? Hi, After upgrading from Airflow 2.3.3 to Airflow 3.0.6, I’ve noticed significantly higher memory usage, even when the system is mostly idle. Over time, memory consumption keeps increasing until swap is heavily used and eventually the system hits OOM. This looks similar to a memory leak. ### Observed behavior - After starting the service, within ~15 hours memory dropped from 16GB free RAM → 5GB free RAM. - Swap usage grew from 10% → 80%. - During this time, the same DAGs were running (some every 5 minutes, others every 1 minute). - `top` does not show significant memory usage because most of it is reported as inactive memory. Example from `vmstat -s`: 20334264 K total memory 13993448 K used memory 3498148 K active memory 10782420 K inactive memory ### Tests performed Disabled all DAGs → memory usage did not drop after 2 hours. Restarted Airflow components individually and observed memory released: - DAG processor: ~100MB - Triggerer: ~300MB - API server: ~300MB - Scheduler: ~10GB ### Notes DAGs are unchanged except for updates required by Airflow 3.0.6. On Airflow 2.3.3, this same setup, with only 10GB RAM, ran stably for months without memory issues. <img width="1151" height="587" alt="Image" src="https://github.com/user-attachments/assets/8ccd7adb-0fe2-46f2-b48e-a36447fca949"; /> <img width="967" height="637" alt="Image" src="https://github.com/user-attachments/assets/b5ee4083-642a-4f93-a91f-15df4adcf48c"; /> ### What you think should happen instead? Memory used by Airflow components (especially the scheduler) should be released when no longer in use, instead of growing indefinitely over time. ### How to reproduce I simply start the airflow and let my dags run. I have 6 ETL dags running that get data from other systems and insert them into elasticsearchdb. 5 dags run every 5 minutes. 1 dag runs every minute (This one uses taskgroups (multithreading)). ### Operating System RHEL 8.10 ### Versions of Apache Airflow Providers apache-airflow-providers-airbyte==5.2.3 apache-airflow-providers-alibaba==3.2.2 apache-airflow-providers-amazon==9.12.0 apache-airflow-providers-apache-beam==6.1.3 apache-airflow-providers-apache-cassandra==3.8.2 apache-airflow-providers-apache-drill==3.1.2 apache-airflow-providers-apache-druid==4.3.0 apache-airflow-providers-apache-flink==1.7.2 apache-airflow-providers-apache-hdfs==4.10.2 apache-airflow-providers-apache-hive==9.1.2 apache-airflow-providers-apache-iceberg==1.3.2 apache-airflow-providers-apache-impala==1.7.2 apache-airflow-providers-apache-kafka==1.10.2 apache-airflow-providers-apache-kylin==3.9.2 apache-airflow-providers-apache-livy==4.4.2 apache-airflow-providers-apache-pig==4.7.2 apache-airflow-providers-apache-pinot==4.8.2 apache-airflow-providers-apache-spark==5.3.2 apache-airflow-providers-apprise==2.1.2 apache-airflow-providers-arangodb==2.8.2 apache-airflow-providers-asana==2.10.2 apache-airflow-providers-atlassian-jira==3.1.2 apache-airflow-providers-celery==3.12.2 apache-airflow-providers-cloudant==4.2.2 apache-airflow-providers-cncf-kubernetes==10.7.0 apache-airflow-providers-cohere==1.5.3 apache-airflow-providers-common-compat==1.7.3 apache-airflow-providers-common-io==1.6.2 apache-airflow-providers-common-messaging==1.0.5 apache-airflow-providers-common-sql==1.27.5 apache-airflow-providers-databricks==7.7.1 apache-airflow-providers-datadog==3.9.2 apache-airflow-providers-dbt-cloud==4.4.2 apache-airflow-providers-dingding==3.8.2 apache-airflow-providers-discord==3.10.2 apache-airflow-providers-docker==4.4.2 apache-airflow-providers-edge3==1.1.3 apache-airflow-providers-elasticsearch==6.3.2 apache-airflow-providers-exasol==4.8.2 apache-airflow-providers-fab==2.4.1 apache-airflow-providers-facebook==3.8.2 apache-airflow-providers-ftp==3.13.2 apache-airflow-providers-git==0.0.6 apache-airflow-providers-github==2.9.2 apache-airflow-providers-google==17.1.0 apache-airflow-providers-grpc==3.8.2 apache-airflow-providers-hashicorp==4.3.2 apache-airflow-providers-http==5.3.3 apache-airflow-providers-imap==3.9.2 apache-airflow-providers-influxdb==2.9.3 apache-airflow-providers-jdbc==5.2.2 apache-airflow-providers-jenkins==4.1.2 apache-airflow-providers-microsoft-azure==12.6.1 apache-airflow-providers-microsoft-mssql==4.3.2 apache-airflow-providers-microsoft-psrp==3.1.2 apache-airflow-providers-microsoft-winrm==3.11.0 apache-airflow-providers-mongo==5.2.2 apache-airflow-providers-mysql==6.3.3 apache-airflow-providers-neo4j==3.10.0 apache-airflow-providers-odbc==4.10.2 apache-airflow-providers-openai==1.6.2 apache-airflow-providers-openfaas==3.8.2 apache-airflow-providers-openlineage==2.6.1 apache-airflow-providers-opensearch==1.7.2 apache-airflow-providers-opsgenie==5.9.2 apache-airflow-providers-oracle==4.1.3 apache-airflow-providers-pagerduty==5.0.2 apache-airflow-providers-papermill==3.11.2 apache-airflow-providers-pgvector==1.5.2 apache-airflow-providers-pinecone==2.3.3 apache-airflow-providers-postgres==6.2.3 apache-airflow-providers-presto==5.9.2 apache-airflow-providers-qdrant==1.4.3 apache-airflow-providers-redis==4.2.0 apache-airflow-providers-salesforce==5.11.2 apache-airflow-providers-samba==4.10.3 apache-airflow-providers-segment==3.8.2 apache-airflow-providers-sendgrid==4.1.3 apache-airflow-providers-sftp==5.3.4 apache-airflow-providers-singularity==3.8.2 apache-airflow-providers-slack==9.1.4 apache-airflow-providers-smtp==2.2.0 apache-airflow-providers-snowflake==6.4.0 apache-airflow-providers-sqlite==4.1.2 apache-airflow-providers-ssh==4.1.3 apache-airflow-providers-standard==1.6.0 apache-airflow-providers-tableau==5.2.0 apache-airflow-providers-telegram==4.8.2 apache-airflow-providers-teradata==3.2.1 apache-airflow-providers-trino==6.3.3 apache-airflow-providers-vertica==4.1.2 apache-airflow-providers-weaviate==3.2.2 apache-airflow-providers-yandex==4.2.0 apache-airflow-providers-ydb==2.2.2 apache-airflow-providers-zendesk==4.10.2 ### Deployment Virtualenv installation ### Deployment details OS: RHEL 8.10 - Deployment: systemd (non-containerized) - Python: 3.12 (virtualenv) - VM resources: 8 cores, 20GB RAM, 4GB swap - Airflow configs: Default ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
