zachliu opened a new issue, #50708: URL: https://github.com/apache/airflow/issues/50708
### Apache Airflow version 3.0.1 ### If "Other Airflow 2 version" selected, which one? _No response_ ### What happened? There was a significant memory leak (1 Gb/hour depending on DAGs), tracked in https://github.com/apache/airflow/issues/50097, which has been addressed by https://github.com/apache/airflow/pull/50558 or via this patch https://github.com/apache/airflow/issues/49887#issuecomment-2868630859. However, a much smaller leak remains—approximately 20 MB/hour—specifically in the `dag-processor`.   There is evidence suggesting that this "pinhole" leak may not originate from the core `DagFileProcessorProcess` component. For example, while inspecting the code, I accidentally removed the comma in the following line: https://github.com/apache/airflow/blob/fe1b11dcd8b78238c5cad06758a43794787c9cdb/airflow-core/src/airflow/dag_processing/processor.py#L252 This caused the entire DAG processor to crash with the following error: ``` File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 890, in _create_process return DagFileProcessorProcess.start( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: DagFileProcessorProcess.start() got an unexpected keyword argument 'id' ```  Despite the crash, the memory usage (`working_set`) continued to grow slowly, indicating that the leak may lie outside the core `DagFileProcessorProcess` logic. ### What you think should happen instead? Memory usage of the `dag-processor` shall be stable. ### How to reproduce Run Airflow with only the example DAGs. Run Airflow with no DAGs. ### Operating System LM 21.3 with docker python:3.12-slim-bookworm ### Versions of Apache Airflow Providers _No response_ ### Deployment Docker-Compose ### Deployment details _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
