zachliu opened a new issue, #50708:
URL: https://github.com/apache/airflow/issues/50708

   ### Apache Airflow version
   
   3.0.1
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   There was a significant memory leak (1 Gb/hour depending on DAGs), tracked 
in https://github.com/apache/airflow/issues/50097, which has been addressed by 
https://github.com/apache/airflow/pull/50558 or via this patch 
https://github.com/apache/airflow/issues/49887#issuecomment-2868630859. 
However, a much smaller leak remains—approximately 20 MB/hour—specifically in 
the `dag-processor`.
   
   
![Image](https://github.com/user-attachments/assets/fc5e660b-0253-42e7-9744-55f67347f01e)
   
   
![Image](https://github.com/user-attachments/assets/1212016a-11dc-4109-9388-612b78400dc6)
   
   There is evidence suggesting that this "pinhole" leak may not originate from 
the core `DagFileProcessorProcess` component. For example, while inspecting the 
code, I accidentally removed the comma in the following line:
   
   
https://github.com/apache/airflow/blob/fe1b11dcd8b78238c5cad06758a43794787c9cdb/airflow-core/src/airflow/dag_processing/processor.py#L252
   
   This caused the entire DAG processor to crash with the following error:
   
   ```
     File 
"/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/manager.py",
 line 890, in _create_process                                                   
                               
       return DagFileProcessorProcess.start(                                    
                                                                                
                                                    
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                    
                                                                                
                                                    
   TypeError: DagFileProcessorProcess.start() got an unexpected keyword 
argument 'id'
   ```
   
   
![Image](https://github.com/user-attachments/assets/4a60890d-a1dd-4423-bc99-81bfe3efb8c5)
   
   Despite the crash, the memory usage (`working_set`) continued to grow 
slowly, indicating that the leak may lie outside the core 
`DagFileProcessorProcess` logic.
   
   ### What you think should happen instead?
   
   Memory usage of the `dag-processor` shall be stable.
   
   ### How to reproduce
   
   Run Airflow with only the example DAGs.
   Run Airflow with no DAGs. 
   
   ### Operating System
   
   LM 21.3 with docker python:3.12-slim-bookworm
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [x] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to