zachliu opened a new issue, #27065: URL: https://github.com/apache/airflow/issues/27065
### Apache Airflow version 2.4.1 ### What happened My Airflow scheduler memory usage started to grow after I turned on the `dag_processor_manager` log by doing ```bash export CONFIG_PROCESSOR_MANAGER_LOGGER=True ``` see the red arrow below  By looking closely at the memory usage as mentioned in https://github.com/apache/airflow/issues/16737#issuecomment-917677177, I discovered that it was the cache memory that's keep growing:  Then I turned off the `dag_processor_manager` log, memory usage returned to normal (not growing anymore, steady at ~400 MB) This issue is similar to #14924 and #16737. This time the culprit is the rotating logs under `~/logs/dag_processor_manager/dag_processor_manager.log*`. ### What you think should happen instead Cache memory shouldn't keep growing like this ### How to reproduce Turn on the `dag_processor_manager` log by doing ```bash export CONFIG_PROCESSOR_MANAGER_LOGGER=True ``` in the `entrypoint.sh` and monitor the scheduler memory usage ### Operating System Debian GNU/Linux 10 (buster) ### Versions of Apache Airflow Providers _No response_ ### Deployment Other Docker-based deployment ### Deployment details k8s ### Anything else _No response_ ### Are you willing to submit PR? - [ ] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
