GitHub user wjddn279 added a comment to the discussion: Discussion: Preventing COW in LocalExecutor Workers
With the above work, the COW issue has been resolved, and we no longer observe cases of sudden large memory increases. However, I did notice a gradual increase in heap memory per worker. In a high-load environment where 100 DAGs with 5 tasks each run every minute, memory increased by about 20MB over 12 hours. Using memray, I found that this phenomenon was related to worker log file creation. A memory leak can be observed in the section where worker log files are generated. [1106-2315.html](https://github.com/user-attachments/files/23470528/1106-2315.html) [1107-1209.html](https://github.com/user-attachments/files/23470531/1107-1209.html) This issue occurs identically in the current version 3.1.2, confirming it's not related to gc.freeze. However, if log file creation doesn't happen frequently, it's likely not a significant problem. I think it would be good to resolve this COW issue first, then have discussions to address this separate issue. GitHub link: https://github.com/apache/airflow/discussions/58143#discussioncomment-14933050 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
