GitHub user potiuk edited a comment on the discussion: Discussion: Preventing COW in LocalExecutor Workers
> Additionally, this approach could potentially prevent > https://github.com/apache/airflow/pull/56044 in forked processes. Very good point. I was looking earlier at that issue and It does look that gc.freeze() should help there without any MySQL specific solution and without adding unnecessary complexity in handling the connections. > Using memray, I found that this phenomenon was related to worker log file > creation. A memory leak can be observed in the section where worker log files > are generated. This is a great catch. I think this is a fantastic find. Yes what happens in that case - we are creating new loggger every time we start a new task and my guess is that the loggers are never deleted and hold all the related resources. It's the first time I look at the way how we are using structlog) - but I think for performance reasons we are using "cache_logger_on_first_use" and that holds the loggers we create in memory, held by a global cache dictionary - and for whatever reason (configuration likely - different file name used for different tasks) - each such logger for different tasks is a new cached logger that is only reused from cache while the task is running and then never deleted after the task completes - but I think it needs a bit deeper knowledge how structlog is used in this case in the Local Executor. Thanks for being so dilligent! @ashb @amoghrajesh -> WDYT ? I think those two finds are quite actionable and we should be able to apply some memory optimisations with it ? GitHub link: https://github.com/apache/airflow/discussions/58143#discussioncomment-14934739 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
