potiuk commented on issue #16703: URL: https://github.com/apache/airflow/issues/16703#issuecomment-1109054650
> @potiuk Did you have a fix related to memory usage of logging? @dan-origami Are you able to test with 2.2.x? Yes. The graphs look very much like what I've fixed. But ut was not (and could not) cause crash - as it was not a real memory usage of the application but the kernel cache for log files (it would never be a cause for crash - the memory is freed and removed if needed). The observed memory behaviour matches what my change addressed (overal growth of memory for the working set but the container RSS memory staying flat). If that was the case, then the crash could be caused by something else and observed memory build-up is unrelated. The kernel advisory (to not cache the written logs) has been released in https://airflow.apache.org/docs/apache-airflow/stable/changelog.html#airflow-2-2-0-2021-10-11 with this fix: https://github.com/apache/airflow/pull/18054 and the history of the issue is here https://github.com/apache/airflow/issues/14924 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
