potiuk commented on issue #28740: URL: https://github.com/apache/airflow/issues/28740#issuecomment-1374795388
> @potiuk , We upgraded to EKS cluster 1.23 but still issue persist memory leak with respect to Workers liveness probe and scheduler pods as well. Which memory is it ? Did you take a look at https://github.com/apache/airflow/pull/27223 and related issue: https://github.com/apache/airflow/issues/27065 and https://github.com/apache/airflow/issues/14924 ? Questions you should find answers to: * Is it a working set memory growing or cache: https://github.com/apache/airflow/issues/14924#issuecomment-914153622 * Does ite eventually cause your containers to crash with OOM (Out of Memory) or does the memory clears itself out when needed? * Does it go away when you (after execing in to the containers that you observe memory growth in) go away when you clear the cache manualy https://github.com/apache/airflow/issues/14924#issuecomment-914143098 * Does it go away when you clear the cache for the whole node that your Airflow pods are running on. * Does it go away when you rotate/compress/remove log files that Airflow produces. * If it all shows that this is the "cache" issue (which is BTW not an issue, it's normal and you just observe wrong memory) - then upgrading to 2.4.3 should remove at least some of that cache being created in the first place Getting answers to those question should help you to understanda if what you have is a cache issue or something else. * If it is something else, then likely you need to find out which process is leaking memory. It might be airflow, but in case you have some monitoring or logging agents for example. Those are the experiments and checks you should do next. I also recommend reading more about memory issues in the threads I linked to, they contain a wealth of information and links that can help you to pin-point not only Airflow-related but also other, similar issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
