potiuk commented on issue #27589: URL: https://github.com/apache/airflow/issues/27589#issuecomment-1316103077
Do you see the same leaks WITHOUT modifying anything from basic airflow? I do not know if your modifications changed it - but just comparing it with "baseline" might give a hint. The whole point about debugging such problems is that they might be casue by changes that do not look suspicious and the BEST way of debugging them is to do something call bisecting - which is a valid debugging technique: 1) You have a base system with no modification at all and no memory leak 2) Yoy have a system with some modifications and a memory leak Now - you are iterating over changes you made until you find tht one single change that causes the leak. This is usually fastest and most effective way of finding the root cause. If you have no obvious reason, this is the ONLY way. Even if you feel like "not much", I've seen totally unexpected things happening with "inocoulous" change. > @potiuk how do I know which process eats memory? If I only knew a straightforward answer, I would give it to you. I usually use top or better Htop to observe what's going on and then I try to dig deeper if I see anything suspicious. But this has its limits due to complex nature of memory usage on Linux. I am afraid I am not able to give simple answer to "how to check memory" - depending on which memory you observe leaking there might be sever different places, but none of them have simple recipes to use. Simply because memory in Linux is extremely complex subject - much more complex that you can think. There are various ways applications, and kernel use memory and simple "do that" solution does not exist. If you try to find how to do it in google you will find plenty of "how you can approach it" - first example I found is this https://www.linuxfoundation.org/blog/blog/classic-sysadmin-linux-101-5-commands-for-checking-memory-usage-in-linux - where youbut you will not get direct answers but you will get a few tools that you can try to use to see if any of those will give you some of answer. In many cases the kernel memory used will grow but it won't be attributed to eny single process (even if they are originated from the same process). At other times the memory used between processes will be partially (seemingly) duplicated, because they share "copy on wite" memory when processes forked and most of it is still shared. Likely observing how various memory values in htop (suggested over top) should give you some clues. But It can be a kernel that is leaking memory and you will not see it there - https://unix.stackexchange.com/questions/97261/how-much-ram-does-the-kernel-use have some other debugging techniques to see it. Most likely if you do not see any of the processes that are leaking memory, then likely you have kernel leaking it - which might mean many things - including your K8S instance is has for example shared volume with a buggy library (which will be nothing airflow might be aware about). Or even the monitoring software (i.e. grafana agent) might cause it. Hard to say and I am afraid I cannot help more by "try to pin-point the root cause". This is also why pin-pointing is very important and often the fastest way to debug stuff . No-one will be able to "guess" what it is by even looking at the modification but getting it down to single change causing the leak might help in getting closer where to look for it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
