What has made this difficult to track down is your usually suspects like heap usage aren’t what’s using all the memory. My production systems that are locking up like this show less than 50% heap usage and garbage collection seems to be functioning normally up until the point the box has run out of memory. Even docker stats shows the container not using all the ram while something like pmap shows the process consuming nearly all the memory on the box. Once I get a reproducible example I’ll start adding things like native memory tracking to see where the memory might be going.
Thanks Shawn From: Chris Sampson <[email protected]> Sent: Tuesday, November 23, 2021 4:01 PM To: [email protected] Subject: Re: ReplaceText Potential Memory Leak Is this a "JVM doesn't play well with containers"[1] sort of situation? I've read a couple of things around this area in the past but not too much in-depth because so far I've been lucky enough to avoid such issues (so far as I'm aware). Generally articles suggest that the various machinations of the different GCs often play a part - I suspect the default GC has changed between 8 & 11 (and even if it's the same GC, likely the implementation is different). So maybe it's a case of looking at the JVM flags at in bootstrap.conf and considering making these more easily configurable when running in docker (although I think nifi does have at least a little configuration in that area already using env vars)? Some applications ship with different default settings depending on the installation/deployment mechanism the user chooses (e.g. RPM vs. Docker) - Elasticsearch is one such example. JVM and logging are often things that are best using different defaults in such artefacts (e.g. for Docker the recommendation would normally be to log everything to STDOUT/ERR and not files that fill up the container and cause it to crash, etc.). [1] https://stackoverflow.com/questions/52607839/unexplained-extra-memory-consumed-by-docker-running-java-process Cheers, Chris Sampson On Tue, 23 Nov 2021, 20:21 Shawn Weeks, <[email protected]<mailto:[email protected]>> wrote: I’ve made some progress narrowing things down and I’m running a long term test to see if it fails. The issue seems to be an interaction between OpenJDK 11, Docker and NiFi where if you continuously run the heap usage up and down memory isn’t released and the NiFi Process slowly uses more and more memory until the host no longer functions My test is being run using the Dockerfile in NiFi’s Github repo with the only change being using the 11-jre image instead of the 8-jre base image. The memory leak seems to only be a few MB a day so it’s going to take a while to reproduce it. Thanks Shawn From: Shawn Weeks <[email protected]<mailto:[email protected]>> Sent: Monday, November 15, 2021 7:20 AM To: [email protected]<mailto:[email protected]> Subject: RE: ReplaceText Potential Memory Leak I’m still trying to get it narrowed down to a single example. It seems to be centered around ReplaceText and some really long lines(100MB+) but when it happens the servers go completely non-responsive so it’s been hard getting thread dumps and logs from them. Thanks Shawn From: Otto Fowler <[email protected]<mailto:[email protected]>> Sent: Sunday, November 14, 2021 9:54 PM To: [email protected]<mailto:[email protected]> Subject: Re: ReplaceText Potential Memory Leak I think this is worth a jira. Do you have a stack trace or anything? From: Shawn Weeks <[email protected]><mailto:[email protected]> Reply: [email protected]<mailto:[email protected]> <[email protected]><mailto:[email protected]> Date: November 8, 2021 at 10:09:12 To: [email protected]<mailto:[email protected]> <[email protected]><mailto:[email protected]> Subject: ReplaceText Potential Memory Leak I’m seeing an issue on 1.14.0 where if ReplaceText continuously hits a buffer overflow it will eventually overrun the memory on the server it’s running on and cause the server to stop responding at the OS Level. In my instance I’ve got vm.swappiness off and am running on the latest OpenJDK 11 on RedHat 7. I was wondering if anyone had seen this before. I’ve found some old java message boards that talk about text buffers bypassing the normal Java memory constraints but nothing all that recent. Example flow will be oncoming once I work out the details. Thanks Shawn
