I’ve made some progress narrowing things down and I’m running a long term test to see if it fails. The issue seems to be an interaction between OpenJDK 11, Docker and NiFi where if you continuously run the heap usage up and down memory isn’t released and the NiFi Process slowly uses more and more memory until the host no longer functions My test is being run using the Dockerfile in NiFi’s Github repo with the only change being using the 11-jre image instead of the 8-jre base image. The memory leak seems to only be a few MB a day so it’s going to take a while to reproduce it.
Thanks Shawn From: Shawn Weeks <[email protected]> Sent: Monday, November 15, 2021 7:20 AM To: [email protected] Subject: RE: ReplaceText Potential Memory Leak I’m still trying to get it narrowed down to a single example. It seems to be centered around ReplaceText and some really long lines(100MB+) but when it happens the servers go completely non-responsive so it’s been hard getting thread dumps and logs from them. Thanks Shawn From: Otto Fowler <[email protected]<mailto:[email protected]>> Sent: Sunday, November 14, 2021 9:54 PM To: [email protected]<mailto:[email protected]> Subject: Re: ReplaceText Potential Memory Leak I think this is worth a jira. Do you have a stack trace or anything? From: Shawn Weeks <[email protected]><mailto:[email protected]> Reply: [email protected]<mailto:[email protected]> <[email protected]><mailto:[email protected]> Date: November 8, 2021 at 10:09:12 To: [email protected]<mailto:[email protected]> <[email protected]><mailto:[email protected]> Subject: ReplaceText Potential Memory Leak I’m seeing an issue on 1.14.0 where if ReplaceText continuously hits a buffer overflow it will eventually overrun the memory on the server it’s running on and cause the server to stop responding at the OS Level. In my instance I’ve got vm.swappiness off and am running on the latest OpenJDK 11 on RedHat 7. I was wondering if anyone had seen this before. I’ve found some old java message boards that talk about text buffers bypassing the normal Java memory constraints but nothing all that recent. Example flow will be oncoming once I work out the details. Thanks Shawn
