HI David & Joe, Thanks for the help and sorry for the late reply. DirectByteBuffers might be the key to all this. What in NiFi would use a direct byte buffer and prevent it from being cleaned up by GC? Maybe related to reading / writing flowfile content? Anyway I tried the suggested 2gb heap limit and it's so far so good! Overall memory use is around 4gb and now I know why it's close to double the heap limit.
Thanks, Eric On Wed, Aug 16, 2023 at 6:03 PM David Handermann < [email protected]> wrote: > Eric, > > It is important to note that Java process memory usage consists of > multiple factors beyond the maximum heap size. Kubernetes limits are based > on total memory usage, so it is essential to account for all Java memory > elements when specifying Kubernetes memory limits. > > The following article has a helpful summary of Java Virtual Machine memory > usage: > > > https://developers.redhat.com/articles/2021/09/09/how-jvm-uses-and-allocates-memory > > The formula for memory usage is helpful: > > > JVM memory = Heap memory + Metaspace + CodeCache + (ThreadStackSize * > Number of Threads) + DirectByteBuffers + Jvm-native > > Based on that formula, several key factors include the total number of > threads, and the size of direct byte buffers. > > Specifically, it is important to highlight that the default direct byte > buffer limit is equal to the maximum heap size. This means that unless the > size of direct memory is configured for the JVM, using a custom argument in > bootstrap.conf, the maximum heap size should be something less than half of > Kubernetes memory limits. > > All of these factors will be different based on the particular Processors > configured, and the specific configuration parameters selected. Some > Processors use direct byte buffers, others do not, so it depends on the > supporting libraries used. > > That's a longer way of saying, Joe's suggestion of 2 GB for maximum heap > size should be a safe starting point with 8 GB as the memory limit to avoid > OOM killed pods. > > Regards, > David Handermann > > On Wed, Aug 16, 2023 at 6:08 PM Joe Witt <[email protected]> wrote: > >> Eric >> >> Try using 2GB for the heap and seeing that helps. I also believe there >> are specific pod settings youll want to use to avoid it getting nuked by >> k8s. >> >> This blog may give you great things to consider >> https://home.robusta.dev/blog/kubernetes-memory-limit >> >> Thanks >> >> On Wed, Aug 16, 2023 at 3:29 PM Eric Secules <[email protected]> wrote: >> >>> My nifi.properties uses the filesystem repository implementations for >>> everything except the component status repository. So it's sticking to the >>> defaults. Could the VolatileComponentStatusRepository be what's eating >>> up my memory. It would fit well with the use pattern of the test framework >>> creating many processors and connections but deleting a processor also >>> makes its component status unavailable, so I would assume that also >>> releases associated memory. >>> >>> On Wed, Aug 16, 2023 at 2:35 PM Eric Secules <[email protected]> wrote: >>> >>>> Hi, >>>> >>>> I was wondering what the recommendation is for how much memory to start >>>> the nifi jvm with and then how much to set the docker container memory >>>> limit as a function of the jvm memory setting. >>>> >>>> I have nifi started with -xms and -xmx set to 4gb and docker memory >>>> resource limits of 8GB on the container for a small test environment. And >>>> it gets killed by k8s for consuming too much memory after the tests run for >>>> a while. My workload is automated tests where each test puts a flow (avg 70 >>>> processors) on the canvas, runs files through, and removes the flow when >>>> the test is done. About 4 of these tests run in parallel and we are only >>>> testing flow functionality so the flows only deal with 10s of flowfiles >>>> each. >>>> >>>> I am still on version 1.14.0 and I would like to know what other things >>>> in nifi contribute to memory use besides flowfiles in flight. Also is it >>>> normal that I see my containers using much more memory than I allotted to >>>> the jvm? >>>> >>>> Thanks, >>>> Eric >>>> >>>
