[
https://issues.apache.org/jira/browse/NIFI-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325717#comment-17325717
]
Josef Zahner commented on NIFI-8435:
------------------------------------
Ok perfect, I can see the sawtooth as well until the heap is exhausted (and the
sawtooth gets smaller as more the memory has been filled up). Please have a
look at the heap "byte" objects, at least if you generate enough flows/records.
I don't think that 3 small records every 0.1 seconds are enough. In my initial
grafana screenshot, where you see the heap size, we were inserting about 600K
records per second, just that you get a rough estimation of our workload and
how long it took to see the issue.
!kudu_inserts_per_sec.png!
> PutKudu 1.13.2 Memory Leak
> --------------------------
>
> Key: NIFI-8435
> URL: https://issues.apache.org/jira/browse/NIFI-8435
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.13.2
> Environment: NiFi 1.13.2, 8-Node Cluster running on CentOS 7, Kudu
> 1.10.0
> Reporter: Josef Zahner
> Assignee: Peter Gyori
> Priority: Critical
> Labels: kudu, nifi, oom
> Attachments: grafana_heap_overview.png, kudu_inserts_per_sec.png,
> putkudu_processor_config.png, visualvm_bytes_detail_view.png,
> visualvm_total_bytes_used.png
>
>
> We just upgraded from NiFi 1.11.4 to 1.13.2 and faced a huge issue with
> PutKudu.
> PutKudu on the 1.13.2 eats up all the heap memory and garbage collection
> can't anymore free up the memory. We allow Java to use 31GB memory and as you
> can see with NiFi 1.11.4 it will be used like it should with GC. However with
> NiFi 1.13.2 with our actual load it fills up the memory relatively fast.
> Manual GC via visualvm tool didn't help at all to free up memory.
> !grafana_heap_overview.png!
>
> Visual VM shows the following culprit: !visualvm_total_bytes_used.png!
> !visualvm_bytes_detail_view.png!
> The bytes array shows millions of char data which isn't cleaned up. In fact
> here 14,9GB memory (heapdump has been taken after a while of full load). If
> we check the same on NiFi 1.11.4, the bytes array is nearly empty, around a
> few hundred MBs.
> As you could imagine we can't upload the heap dump as currently we have only
> productive data on the system. But don't hesitate to ask questions about the
> heapdump if you need more information.
> I haven't done any screenshot of the processor config, but I can do that if
> you wish (we are back to NiFi 1.11.4 at the moment).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
