[
https://issues.apache.org/jira/browse/NIFI-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17348128#comment-17348128
]
Pierre Villard commented on NIFI-8435:
--------------------------------------
In the configurations [~pgyori], [~exceptionfactory] and I tested, we think we
didn't have a memory leak leading to OOME. However we did observe a large
number of threads being instantiated by Netty and we noticed that the default
number would be a factor of the number of cores on the host. That's why we
think that if having many instances of the processor, with many concurrent
tasks and when running on nodes with a lot of cores, this can cause issues. We
think that this change is enough because it gives the option to override this
default value. There is still the open Netty issue linked in this JIRA that we
are going to monitor closely. To be honest, if you have the chance to test this
change in one of your environments, that would definitely be a valuable
feedback and we can re-open the JIRA if the situation didn't improve.
> PutKudu 1.13.2 Memory Leak
> --------------------------
>
> Key: NIFI-8435
> URL: https://issues.apache.org/jira/browse/NIFI-8435
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.13.2
> Environment: NiFi 1.13.2, 8-Node Cluster running on CentOS 7, Kudu
> 1.10.0
> Reporter: Josef Zahner
> Assignee: Peter Gyori
> Priority: Critical
> Labels: kudu, nifi, oom
> Fix For: 1.14.0
>
> Attachments: Screenshot 2021-04-20 at 14.27.11.png,
> grafana_heap_overview.png, kudu_inserts_per_sec.png,
> putkudu_processor_config.png, visualvm_bytes_detail_view.png,
> visualvm_total_bytes_used.png
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> We just upgraded from NiFi 1.11.4 to 1.13.2 and faced a huge issue with
> PutKudu.
> PutKudu on the 1.13.2 eats up all the heap memory and garbage collection
> can't anymore free up the memory. We allow Java to use 31GB memory and as you
> can see with NiFi 1.11.4 it will be used like it should with GC. However with
> NiFi 1.13.2 with our actual load it fills up the memory relatively fast.
> Manual GC via visualvm tool didn't help at all to free up memory.
> !grafana_heap_overview.png!
>
> Visual VM shows the following culprit: !visualvm_total_bytes_used.png!
> !visualvm_bytes_detail_view.png!
> The bytes array shows millions of char data which isn't cleaned up. In fact
> here 14,9GB memory (heapdump has been taken after a while of full load). If
> we check the same on NiFi 1.11.4, the bytes array is nearly empty, around a
> few hundred MBs.
> As you could imagine we can't upload the heap dump as currently we have only
> productive data on the system. But don't hesitate to ask questions about the
> heapdump if you need more information.
> I haven't done any screenshot of the processor config, but I can do that if
> you wish (we are back to NiFi 1.11.4 at the moment).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
