[
https://issues.apache.org/jira/browse/NIFI-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17326617#comment-17326617
]
David Handermann commented on NIFI-8435:
----------------------------------------
Thanks for the confirmation [~jzahner]. In attempting to reproduce the issue,
I observed that the each KuduClient instance can spawn a number of threads in
some circumstances. On a small NiFi instance, stopping the PutKudu processor
temporarily increases the number of threads. More recent versions of the Kudu
client library incorporated an upgrade from Netty 3 to Netty 4, which included
changes to how Netty handles pooled byte buffering. Initial testing with Netty
leak detection enabled did not yield any definite findings, but memory usage is
clearly connected directly to the number of PutKudu Processor instances, and
the associated KuduClient instances. The KuduClient supports configuring the
number of worker threads, so this may be an area of improvement, although it
does not appear to be a complete answer to the problem.
> PutKudu 1.13.2 Memory Leak
> --------------------------
>
> Key: NIFI-8435
> URL: https://issues.apache.org/jira/browse/NIFI-8435
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.13.2
> Environment: NiFi 1.13.2, 8-Node Cluster running on CentOS 7, Kudu
> 1.10.0
> Reporter: Josef Zahner
> Assignee: Peter Gyori
> Priority: Critical
> Labels: kudu, nifi, oom
> Attachments: Screenshot 2021-04-20 at 14.27.11.png,
> grafana_heap_overview.png, kudu_inserts_per_sec.png,
> putkudu_processor_config.png, visualvm_bytes_detail_view.png,
> visualvm_total_bytes_used.png
>
>
> We just upgraded from NiFi 1.11.4 to 1.13.2 and faced a huge issue with
> PutKudu.
> PutKudu on the 1.13.2 eats up all the heap memory and garbage collection
> can't anymore free up the memory. We allow Java to use 31GB memory and as you
> can see with NiFi 1.11.4 it will be used like it should with GC. However with
> NiFi 1.13.2 with our actual load it fills up the memory relatively fast.
> Manual GC via visualvm tool didn't help at all to free up memory.
> !grafana_heap_overview.png!
>
> Visual VM shows the following culprit: !visualvm_total_bytes_used.png!
> !visualvm_bytes_detail_view.png!
> The bytes array shows millions of char data which isn't cleaned up. In fact
> here 14,9GB memory (heapdump has been taken after a while of full load). If
> we check the same on NiFi 1.11.4, the bytes array is nearly empty, around a
> few hundred MBs.
> As you could imagine we can't upload the heap dump as currently we have only
> productive data on the system. But don't hesitate to ask questions about the
> heapdump if you need more information.
> I haven't done any screenshot of the processor config, but I can do that if
> you wish (we are back to NiFi 1.11.4 at the moment).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
