[
https://issues.apache.org/jira/browse/NIFI-8435?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17325677#comment-17325677
]
Josef Zahner edited comment on NIFI-8435 at 4/20/21, 10:09 AM:
---------------------------------------------------------------
[~pvillard] I've added the processor config below. We can clearly reproduce the
memory leak issue within our lab with the same config on a 4-node NiFi cluster.
We can reproduce it as well on a single node NiFi with just one PutKudu.
Yes we are using multiple concurrent tasks and on our actual config we have
about 34 PutKudu processors running in parallel (with multiple threads).
!putkudu_processor_config.png|width=1042,height=729!
As emergency fix we have built the NAR from 1.11.4 for NiFi 1.13.2. It runs
since a few hours and no leak so far. But of course we would like to use the
official PutKudu and not an old version of it.
was (Author: jzahner):
[~pvillard] I've added the processor config below. We can clearly reproduce the
memory leak issue within our lab with the same config on a 4-node NiFi cluster.
Yes we are using multiple concurrent tasks and on our actual config we have
about 34 PutKudu processors running in parallel (with multiple threads).
!putkudu_processor_config.png|width=1042,height=729!
As emergency fix we have built the NAR from 1.11.4 for NiFi 1.13.2. It runs
since a few hours and no leak so far. But of course we would like to use the
official PutKudu and not an old version of it.
> PutKudu 1.13.2 Memory Leak
> --------------------------
>
> Key: NIFI-8435
> URL: https://issues.apache.org/jira/browse/NIFI-8435
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.13.2
> Environment: NiFi 1.13.2, 8-Node Cluster running on CentOS 7, Kudu
> 1.10.0
> Reporter: Josef Zahner
> Assignee: Peter Gyori
> Priority: Critical
> Labels: kudu, nifi, oom
> Attachments: grafana_heap_overview.png, putkudu_processor_config.png,
> visualvm_bytes_detail_view.png, visualvm_total_bytes_used.png
>
>
> We just upgraded from NiFi 1.11.4 to 1.13.2 and faced a huge issue with
> PutKudu.
> PutKudu on the 1.13.2 eats up all the heap memory and garbage collection
> can't anymore free up the memory. We allow Java to use 31GB memory and as you
> can see with NiFi 1.11.4 it will be used like it should with GC. However with
> NiFi 1.13.2 with our actual load it fills up the memory relatively fast.
> Manual GC via visualvm tool didn't help at all to free up memory.
> !grafana_heap_overview.png!
>
> Visual VM shows the following culprit: !visualvm_total_bytes_used.png!
> !visualvm_bytes_detail_view.png!
> The bytes array shows millions of char data which isn't cleaned up. In fact
> here 14,9GB memory (heapdump has been taken after a while of full load). If
> we check the same on NiFi 1.11.4, the bytes array is nearly empty, around a
> few hundred MBs.
> As you could imagine we can't upload the heap dump as currently we have only
> productive data on the system. But don't hesitate to ask questions about the
> heapdump if you need more information.
> I haven't done any screenshot of the processor config, but I can do that if
> you wish (we are back to NiFi 1.11.4 at the moment).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
