[
https://issues.apache.org/jira/browse/NIFI-9767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17502275#comment-17502275
]
David Handermann commented on NIFI-9767:
----------------------------------------
Thanks for reporting this issue [~jian659]. Can you provide some additional
details about your system configuration? In particular, providing the amount
of physical memory, along with the maximum heap size configured in
bootstrap.conf would be helpful.
The {{PutKudu}} processor includes the following system resource consideration
note regarding memory:
??An instance of this component can cause high usage of this system resource.
Multiple instances or high concurrency settings may result a degradation of
performance.??
The Kudu client leverages the Netty library, which can result in using notable
amounts of memory due to thread pool configuration. Each PutKudu processor
instances creates a new thread pool, which directly impacts memory consumption.
For this reason, the number of PutKudu processors in a given NiFi flow should
be as limited as possible. The {{Kudu Client Worker Count}} property controls
the number of client threads for the processor instance. The default value is
two times the number of reported CPU cores. When configuring multiple
instances of PutKudu, the worker count property should be reduced to lessen the
impact on memory consumption. The worker count property should generally align
with the concurrent task configuration for the processor.
With that background, limiting the number of PutKudu processor instances, and
reducing the worker count for each processor, should help address memory
consumption.
> PutKudu OutOfMemory Exception (GC overhead limit exceeded)
> ----------------------------------------------------------
>
> Key: NIFI-9767
> URL: https://issues.apache.org/jira/browse/NIFI-9767
> Project: Apache NiFi
> Issue Type: Bug
> Components: Extensions
> Affects Versions: 1.15.0
> Environment: NiFi 1.15.0
> Reporter: jason
> Priority: Blocker
> Attachments: free_heap.png, pipeline.png, total_threads.png
>
>
> PutKudu 1.15.0 eats up all the heap memory. Must restart NiFi to free heap.
> We create twenty the same of pipelines for test, will out of memory on few
> hours.Pipeline as shown in pipeline.png
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
