[
https://issues.apache.org/jira/browse/NIFI-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17050322#comment-17050322
]
Harald Dobbernack commented on NIFI-6790:
-----------------------------------------
We (NiFi Beginners) have been experiencing 'Too many open files' Problems with
an one node development system (NiFi 1.11.1) , but many Processors and some
Wait Processors relying on nofication via DistributedCache fed by Notify
Processor which get's fed by listfile Processors checking every 30sec with
yield and penalty set to 30sec. Even if the only active flows are in the wait
connection of the wait processor and nothing else is happening then the open
pipes will accumalate over time. In our use case the waiting flows can easily
be waiting a few days. From Friday Evening (starting with about 4000 open
Files for the nifi user) till Monday Morning with only three active flows we
managed to surpass 50000 open files. We will now as a workaround restart the
service regularly, but I am hoping a solution can be found. As a side note: we
had to put 'export FD_MAX=50000' in the nifi-env.sh, as the settings described
in https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html under
Maximum File Handles in _/etc/security/limits.conf_ did not suffice to be able
to set the max open file limit of nifi over 4096
> Load-balanced queues cause huge number of open pipes
> ----------------------------------------------------
>
> Key: NIFI-6790
> URL: https://issues.apache.org/jira/browse/NIFI-6790
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.9.2
> Environment: Ubuntu,OpenJDK8
> Reporter: Matthew Knight
> Priority: Critical
> Labels: Cluster, LoadBalancing
> Attachments: flow_exception.png, lsof_loadbalanced.png,
> lsof_notloadbalanced.png
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Even very basic flows using load balance connecting queues result in huge
> numbers of open pipes which don't seem to get closed and result in the entire
> cluster being brought to a standstill because of "too many open files". I
> tried apples-to-apples with equivalent very basic flows, one load balanced
> and one not, to show how easy this has been to reproduce. I'm attaching some
> screenshots that show the "too many open files" issue in action in my NiFi
> cluster, along with the output of LSOF for the primary node of a cluster with
> vs. without load balancing. When load balancing is enabled there is a huge
> number of 'pipe' and 'eventpoll' items, without load balancing things are a
> bit more balanced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
