[
https://issues.apache.org/jira/browse/NIFI-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954606#comment-16954606
]
Mark Payne commented on NIFI-6790:
----------------------------------
[~Matthew Knight] thanks for the update. I do notice in your screenshot that
you've got a 100-node cluster, which is certainly different than the 2-node
cluster that I've got running to try to replicate. Are you using default
settings for load balanced configuration in your `nifi.properties`? Also, when
you say "this is still present in master as of a few days ago" - do you know
how long ago that was? I'm wondering if this problem is addressed by
https://issues.apache.org/jira/browse/NIFI-6736. What we saw in that that Jira
was that there could be a lot of sockets left open in a situation like this.
But it wasn't reported as pipe's. But that may be a difference in OSX vs.
Ubuntu, perhaps?
> Load-balanced queues cause huge number of open pipes
> ----------------------------------------------------
>
> Key: NIFI-6790
> URL: https://issues.apache.org/jira/browse/NIFI-6790
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.9.2
> Environment: Ubuntu,OpenJDK8
> Reporter: Matthew Knight
> Priority: Critical
> Labels: Cluster, LoadBalancing
> Attachments: flow_exception.png, lsof_loadbalanced.png,
> lsof_notloadbalanced.png
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Even very basic flows using load balance connecting queues result in huge
> numbers of open pipes which don't seem to get closed and result in the entire
> cluster being brought to a standstill because of "too many open files". I
> tried apples-to-apples with equivalent very basic flows, one load balanced
> and one not, to show how easy this has been to reproduce. I'm attaching some
> screenshots that show the "too many open files" issue in action in my NiFi
> cluster, along with the output of LSOF for the primary node of a cluster with
> vs. without load balancing. When load balancing is enabled there is a huge
> number of 'pipe' and 'eventpoll' items, without load balancing things are a
> bit more balanced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
