[
https://issues.apache.org/jira/browse/NIFI-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954610#comment-16954610
]
Matthew Knight edited comment on NIFI-6790 at 10/18/19 1:30 PM:
----------------------------------------------------------------
It was after 6736 got merged, I was trying to see if that fixed it, leaky pipes
were still a problem.
nifi.cluster.load.balance.connections.per.node=10
nifi.cluster.load.balance.max.thread.count=10
nifi.cluster.load.balance.comms.timeout=30 sec
was (Author: matthew knight):
It was after 6736 got merged, I was trying to see if that fixed it, leaky pipes
were skill a problem.
nifi.cluster.load.balance.connections.per.node=10
nifi.cluster.load.balance.max.thread.count=10
nifi.cluster.load.balance.comms.timeout=30 sec
> Load-balanced queues cause huge number of open pipes
> ----------------------------------------------------
>
> Key: NIFI-6790
> URL: https://issues.apache.org/jira/browse/NIFI-6790
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.9.2
> Environment: Ubuntu,OpenJDK8
> Reporter: Matthew Knight
> Priority: Critical
> Labels: Cluster, LoadBalancing
> Attachments: flow_exception.png, lsof_loadbalanced.png,
> lsof_notloadbalanced.png
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Even very basic flows using load balance connecting queues result in huge
> numbers of open pipes which don't seem to get closed and result in the entire
> cluster being brought to a standstill because of "too many open files". I
> tried apples-to-apples with equivalent very basic flows, one load balanced
> and one not, to show how easy this has been to reproduce. I'm attaching some
> screenshots that show the "too many open files" issue in action in my NiFi
> cluster, along with the output of LSOF for the primary node of a cluster with
> vs. without load balancing. When load balancing is enabled there is a huge
> number of 'pipe' and 'eventpoll' items, without load balancing things are a
> bit more balanced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
