[
https://issues.apache.org/jira/browse/NIFI-6790?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16954090#comment-16954090
]
Mark Payne commented on NIFI-6790:
----------------------------------
[~Matthew Knight] thanks for reporting this! I'm trying to replicate this on
master but with no luck:
{code:java}
nifi-1.10.0-SNAPSHOT-bin $ lsof -p 21433 | wc -l
5634
nifi-1.10.0-SNAPSHOT-bin $ lsof -p 21433 | wc -l
5661
nifi-1.10.0-SNAPSHOT-bin $ lsof -p 21433 | wc -l
5658
nifi-1.10.0-SNAPSHOT-bin $ lsof -p 21433 | wc -l
5660
nifi-1.10.0-SNAPSHOT-bin $ lsof -p 21433 | wc -l
5655
nifi-1.10.0-SNAPSHOT-bin $ lsof -p 21433 | wc -l
5657 {code}
The number of open file handles for the nifi process is quite steady. If I look
at only PIPE file handles, I see it very steady as well:
{code:java}
nifi-1.10.0-SNAPSHOT-bin $ lsof -p 21433 | grep PIPE | wc -l
27
nifi-1.10.0-SNAPSHOT-bin $ lsof -p 21433 | grep PIPE | wc -l
27
nifi-1.10.0-SNAPSHOT-bin $ lsof -p 21433 | grep PIPE | wc -l
27
nifi-1.10.0-SNAPSHOT-bin $ lsof -p 21433 | grep PIPE | wc -l
27 {code}
I'm curious what's different. How long did you allow your system to run before
starting to see an uptick in open file handles? Can you try running the command
that I showed above and see if you get something different?
> Load-balanced queues cause huge number of open pipes
> ----------------------------------------------------
>
> Key: NIFI-6790
> URL: https://issues.apache.org/jira/browse/NIFI-6790
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
> Affects Versions: 1.9.2
> Environment: Ubuntu,OpenJDK8
> Reporter: Matthew Knight
> Priority: Critical
> Labels: Cluster, LoadBalancing
> Attachments: flow_exception.png, lsof_loadbalanced.png,
> lsof_notloadbalanced.png
>
> Original Estimate: 672h
> Remaining Estimate: 672h
>
> Even very basic flows using load balance connecting queues result in huge
> numbers of open pipes which don't seem to get closed and result in the entire
> cluster being brought to a standstill because of "too many open files". I
> tried apples-to-apples with equivalent very basic flows, one load balanced
> and one not, to show how easy this has been to reproduce. I'm attaching some
> screenshots that show the "too many open files" issue in action in my NiFi
> cluster, along with the output of LSOF for the primary node of a cluster with
> vs. without load balancing. When load balancing is enabled there is a huge
> number of 'pipe' and 'eventpoll' items, without load balancing things are a
> bit more balanced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
