[
https://issues.apache.org/jira/browse/NIFI-7114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17041973#comment-17041973
]
Paul Kelly commented on NIFI-7114:
----------------------------------
It looks like lsof -p (pid) is growing for me as well. I adjusted the
scheduling on the GenerateFlowFile to generate 200 files every 45 seconds and
changed the size to 100B. Now I don't have to start and stop the processor to
see the batching effect. I've only had it running for about 15 minutes and
already the FIFOs within LSOF -p have grown from 21 to 39. I'll keep it
running. This isn't nearly as dramatic as our production flow, but it is
happening.
I upgraded our production environment to 1.11.2 again just now and have it
running without restarts. I will try to grab some logs when it crashes.
I agree with [~vzolin] that it seems to be related to our use of
DistributedMapCache. Setting up a new flow using one on 1.10.0 (on Windows)
involving a DistributedMapCache was the first time I experienced an running out
of available TCP client sockets at the OS level with NiFi. After upgrading our
Linux servers to 1.10.0 a week or so later, we ran the file descriptor issue
with FIFOs. From some research, it seems that java.nio uses socket pairs on
Windows and pipes on Unix for IPC. So a leak with some java.nio IPC would
cause what we are seeing on both OSes.
> NiFi not closing file handles
> -----------------------------
>
> Key: NIFI-7114
> URL: https://issues.apache.org/jira/browse/NIFI-7114
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
> Affects Versions: 1.10.0, 1.11.0
> Environment: Amazon EC2 running either Amazon Linux 2 or Ubuntu 18.04.
> NiFi has been installed with no change to any configuration file.
> Reporter: Vinicius Zolin
> Priority: Major
> Attachments: destination.xml, fifocounts.txt, flow.xml.gz, lsof.log,
> lsof.zip, lsofAfter.log, lsofBefore.log, openFiles.xlsx, reproduction.zip,
> source.xml
>
>
> Since at least version 1.10 NiFi stopped closing file handles. It opens circa
> 500 files per hour (measured using lsof) without any apparent limit until it
> crashes due to too many open files.
>
> Increasing the computer open file limit is not a solution since NiFi will
> still crash, it'll only take longer to do so.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
