[
https://issues.apache.org/jira/browse/NIFI-7454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Simon Bence resolved NIFI-7454.
-------------------------------
Resolution: Duplicate
*Summary:*
This issue is fixed by [#NIFI-7292]. There is an ongoing discussion about a
1.12.0 release or a 1.11.5 release to happen soon.
*In details:*
I could reproduce this easily (thank you for the flow!) using MacOS. It turned
out that the stream usage keeps a reference alive after the exception (based on
what I did see, most probably it's a _sun.nio.fs.UnixPath_, but I did not
follow up with this). The fix with the current version of _ListFile_ looks
fairly simple:
{code:java}
finally {
performanceTracker.completeActiveDirectory();
listing.close();
}
{code}
If the listing is closed
[here|https://github.com/apache/nifi/blob/rel/nifi-1.11.4/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/ListFile.java#L579]
the handler will be released. This issue does not effect the master branch
however, as the listing of the files was replaced but a different
implementation, not using the stream in this way. It is also tested that, with
the current head, no similar issue appears.
> Directory handle leak in ListFile
> ---------------------------------
>
> Key: NIFI-7454
> URL: https://issues.apache.org/jira/browse/NIFI-7454
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.11.4
> Reporter: Paul Kelly
> Assignee: Simon Bence
> Priority: Major
> Attachments: flow.xml.gz
>
>
> ListFile does not seem to be closing directory handles if an exception is
> raised during the directory walk. For example, if ListFile is recursively
> polling a folder and the NiFi user does not have permission to list the
> contents of one of the subfolders, an "Access Denied" bulletin is raised and
> the number of DIR handles opened by the NiFi process as shown by lsof
> increases. Eventually, NiFi stops working due to "Too many open files".
> This has been tested with NiFi 1.11.4 on Ubuntu 18.04 with Java 11. I'm not
> sure in which version this behavior started.
> Attached is a sample flow.xml.gz that demonstrates this behavior. A clean
> copy of NiFi 1.11.4 was installed with no modifications to any configuration
> file. NiFi is running as the user "xubuntu". The permissions on the folders
> are as follows:
> drwxrwxr-x 3 xubuntu xubuntu 4096 May 15 08:55 /home/xubuntu/test
> drwxrwx--- 2 root root 4096 May 15 08:55 /home/xubuntu/test/subtest
> NiFi has permission to list the contents of "/home/xubuntu/test", but not
> "/home/xubuntu/test/subtest".
> The number of open DIR handles is being monitored with the command:
> lsof +E -P -p 347813 | tr -s " " | cut -d" " -f5 | sort | uniq -c
> where 347813 is the pid of the NiFi process.
> DIR handle counts start at 4 immediately after NiFi is launched and steadily
> grow to tens of thousands, until file handles are exhausted. Additionally,
> these handles do not seem to be released even if the ListFile processor is
> stopped.
> Please let me know if there is any additional information I can provide.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
