[
https://issues.apache.org/jira/browse/NIFI-10111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17552869#comment-17552869
]
Mark Payne commented on NIFI-10111:
-----------------------------------
For comparison purposes, running on {{main}} branch, with all nar files loaded,
etc.
Performance:
* With option to unpack to uber jar turned on: startup took about 1.5-2 mins
longer
* With option turned off, no difference in startup performance
Open file handle count:
* Empty flow, default behavior: OpenFileDescriptorCount : 3,017
* A few standard processors on canvas, default behavior:
OpenFileDescriptorCount : 2,945 (we expect some fluctuation here, adding a
couple standard processors like GenerateFlowFile, UpdateAttribute caused no
significant difference in open file handles)
* Empty flow, feature enabled to extract to Uber Jar: OpenFileDescriptorCount
: 298
* A few standard processors on canvas, feature enabled to extract to Uber Jar:
OpenFileDescriptorCount : 316
So we see that with the feature enabled, it reduced our open file handle by
default from over 3,000 to about 300.
> When unpacking NARs, provide ability to unpack into uber jar to avoid using
> too many open file handles
> ------------------------------------------------------------------------------------------------------
>
> Key: NIFI-10111
> URL: https://issues.apache.org/jira/browse/NIFI-10111
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework, Documentation & Website, NiFi
> Stateless
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When we unpack a NAR file, we expand all .jar files within it into the
> working directory. We then create a NAR ClassLoader for the NAR. The NAR
> ClassLoader extends URLClassLoader. As a result, we end up with a ClassLoader
> that holds an open file handle for every JAR file (and potentially other
> files) that gets unpacked.
> This can result in NiFi needing to use a lot of open file handles.
> When using Stateless NiFi, this is of particular concern, as it may need to
> run in more constrained environments, since it's intended to be embedded.
> We can significantly reduce the number of open file handles by unpacking each
> NAR into a single "Uber Jar File" instead of individual files. While this is
> particularly of concern for for Stateless NiFi, it may provide benefits to
> NiFi also. But for NiFi, it should not be "on by default". Users should have
> to opt into it via an entry in nifi.properties.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
