[
https://issues.apache.org/jira/browse/NIFI-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459876#comment-16459876
]
ASF GitHub Bot commented on NIFI-5136:
--------------------------------------
Github user ottobackwards commented on a diff in the pull request:
https://github.com/apache/nifi/pull/2668#discussion_r185276958
--- Diff:
nifi-nar-bundles/nifi-extension-utils/nifi-hadoop-utils/src/main/java/org/apache/nifi/processors/hadoop/AbstractHadoopProcessor.java
---
@@ -257,9 +260,63 @@ public final void abstractOnScheduled(ProcessContext
context) throws IOException
@OnStopped
public final void abstractOnStopped() {
+ final HdfsResources resources = hdfsResources.get();
+ if (resources != null) {
+ // Attempt to close the FileSystem
+ final FileSystem fileSystem = resources.getFileSystem();
+ try {
+ interruptStatisticsThread(fileSystem);
+ } catch (Exception e) {
+ getLogger().warn("Error stopping FileSystem statistics
thread: " + e.getMessage(), e);
+ } finally {
+ if (fileSystem != null) {
+ try {
+ fileSystem.close();
+ } catch (IOException e) {
+ getLogger().warn("Error close FileSystem: " +
e.getMessage(), e);
+ }
+ }
+ }
+
+ // Clean-up the static reference to the Configuration instance
+ UserGroupInformation.setConfiguration(new Configuration());
+
+ // Clean-up the reference to the InstanceClassLoader that was
put into Configuration
+ final Configuration configuration =
resources.getConfiguration();
+ configuration.setClassLoader(null);
+
+ // Need to remove the Provider instance from the JVM's
Providers class so that InstanceClassLoader can be GC'd eventually
+ final SaslPlainServer.SecurityProvider saslProvider = new
SaslPlainServer.SecurityProvider();
+ Security.removeProvider(saslProvider.getName());
+ }
+
+ // Clear out the reference to the resources
hdfsResources.set(new HdfsResources(null, null, null));
}
--- End diff --
You *could* use commons reflect here, but I'm not sure it matters
> Leaked component references preventing GC of components and class loaders
> -------------------------------------------------------------------------
>
> Key: NIFI-5136
> URL: https://issues.apache.org/jira/browse/NIFI-5136
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0
> Reporter: Bryan Bende
> Assignee: Bryan Bende
> Priority: Major
>
> A user on the mailing list reported that after some time of creating/deleting
> HDFS processors, it appeared that the classes/instances were still around and
> eventually the NiFi instance would get out of memory and need to be restarted.
> After investigation there are multiple issues preventing garbage collection
> of deleted components. One issue is specific to the HDFS processors, the
> other issues are for all components...
> 1) The LogRepository still has a reference to a ComponentLogger which has a
> reference to the component
> 2) The processor scheduler has a map of scheduled states which has references
> to processors that have been deleted
> 3) The Hadoop processors start a thread that is never stopped when the
> processor is stopped/deleted, this means the class loader can't be cleaned up
> b/c the Runnable came from the InstanceClassLoader of the deleted processor
> 4) Importing a flow from registry will instantiate an instance of each
> component to ensure the incoming types are valid, but the InstanceClassLoader
> and ComponentLogger are not cleaned up for these temp instances
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
