[
https://issues.apache.org/jira/browse/NIFI-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467856#comment-16467856
]
ASF GitHub Bot commented on NIFI-5136:
--------------------------------------
Github user markap14 commented on the issue:
https://github.com/apache/nifi/pull/2668
@bbende thanks for addressing these issues! I can see that we've tackled a
handful of spots that could be leaking the references, and I've seen the heap
dumps showing that they are no longer concerns. The changes to the Abstract
Hadoop Processors are less desirable than I'd prefer, but I agree that the API
doesn't really expose what it would need to expose in order to do this more
effectively, so it's a reasonable workaround. The LogRepositoryFactory change
in FlowController's create* methods is good - i hadn't realized that we were
doing that there, within the nar loader.
With this particular scenario, I don't think it's really replicable in a
unit test. All that could be done in unit tests would be to verify that very
trivial functionality like "removeLogger() calls remove() on the underlying
map" which make for bad unit testing IMO. You could attempt to create some very
involved integration tests, but they would involve mocking out so much that it
would be hard to know what is really being verified. On top of that, you still
would not be able to verify that the appropriate objects are reclaimable via
garbage collection.
This all looks good to me, though. Can verify behavior and code changes
look good. +1 will merge to master.
> Leaked component references preventing GC of components and class loaders
> -------------------------------------------------------------------------
>
> Key: NIFI-5136
> URL: https://issues.apache.org/jira/browse/NIFI-5136
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.2.0, 1.3.0, 1.4.0, 1.5.0, 1.6.0
> Reporter: Bryan Bende
> Assignee: Bryan Bende
> Priority: Major
>
> A user on the mailing list reported that after some time of creating/deleting
> HDFS processors, it appeared that the classes/instances were still around and
> eventually the NiFi instance would get out of memory and need to be restarted.
> After investigation there are multiple issues preventing garbage collection
> of deleted components. One issue is specific to the HDFS processors, the
> other issues are for all components...
> 1) The LogRepository still has a reference to a ComponentLogger which has a
> reference to the component
> 2) The processor scheduler has a map of scheduled states which has references
> to processors that have been deleted
> 3) The Hadoop processors start a thread that is never stopped when the
> processor is stopped/deleted, this means the class loader can't be cleaned up
> b/c the Runnable came from the InstanceClassLoader of the deleted processor
> 4) Importing a flow from registry will instantiate an instance of each
> component to ensure the incoming types are valid, but the InstanceClassLoader
> and ComponentLogger are not cleaned up for these temp instances
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
