This problem has been captured by the issue: https://issues.apache.org/jira/browse/NIFI-5136
Thanks for the help and the quick work finding the problems. -Dann On Fri, Apr 27, 2018 at 3:51 PM Bryan Bende <[email protected]> wrote: > Hey Dann, > > I've spent most of the afternoon debugging this and you are correct, > there are actually a couple of issues that are causing the > InstanceClassLoader to stick around. One of them is what you mentioned > with the scheduled states (now called lifecycle states), nice find!!! > > I have a few them of fixed here [1], but there is one more issue I'm > still looking into that I will have to come back to on Monday. > > The one you might find most interesting is that the Hadoop FileSystem > class starts a thread that it never stops and provides no access to: > > > https://github.com/apache/hadoop/blob/release-3.0.0-RC1/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileSystem.java#L3655-L3665 > > I was able to use reflection to make it accessible and access the > thread and call interrupt and get it to stop. > > I'll report back once I know more about the last issue I was looking into. > > Thanks, > > Bryan > > [1] https://github.com/bbende/nifi/commits/stop-leaking-processors > > > On Fri, Apr 27, 2018 at 2:30 PM, Dann <[email protected]> wrote: > > This might be related to NIFI-5110. Since the StandardProcessScheduler > is > > keeping the reference through a StandardProcessorNode in the > scheduleStates > > map, the PutHDFS cannot be garbage collected and thus the > > InstanceClassloader is not able to be cleaned up. > > > > Note: It looks like between 1.5.0 and 1.6.0, the StandardProcessScheduler > > change a little and scheduleStates was changed to lifecycleStates. > > > > -Dann > > > > On Fri, Apr 27, 2018 at 8:48 AM Dann <[email protected]> wrote: > >> > >> The behavior is the same with both 1.2.0 and 1.5.0. > >> > >> If the instance class loading is needed, there is something keeping that > >> instance classloader from being garbage collected. Removing instance > class > >> loading (if possible) would fix the problem. If instance class loading > is > >> needed, then finding the reason that classloader can't be cleaned up > would > >> be the fix. > >> > >> > >> On Fri, Apr 27, 2018 at 8:38 AM Bryan Bende <[email protected]> wrote: > >>> > >>> Is the behavior the same in 1.5.0, or only in 1.2.0? > >>> > >>> We can't undo the instance class loading for Hadoop processors unless > >>> we undo the static usage of UserGroupInformation. > >>> > >>> I think it is possible to go back to the non-static usage since we the > >>> kerberos issues were actually resolved by the useSubjectCredsOnly > >>> property, but it would just take a bit of work and testing and not as > >>> easy as just removing the instance class loading annotation. > >>> > >>> > >>> On Fri, Apr 27, 2018 at 10:25 AM, Joe Witt <[email protected]> wrote: > >>> > Dann > >>> > > >>> > The hadoop processors as of a version around 1.2.0 switched to > >>> > 'instance class loading' meaning that while we always have components > >>> > from various nars in their own classloader in the case of Hadoop > >>> > processors we went a step further and we create a new classloader for > >>> > every single instance of the component. This was done to overcome > >>> > what appears to be problematic usage of static values in the hadoop > >>> > client code. However, it is possible we've changed our usage enough > >>> > that these would not create problems for us any longer and we could > >>> > consider going back to typical class loading. > >>> > > >>> > Would be good to hear others thoughts. > >>> > > >>> > Thanks > >>> > > >>> > On Fri, Apr 27, 2018 at 10:20 AM, Dann <[email protected]> > wrote: > >>> >> NiFi Versions: 1.2.0 and 1.5.0 > >>> >> Java Version: 1.8.0_162 > >>> >> > >>> >> It appears to me that when I add a PutHDFS processor to the canvas, > >>> >> the > >>> >> hadoop classes are loaded along with dependencies (normal/desired > >>> >> behavior). > >>> >> Then if I delete the PutHDFS processor, the garbage collection is > not > >>> >> able > >>> >> to unload any of the classes that were loaded. This is the same > >>> >> behavior > >>> >> with every PutHDFS processor that is added and then deleted. The > >>> >> results in > >>> >> the slow degradation of NiFi over time with the way we use NiFi. > >>> >> > >>> >> Our usage of NiFi does not expose the NiFi interface for those that > >>> >> are > >>> >> designing data flows. We have a web interface that allows the user > to > >>> >> define the data they need and our application will build data flows > >>> >> for the > >>> >> user. As we upgrade our application we make changes that require us > >>> >> to > >>> >> remove the old data flow and rebuild it with the new changes. Over > >>> >> time, we > >>> >> add and remove a lot of processors using the NiFi API and some of > >>> >> those > >>> >> processors are HDFS related. > >>> >> > >>> >> To me, there seems to be a problem with the Hadoop dependency > loading > >>> >> and > >>> >> processor implementation that doesn't allow the JVM to unload those > >>> >> classes > >>> >> when they are no longer needed. > >>> >> > >>> >> Thanks, > >>> >> > >>> >> Dann >
