What version of NiFi is Kylo based on? There was a memory leak related to HDFS processors that was fixed back in 1.7.0:
https://issues.apache.org/jira/browse/NIFI-5136 On Mon, Nov 11, 2019 at 11:14 PM Jeff <[email protected]> wrote: > > If you remove the @RequiresInstanceClassloading, the UserGroupInformation > class from Hadoop (hadoop-common, if I remember correctly) will be shared > across all instances that come from a particular NAR (such as PutHDFS, > ListHDFS, FetchHDFS, etc, from nifi-hadoop-nar-x.y.z.nar). If you are using > Kerberos in those processors and configured different principals across the > various processors, you could run into issues when the processors attempt to > acquire new TGTs, most likely the first time a relogin is attempted. UGI has > some static state and @RequiresInstanceClassloading makes sure each instance > of a processor with that annotation has its own classloader to keep that kind > of state from being shared across instances. > > On Mon, Nov 11, 2019 at 9:41 PM abellnotring <[email protected]> wrote: >> >> Hi,Peter & All >> I’m using kylo to manage the nifi flow(called feed in kylo),and there >> are 4200 instances(600+ instances extended from AbstractHadoopProcessor) in >> my nifi canvas,The NIFI Non-Heap Memory has increased more than 6GB after >> some days running ,which is extremely abnormal . I have analyzed the class >> loaded into Compressed Class Space ,and found most of the CCS was used by >> classes related by AbstractHadoopProcessor. >> So I think removing RequiresInstanceClassLoading from >> AbstractHadoopProcessor processor may be a Solution for reducing the CCS >> used. >> Do you have any ideas for this ? >> >> >> >> Thanks >> >> >> By Hai Luo >> >> On 11/12/2019 02:17，Shawn Weeks<[email protected]> wrote： >> >> I’m assuming your talking about the snappy problem. If you use compress >> content prior to puthdfs you can compress with Snappy as it uses the Java >> Native Snappy Lib. The HDFS processors are limited to the actual Hadoop >> Libraries so they’d have to change from Native to get around this. I’m >> pretty sure we need instance loading to handle the other issues mentioned. >> >> >> >> Thanks >> >> Shawn >> >> >> >> From: Joe Witt <[email protected]> >> Reply-To: "[email protected]" <[email protected]> >> Date: Monday, November 11, 2019 at 8:56 AM >> To: "[email protected]" <[email protected]> >> Subject: Re: Influence about removing RequiresInstanceClassLoading from >> AbstractHadoopProcessor processor >> >> >> >> Peter >> >> >> >> The most common challenge is if two isolated instances both want to use a >> native lib. No two native libs with the same name can be in the same jvm. >> We need to solve that for sure. >> >> >> >> Thanks >> >> >> >> On Mon, Nov 11, 2019 at 9:53 AM Peter Turcsanyi <[email protected]> wrote: >> >> Hi Hai Luo, >> >> >> >> @RequiresInstanceClassLoading makes possible to configure separate / >> isolated "Additional Classpath Resources" settings on your HDFS processors >> (eg. S3 storage driver on one of your PutHDFS and Azure Blob on the other). >> >> >> >> Is there any specific reason / use case why you are considering to remove it? >> >> >> >> Regards, >> >> Peter Turcsanyi >> >> >> >> On Mon, Nov 11, 2019 at 3:30 PM abellnotring <[email protected]> wrote: >> >> Hi,all >> >> I’m considering removing the RequiresInstanceClassLoading annotation >> from class AbstractHadoopProcessor, >> >> Does anybody know the potential Influence? >> >> >> >> Thanks >> >> By Hai Luo
