Hi Joe, Thanks for your reply. I will try to do those tests. And update you with the results.
On Mon, Sep 25, 2017 at 3:56 PM, Joe Witt <[email protected]> wrote: > Tian > > The most common sources of memory leaks in custom processors > 1) Loading large objects (contents of the flowfile, for example) into > memory through byte[] or doing so using libraries that do this and not > realizing it. Doing this in parallel makes the problem even more > obvious. > 2) Caching objects in memory and not providing bounds on that or not > sizing the JVM Heap appropriate to your flow. > 3) Pull in lots of flowfiles to a single session or creating many in a > single session. > > Try moving to a 1GB heap and see if the problem still happens. Is it > as fast? Does it not happen. Try 2GB if needed. After that suspect > a leak. > > We dont have a benchmarking unit test sort of mechanism. > > Thanks > > On Mon, Sep 25, 2017 at 9:45 AM, Lou Tian <[email protected]> wrote: > > Hi Joe, > > > > 1. I will build a simple flow without our customised processor to test > > again. > > It is a good test idea. We saw the OOME is under the > HandleHttpRequest, > > we never thought about others. > > > > 2. About our customised processor, we use lots of these customised > > processors. > > Properties are dynamic. We fetch the properties by a rest call and > > cached it. > > Sorry, I cannot show you the code. > > > > 3. We had the unit test for the customised processors. > > Is there a way to test the memory leak in unit test using some given > > methods from nifi? > > > > Thanks. > > > > On Mon, Sep 25, 2017 at 3:28 PM, Joe Witt <[email protected]> wrote: > >> > >> Tian, > >> > >> Ok thanks. I'd try to removing your customized processor from the > >> flow entirely and running your tests. This will give you a sense of > >> base nifi and the stock processors. Once you're comfortable with that > >> then add your processor in. > >> > >> I say this because if your custom processor is using up the heap we > >> will see OOME in various places. That it shows up in the core > >> framework code, for example, does not mean that is the cause. > >> > >> Does your custom processor hold anything in class level variables? > >> Does it open a session and keep accumulating flowfiles? If you can > >> talk more about what it is doing or show a link to the code we could > >> quickly assess that. > >> > >> Thanks > >> > >> On Mon, Sep 25, 2017 at 9:24 AM, Lou Tian <[email protected]> > wrote: > >> > 1. The HandleHttpRequest Processor get the message. > >> > 2. The message route to other processors based on the attribute. > >> > 3. We have our customised processor to process the message. > >> > 4. Then message would be redirected to the HandleHttpResponse. > >> > > >> > On Mon, Sep 25, 2017 at 3:20 PM, Joe Witt <[email protected]> wrote: > >> >> > >> >> What is the flow doing in between the request/response portion? > >> >> Please share more details about the configuration overall. > >> >> > >> >> Thanks > >> >> > >> >> On Mon, Sep 25, 2017 at 9:16 AM, Lou Tian <[email protected]> > >> >> wrote: > >> >> > Hi Joe, > >> >> > > >> >> > java version: 1.8.0_121 > >> >> > heap size: > >> >> > # JVM memory settings > >> >> > java.arg.2=-Xms512m > >> >> > java.arg.3=-Xmx512m > >> >> > nifi version: 1.3.0 > >> >> > > >> >> > Also, we put Nifi in the Docker. > >> >> > > >> >> > Kind Regrads, > >> >> > Tian > >> >> > > >> >> > On Mon, Sep 25, 2017 at 2:39 PM, Joe Witt <[email protected]> > wrote: > >> >> >> > >> >> >> Tian, > >> >> >> > >> >> >> Please provide information on the JRE being used (java -version) > and > >> >> >> the environment configuration. How large is your heap? This can > be > >> >> >> found in conf/bootstrap.conf. What version of nifi are you using? > >> >> >> > >> >> >> Thanks > >> >> >> > >> >> >> On Mon, Sep 25, 2017 at 8:29 AM, Lou Tian <[email protected] > > > >> >> >> wrote: > >> >> >> > Hi, > >> >> >> > > >> >> >> > We are doing performance test for our NIFI flow with Gatling. > But > >> >> >> > after > >> >> >> > several run, the NIFI always has the OutOfMemory error. I did > not > >> >> >> > find > >> >> >> > similar questions in the mailing list, if you already answered > >> >> >> > similar > >> >> >> > questions please let me know. > >> >> >> > > >> >> >> > Problem description: > >> >> >> > We have the Nifi flow. The normal flow works fine. To evaluate > >> >> >> > whether > >> >> >> > our > >> >> >> > flow can handle the load, we decided to do the performance test > >> >> >> > with > >> >> >> > Gatling. > >> >> >> > > >> >> >> > 1) We add the two processors HandleHttpRequest at the start of > the > >> >> >> > flow > >> >> >> > and > >> >> >> > HandleHttpResponse at the end of the flow. So our nifi is like a > >> >> >> > webservice > >> >> >> > and Gatling will evaluate the response time. 2) Then we > >> >> >> > continuously > >> >> >> > push > >> >> >> > messages to HandleHttpRequest processor. > >> >> >> > > >> >> >> > Problem: > >> >> >> > Nifi can only handle two runs. Then the third time, it failed > and > >> >> >> > we > >> >> >> > have to > >> >> >> > restart the NIFI. I copied some error log here. > >> >> >> > > >> >> >> >> o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=**] > >> >> >> >> HandleHttpRequest[id=**] failed to process session due to > >> >> >> >> java.lang.OutOfMemoryError: Java heap space: {} > >> >> >> >> o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=**] > >> >> >> >> HandleHttpRequest[id=**] failed to process session due to > >> >> >> >> java.lang.OutOfMemoryError: Java heap space: {} > >> >> >> >> java.lang.OutOfMemoryError: Java heap space > >> >> >> >> at java.util.HashMap.values(HashMap.java:958) > >> >> >> >> at > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> org.apache.nifi.controller.repository.StandardProcessSession. > resetWriteClaims(StandardProcessSession.java:2720) > >> >> >> >> at > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> org.apache.nifi.controller.repository.StandardProcessSession. > checkpoint(StandardProcessSession.java:213) > >> >> >> >> at > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> org.apache.nifi.controller.repository. > StandardProcessSession.commit(StandardProcessSession.java:318) > >> >> >> >> at > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> org.apache.nifi.processor.AbstractProcessor.onTrigger( > AbstractProcessor.java:28) > >> >> >> >> at > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> org.apache.nifi.controller.StandardProcessorNode.onTrigger( > StandardProcessorNode.java:1120) > >> >> >> >> at > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask. > call(ContinuallyRunProcessorTask.java:147) > >> >> >> >> at > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask. > call(ContinuallyRunProcessorTask.java:47) > >> >> >> >> at > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> org.apache.nifi.controller.scheduling. > TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) > >> >> >> >> at > >> >> >> >> > >> >> >> >> > >> >> >> >> java.util.concurrent.Executors$RunnableAdapter. > call(Executors.java:511) > >> >> >> >> at > >> >> >> >> java.util.concurrent.FutureTask.runAndReset( > FutureTask.java:308) > >> >> >> >> at > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) > >> >> >> >> at > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ > ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) > >> >> >> >> at > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1142) > >> >> >> >> at > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:617) > >> >> >> >> at java.lang.Thread.run(Thread.java:748) > >> >> >> > > >> >> >> > > >> >> >> > So our final questions: > >> >> >> > 1. Do you think it is the HandleHttpRequest processors problem? > Or > >> >> >> > there > >> >> >> > is > >> >> >> > something wrong in our configuration. Anything we can do to > avoid > >> >> >> > such > >> >> >> > problem? > >> >> >> > 2. If it's the processor, will you plan to fix it in the coming > >> >> >> > version? > >> >> >> > > >> >> >> > Thank you so much for your reply. > >> >> >> > > >> >> >> > Kind Regards, > >> >> >> > Tian > >> >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > -- > >> >> > Kind Regards, > >> >> > > >> >> > Tian Lou > >> >> > > >> > > >> > > >> > > >> > > >> > -- > >> > Kind Regards, > >> > > >> > Tian Lou > >> > > > > > > > > > > > -- > > Kind Regards, > > > > Tian Lou > > > -- Kind Regards, Tian Lou
