What is the flow doing in between the request/response portion? Please share more details about the configuration overall.
Thanks On Mon, Sep 25, 2017 at 9:16 AM, Lou Tian <[email protected]> wrote: > Hi Joe, > > java version: 1.8.0_121 > heap size: > # JVM memory settings > java.arg.2=-Xms512m > java.arg.3=-Xmx512m > nifi version: 1.3.0 > > Also, we put Nifi in the Docker. > > Kind Regrads, > Tian > > On Mon, Sep 25, 2017 at 2:39 PM, Joe Witt <[email protected]> wrote: >> >> Tian, >> >> Please provide information on the JRE being used (java -version) and >> the environment configuration. How large is your heap? This can be >> found in conf/bootstrap.conf. What version of nifi are you using? >> >> Thanks >> >> On Mon, Sep 25, 2017 at 8:29 AM, Lou Tian <[email protected]> wrote: >> > Hi, >> > >> > We are doing performance test for our NIFI flow with Gatling. But after >> > several run, the NIFI always has the OutOfMemory error. I did not find >> > similar questions in the mailing list, if you already answered similar >> > questions please let me know. >> > >> > Problem description: >> > We have the Nifi flow. The normal flow works fine. To evaluate whether >> > our >> > flow can handle the load, we decided to do the performance test with >> > Gatling. >> > >> > 1) We add the two processors HandleHttpRequest at the start of the flow >> > and >> > HandleHttpResponse at the end of the flow. So our nifi is like a >> > webservice >> > and Gatling will evaluate the response time. 2) Then we continuously >> > push >> > messages to HandleHttpRequest processor. >> > >> > Problem: >> > Nifi can only handle two runs. Then the third time, it failed and we >> > have to >> > restart the NIFI. I copied some error log here. >> > >> >> o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=**] >> >> HandleHttpRequest[id=**] failed to process session due to >> >> java.lang.OutOfMemoryError: Java heap space: {} >> >> o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=**] >> >> HandleHttpRequest[id=**] failed to process session due to >> >> java.lang.OutOfMemoryError: Java heap space: {} >> >> java.lang.OutOfMemoryError: Java heap space >> >> at java.util.HashMap.values(HashMap.java:958) >> >> at >> >> >> >> org.apache.nifi.controller.repository.StandardProcessSession.resetWriteClaims(StandardProcessSession.java:2720) >> >> at >> >> >> >> org.apache.nifi.controller.repository.StandardProcessSession.checkpoint(StandardProcessSession.java:213) >> >> at >> >> >> >> org.apache.nifi.controller.repository.StandardProcessSession.commit(StandardProcessSession.java:318) >> >> at >> >> >> >> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:28) >> >> at >> >> >> >> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1120) >> >> at >> >> >> >> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:147) >> >> at >> >> >> >> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47) >> >> at >> >> >> >> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >> >> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) >> >> at >> >> >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) >> >> at >> >> >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) >> >> at >> >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >> >> at >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >> >> at java.lang.Thread.run(Thread.java:748) >> > >> > >> > So our final questions: >> > 1. Do you think it is the HandleHttpRequest processors problem? Or there >> > is >> > something wrong in our configuration. Anything we can do to avoid such >> > problem? >> > 2. If it's the processor, will you plan to fix it in the coming version? >> > >> > Thank you so much for your reply. >> > >> > Kind Regards, >> > Tian >> > > > > > > -- > Kind Regards, > > Tian Lou >
