Hi Joe,
1. I will build a simple flow without our customised processor to test
again.
It is a good test idea. We saw the OOME is under the HandleHttpRequest,
we never thought about others.
2. About our customised processor, we use lots of these customised
processors.
Properties are dynamic. We fetch the properties by a rest call and
cached it.
Sorry, I cannot show you the code.
3. We had the unit test for the customised processors.
Is there a way to test the memory leak in unit test using some given
methods from nifi?
Thanks.
On Mon, Sep 25, 2017 at 3:28 PM, Joe Witt <[email protected]> wrote:
> Tian,
>
> Ok thanks. I'd try to removing your customized processor from the
> flow entirely and running your tests. This will give you a sense of
> base nifi and the stock processors. Once you're comfortable with that
> then add your processor in.
>
> I say this because if your custom processor is using up the heap we
> will see OOME in various places. That it shows up in the core
> framework code, for example, does not mean that is the cause.
>
> Does your custom processor hold anything in class level variables?
> Does it open a session and keep accumulating flowfiles? If you can
> talk more about what it is doing or show a link to the code we could
> quickly assess that.
>
> Thanks
>
> On Mon, Sep 25, 2017 at 9:24 AM, Lou Tian <[email protected]> wrote:
> > 1. The HandleHttpRequest Processor get the message.
> > 2. The message route to other processors based on the attribute.
> > 3. We have our customised processor to process the message.
> > 4. Then message would be redirected to the HandleHttpResponse.
> >
> > On Mon, Sep 25, 2017 at 3:20 PM, Joe Witt <[email protected]> wrote:
> >>
> >> What is the flow doing in between the request/response portion?
> >> Please share more details about the configuration overall.
> >>
> >> Thanks
> >>
> >> On Mon, Sep 25, 2017 at 9:16 AM, Lou Tian <[email protected]>
> wrote:
> >> > Hi Joe,
> >> >
> >> > java version: 1.8.0_121
> >> > heap size:
> >> > # JVM memory settings
> >> > java.arg.2=-Xms512m
> >> > java.arg.3=-Xmx512m
> >> > nifi version: 1.3.0
> >> >
> >> > Also, we put Nifi in the Docker.
> >> >
> >> > Kind Regrads,
> >> > Tian
> >> >
> >> > On Mon, Sep 25, 2017 at 2:39 PM, Joe Witt <[email protected]> wrote:
> >> >>
> >> >> Tian,
> >> >>
> >> >> Please provide information on the JRE being used (java -version) and
> >> >> the environment configuration. How large is your heap? This can be
> >> >> found in conf/bootstrap.conf. What version of nifi are you using?
> >> >>
> >> >> Thanks
> >> >>
> >> >> On Mon, Sep 25, 2017 at 8:29 AM, Lou Tian <[email protected]>
> >> >> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > We are doing performance test for our NIFI flow with Gatling. But
> >> >> > after
> >> >> > several run, the NIFI always has the OutOfMemory error. I did not
> >> >> > find
> >> >> > similar questions in the mailing list, if you already answered
> >> >> > similar
> >> >> > questions please let me know.
> >> >> >
> >> >> > Problem description:
> >> >> > We have the Nifi flow. The normal flow works fine. To evaluate
> >> >> > whether
> >> >> > our
> >> >> > flow can handle the load, we decided to do the performance test
> with
> >> >> > Gatling.
> >> >> >
> >> >> > 1) We add the two processors HandleHttpRequest at the start of the
> >> >> > flow
> >> >> > and
> >> >> > HandleHttpResponse at the end of the flow. So our nifi is like a
> >> >> > webservice
> >> >> > and Gatling will evaluate the response time. 2) Then we
> >> >> > continuously
> >> >> > push
> >> >> > messages to HandleHttpRequest processor.
> >> >> >
> >> >> > Problem:
> >> >> > Nifi can only handle two runs. Then the third time, it failed and
> we
> >> >> > have to
> >> >> > restart the NIFI. I copied some error log here.
> >> >> >
> >> >> >> o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=**]
> >> >> >> HandleHttpRequest[id=**] failed to process session due to
> >> >> >> java.lang.OutOfMemoryError: Java heap space: {}
> >> >> >> o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=**]
> >> >> >> HandleHttpRequest[id=**] failed to process session due to
> >> >> >> java.lang.OutOfMemoryError: Java heap space: {}
> >> >> >> java.lang.OutOfMemoryError: Java heap space
> >> >> >> at java.util.HashMap.values(HashMap.java:958)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >> >> org.apache.nifi.controller.repository.StandardProcessSession.
> resetWriteClaims(StandardProcessSession.java:2720)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >> >> org.apache.nifi.controller.repository.StandardProcessSession.
> checkpoint(StandardProcessSession.java:213)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >> >> org.apache.nifi.controller.repository.
> StandardProcessSession.commit(StandardProcessSession.java:318)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >> >> org.apache.nifi.processor.AbstractProcessor.onTrigger(
> AbstractProcessor.java:28)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >> >> org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> StandardProcessorNode.java:1120)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >> >> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
> call(ContinuallyRunProcessorTask.java:147)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >> >> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.
> call(ContinuallyRunProcessorTask.java:47)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >> >> org.apache.nifi.controller.scheduling.
> TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:132)
> >> >> >> at
> >> >> >>
> >> >> >> java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:511)
> >> >> >> at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:308)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >> >> java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >> >> java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> >> >> >> at
> >> >> >>
> >> >> >>
> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> >> >> >> at java.lang.Thread.run(Thread.java:748)
> >> >> >
> >> >> >
> >> >> > So our final questions:
> >> >> > 1. Do you think it is the HandleHttpRequest processors problem? Or
> >> >> > there
> >> >> > is
> >> >> > something wrong in our configuration. Anything we can do to avoid
> >> >> > such
> >> >> > problem?
> >> >> > 2. If it's the processor, will you plan to fix it in the coming
> >> >> > version?
> >> >> >
> >> >> > Thank you so much for your reply.
> >> >> >
> >> >> > Kind Regards,
> >> >> > Tian
> >> >> >
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Kind Regards,
> >> >
> >> > Tian Lou
> >> >
> >
> >
> >
> >
> > --
> > Kind Regards,
> >
> > Tian Lou
> >
>
--
Kind Regards,
Tian Lou
