Hi Joe,
I tested with a simple flow file.
Only 4 processors: HandleHttpRequest, RouteOnContent, HandleHttpResponse
and DebugFlow.
I run the test 3 times (10 m/time and at most 50 users).
It works fine for the first 2 run. And on the third run, got the error.
I copied part of the log file. Please check if it is helpful to identify
the error.
2017-09-25 18:21:45,673 INFO [Provenance Maintenance Thread-2]
o.a.n.p.PersistentProvenanceRepository Created new Provenance Event
Writers for events starting with ID 131158
2017-09-25 18:24:00,921 ERROR [FileSystemRepository Workers Thread-3]
o.a.n.c.repository.FileSystemRepository Failed to handle destructable
claims due to java.lang.OutOfMemoryError: Java heap space
2017-09-25 18:24:00,921 ERROR [Flow Service Tasks Thread-1]
org.apache.nifi.NiFi An Unknown Error Occurred in Thread Thread[Flow
Service Tasks Thread-1,5,main]: java.lang.OutOfMemoryError: Java heap
space
2017-09-25 18:24:00,922 WARN [qtp574205748-107]
o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death:
org.eclipse.jetty.util.thread.QueuedThreadPool$2@1e3a5886 in
qtp574205748{STARTED,8<=13<=200,i=4,q=0}
2017-09-25 18:24:00,923 INFO [Provenance Repository Rollover Thread-1]
o.a.n.p.lucene.SimpleIndexManager Index Writer for
./provenance_repository/index-1506354574000 has been returned to Index
Manager and is no longer in use. Closing Index Writer
2017-09-25 18:24:00,925 ERROR [qtp574205748-107] org.apache.nifi.NiFi
An Unknown Error Occurred in Thread Thread[qtp574205748-107,5,main]:
java.lang.OutOfMemoryError: Java heap space
2017-09-25 18:24:00,929 INFO [pool-10-thread-1]
o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of
FlowFile Repository
2017-09-25 18:24:00,929 ERROR [Flow Service Tasks Thread-1]
org.apache.nifi.NiFi
java.lang.OutOfMemoryError: Java heap space
2017-09-25 18:24:00,928 ERROR [Listen to Bootstrap]
org.apache.nifi.BootstrapListener Failed to process request from
Bootstrap due to java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
2017-09-25 18:24:00,929 WARN [NiFi Web Server-215]
org.eclipse.jetty.server.HttpChannel
/nifi-api/flow/controller/bulletins
java.lang.OutOfMemoryError: Java heap space
2017-09-25 18:24:00,930 ERROR [pool-30-thread-1] org.apache.nifi.NiFi
An Unknown Error Occurred in Thread Thread[pool-30-thread-1,5,main]:
java.lang.OutOfMemoryError: Java heap space
2017-09-25 18:24:00,929 ERROR [Event-Driven Process Thread-3]
org.apache.nifi.engine.FlowEngine A flow controller task execution
stopped abnormally
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java heap space
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
2017-09-25 18:24:00,931 ERROR [Scheduler-1985086499]
org.apache.nifi.NiFi An Unknown Error Occurred in Thread
Thread[Scheduler-1985086499,5,main]: java.lang.OutOfMemoryError: Java
heap space
2017-09-25 18:24:00,930 ERROR [Cleanup Archive for default]
org.apache.nifi.engine.FlowEngine A flow controller task execution
stopped abnormally
java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError:
Java heap space
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at org.apache.nifi.engine.FlowEngine.afterExecute(FlowEngine.java:100)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1150)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
Kind Regards,
Tian
On Mon, Sep 25, 2017 at 4:02 PM, Lou Tian <[email protected]> wrote:
> Hi Joe, Thanks for your reply.
> I will try to do those tests. And update you with the results.
>
> On Mon, Sep 25, 2017 at 3:56 PM, Joe Witt <[email protected]> wrote:
>
>> Tian
>>
>> The most common sources of memory leaks in custom processors
>> 1) Loading large objects (contents of the flowfile, for example) into
>> memory through byte[] or doing so using libraries that do this and not
>> realizing it. Doing this in parallel makes the problem even more
>> obvious.
>> 2) Caching objects in memory and not providing bounds on that or not
>> sizing the JVM Heap appropriate to your flow.
>> 3) Pull in lots of flowfiles to a single session or creating many in a
>> single session.
>>
>> Try moving to a 1GB heap and see if the problem still happens. Is it
>> as fast? Does it not happen. Try 2GB if needed. After that suspect
>> a leak.
>>
>> We dont have a benchmarking unit test sort of mechanism.
>>
>> Thanks
>>
>> On Mon, Sep 25, 2017 at 9:45 AM, Lou Tian <[email protected]> wrote:
>> > Hi Joe,
>> >
>> > 1. I will build a simple flow without our customised processor to test
>> > again.
>> > It is a good test idea. We saw the OOME is under the
>> HandleHttpRequest,
>> > we never thought about others.
>> >
>> > 2. About our customised processor, we use lots of these customised
>> > processors.
>> > Properties are dynamic. We fetch the properties by a rest call and
>> > cached it.
>> > Sorry, I cannot show you the code.
>> >
>> > 3. We had the unit test for the customised processors.
>> > Is there a way to test the memory leak in unit test using some given
>> > methods from nifi?
>> >
>> > Thanks.
>> >
>> > On Mon, Sep 25, 2017 at 3:28 PM, Joe Witt <[email protected]> wrote:
>> >>
>> >> Tian,
>> >>
>> >> Ok thanks. I'd try to removing your customized processor from the
>> >> flow entirely and running your tests. This will give you a sense of
>> >> base nifi and the stock processors. Once you're comfortable with that
>> >> then add your processor in.
>> >>
>> >> I say this because if your custom processor is using up the heap we
>> >> will see OOME in various places. That it shows up in the core
>> >> framework code, for example, does not mean that is the cause.
>> >>
>> >> Does your custom processor hold anything in class level variables?
>> >> Does it open a session and keep accumulating flowfiles? If you can
>> >> talk more about what it is doing or show a link to the code we could
>> >> quickly assess that.
>> >>
>> >> Thanks
>> >>
>> >> On Mon, Sep 25, 2017 at 9:24 AM, Lou Tian <[email protected]>
>> wrote:
>> >> > 1. The HandleHttpRequest Processor get the message.
>> >> > 2. The message route to other processors based on the attribute.
>> >> > 3. We have our customised processor to process the message.
>> >> > 4. Then message would be redirected to the HandleHttpResponse.
>> >> >
>> >> > On Mon, Sep 25, 2017 at 3:20 PM, Joe Witt <[email protected]>
>> wrote:
>> >> >>
>> >> >> What is the flow doing in between the request/response portion?
>> >> >> Please share more details about the configuration overall.
>> >> >>
>> >> >> Thanks
>> >> >>
>> >> >> On Mon, Sep 25, 2017 at 9:16 AM, Lou Tian <[email protected]>
>> >> >> wrote:
>> >> >> > Hi Joe,
>> >> >> >
>> >> >> > java version: 1.8.0_121
>> >> >> > heap size:
>> >> >> > # JVM memory settings
>> >> >> > java.arg.2=-Xms512m
>> >> >> > java.arg.3=-Xmx512m
>> >> >> > nifi version: 1.3.0
>> >> >> >
>> >> >> > Also, we put Nifi in the Docker.
>> >> >> >
>> >> >> > Kind Regrads,
>> >> >> > Tian
>> >> >> >
>> >> >> > On Mon, Sep 25, 2017 at 2:39 PM, Joe Witt <[email protected]>
>> wrote:
>> >> >> >>
>> >> >> >> Tian,
>> >> >> >>
>> >> >> >> Please provide information on the JRE being used (java -version)
>> and
>> >> >> >> the environment configuration. How large is your heap? This
>> can be
>> >> >> >> found in conf/bootstrap.conf. What version of nifi are you
>> using?
>> >> >> >>
>> >> >> >> Thanks
>> >> >> >>
>> >> >> >> On Mon, Sep 25, 2017 at 8:29 AM, Lou Tian <
>> [email protected]>
>> >> >> >> wrote:
>> >> >> >> > Hi,
>> >> >> >> >
>> >> >> >> > We are doing performance test for our NIFI flow with Gatling.
>> But
>> >> >> >> > after
>> >> >> >> > several run, the NIFI always has the OutOfMemory error. I did
>> not
>> >> >> >> > find
>> >> >> >> > similar questions in the mailing list, if you already answered
>> >> >> >> > similar
>> >> >> >> > questions please let me know.
>> >> >> >> >
>> >> >> >> > Problem description:
>> >> >> >> > We have the Nifi flow. The normal flow works fine. To evaluate
>> >> >> >> > whether
>> >> >> >> > our
>> >> >> >> > flow can handle the load, we decided to do the performance test
>> >> >> >> > with
>> >> >> >> > Gatling.
>> >> >> >> >
>> >> >> >> > 1) We add the two processors HandleHttpRequest at the start of
>> the
>> >> >> >> > flow
>> >> >> >> > and
>> >> >> >> > HandleHttpResponse at the end of the flow. So our nifi is like
>> a
>> >> >> >> > webservice
>> >> >> >> > and Gatling will evaluate the response time. 2) Then we
>> >> >> >> > continuously
>> >> >> >> > push
>> >> >> >> > messages to HandleHttpRequest processor.
>> >> >> >> >
>> >> >> >> > Problem:
>> >> >> >> > Nifi can only handle two runs. Then the third time, it failed
>> and
>> >> >> >> > we
>> >> >> >> > have to
>> >> >> >> > restart the NIFI. I copied some error log here.
>> >> >> >> >
>> >> >> >> >> o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=**]
>> >> >> >> >> HandleHttpRequest[id=**] failed to process session due to
>> >> >> >> >> java.lang.OutOfMemoryError: Java heap space: {}
>> >> >> >> >> o.a.n.p.standard.HandleHttpRequest HandleHttpRequest[id=**]
>> >> >> >> >> HandleHttpRequest[id=**] failed to process session due to
>> >> >> >> >> java.lang.OutOfMemoryError: Java heap space: {}
>> >> >> >> >> java.lang.OutOfMemoryError: Java heap space
>> >> >> >> >> at java.util.HashMap.values(HashMap.java:958)
>> >> >> >> >> at
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> org.apache.nifi.controller.repository.StandardProcessSession
>> .resetWriteClaims(StandardProcessSession.java:2720)
>> >> >> >> >> at
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> org.apache.nifi.controller.repository.StandardProcessSession
>> .checkpoint(StandardProcessSession.java:213)
>> >> >> >> >> at
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> org.apache.nifi.controller.repository.StandardProcessSession
>> .commit(StandardProcessSession.java:318)
>> >> >> >> >> at
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> org.apache.nifi.processor.AbstractProcessor.onTrigger(Abstra
>> ctProcessor.java:28)
>> >> >> >> >> at
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> org.apache.nifi.controller.StandardProcessorNode.onTrigger(S
>> tandardProcessorNode.java:1120)
>> >> >> >> >> at
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>> .call(ContinuallyRunProcessorTask.java:147)
>> >> >> >> >> at
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask
>> .call(ContinuallyRunProcessorTask.java:47)
>> >> >> >> >> at
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingA
>> gent$1.run(TimerDrivenSchedulingAgent.java:132)
>> >> >> >> >> at
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> java.util.concurrent.Executors$RunnableAdapter.call(
>> Executors.java:511)
>> >> >> >> >> at
>> >> >> >> >> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:
>> 308)
>> >> >> >> >> at
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>> >> >> >> >> at
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFu
>> tureTask.run(ScheduledThreadPoolExecutor.java:294)
>> >> >> >> >> at
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
>> Executor.java:1142)
>> >> >> >> >> at
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >>
>> >> >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
>> lExecutor.java:617)
>> >> >> >> >> at java.lang.Thread.run(Thread.java:748)
>> >> >> >> >
>> >> >> >> >
>> >> >> >> > So our final questions:
>> >> >> >> > 1. Do you think it is the HandleHttpRequest processors
>> problem? Or
>> >> >> >> > there
>> >> >> >> > is
>> >> >> >> > something wrong in our configuration. Anything we can do to
>> avoid
>> >> >> >> > such
>> >> >> >> > problem?
>> >> >> >> > 2. If it's the processor, will you plan to fix it in the coming
>> >> >> >> > version?
>> >> >> >> >
>> >> >> >> > Thank you so much for your reply.
>> >> >> >> >
>> >> >> >> > Kind Regards,
>> >> >> >> > Tian
>> >> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Kind Regards,
>> >> >> >
>> >> >> > Tian Lou
>> >> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Kind Regards,
>> >> >
>> >> > Tian Lou
>> >> >
>> >
>> >
>> >
>> >
>> > --
>> > Kind Regards,
>> >
>> > Tian Lou
>> >
>>
>
>
>
> --
> Kind Regards,
>
> Tian Lou
>
>
--
Kind Regards,
Tian Lou
