Joseph,
An inspection of a heap dump created when the out of memory
condition occurs will shed some light on the specific cause of the issue.
The following command can be used to get a heap dump.:
jmap -dump:format=b,file=nifi-heap-dump.bin <PID>
you can get the PID using the following command:
ps -ef|grep org.apache.nifi.NiFi
One common cause of out of memory on the NCM is related to the amount of
components status retained. You can edit the nifi.properties file to
decrease the amount of stats retained or change the duration of those stats
collected:
# Component Status Repository
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
nifi.components.status.repository.buffer.size=1440
nifi.components.status.snapshot.frequency=1 min
Either decrease the buffer size (will retain less stats for each component)
or decrease frequency of which stats are collected.
Thanks,
Matt
On Mon, Feb 15, 2016 at 9:33 AM, Gresock, Joseph <[email protected]>
wrote:
> Devs,
>
> We've been seeing some OutOfMemoryErrors on the NCM of our 10-node cluster
> recently. The flow has ~600 processors, and the NCM runs on a VM with 8GB
> RAM. We have 6G allocated to the Nifi JVM on this node.
>
> The specific log message we see is:
>
> WARN [Process NCM Request-6] org.apache.nifi.io.socket.SocketListener
> Dispatching socket request encountered exception due to:
> java.lang.OutOfMemoryError: Java heap space
>
> First, I'm hoping there's some advice on how to avoid this in the first
> place, but barring that, is there a way to configure Nifi to auto-restart
> the NCM when it gets this error? I seem to remember seeing this in the
> past, but I couldn't find anything in bootstrap.conf or nifi.properties
> that looked related.
>
> Thanks,
> Joe
>
