Really the setting of that depends on what you are trying to do in the flow and what else runs on your systems. I have a cluster of 4 boxes that have 12GB of Ram and I set those to 4096. That cluster runs just fine and pushes a few hundred gigs a day at about 30% system usage. Depending on what your custom processors are doing with the heap, I would boost that up a bit. You should also look at ganglia or something in that family to watch for memory leaks in the custom code. NIFI has a really good ganglia reporting task to show you what is happening at a system level within the heap.
On Fri, Sep 4, 2015 at 11:24 AM, Whitmore, Mattie [USA] < [email protected]> wrote: > Thanks for all the speedy replies! > > > 1. I will up the heap currently it is set at: java.arg.3=-Xmx512m > 2. I have 15 GB of memory (m3.xlarge) > 3. I do have a number of custom processors in my workflow > > I’ll update ya’ll about how the increase in heap affects this issue. I > am also working on serializing these HTTP calls into one processor, do > ya’ll have any thoughts on that solution? > > > *Mattie Whitmore* > > > From: Corey Flowers > Reply-To: "[email protected]" > Date: Friday, September 4, 2015 at 11:05 AM > To: "[email protected]" > Subject: [External] Re: GitHTTP out of memory error > > What is your heap space set to and how much memory do you have on the box? > > > On Fri, Sep 4, 2015 at 10:57 AM, Whitmore, Mattie [USA] < > [email protected]> wrote: > >> Hi, >> >> I am 8 GetHTTP processors which run nightly, however 5 hours after these >> run some threads out of memory: >> >> ERROR [Timer-Driven Process Thread-23] >> o.a.nifi.processors.standard.GetHTTP >> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] >> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] failed to process session >> due to java.lang.OutOfMemoryError: Java heap space: >> java.lang.OutOfMemoryError: Java heap space >> >> 2015-09-03 07:10:41,167 ERROR [Timer-Driven Process Thread-11] >> o.a.nifi.processors.standard.GetHTTP >> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] >> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] failed to process session >> due to java.lang.OutOfMemoryError: Java heap space: >> java.lang.OutOfMemoryError: Java heap space >> >> 2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-23] >> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding >> GetHTTP[id=021562be-b123-4b3f-b5ed-2b8c47af6d4c] due to uncaught Exception: >> java.lang.OutOfMemoryError: Java heap space >> >> 2015-09-03 07:10:41,168 WARN [Timer-Driven Process Thread-11] >> o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding >> GetHTTP[id=0a5158d2-f082-4007-b646-3da8fb436fd0] due to uncaught Exception: >> java.lang.OutOfMemoryError: Java heap space >> >> I am running on Amazon Linux and have tuned the server as described in >> the admin page. >> >> Thanks! >> >> *Mattie Whitmore* >> Lead Technologist >> >> Booz | Allen | Hamilton >> *________________________* >> >> >> > > > -- > Corey Flowers > Vice President, Onyx Point, Inc > (410) 541-6699 > [email protected] > > -- This account not approved for unencrypted proprietary information -- > -- Corey Flowers Vice President, Onyx Point, Inc (410) 541-6699 [email protected] -- This account not approved for unencrypted proprietary information --
