Awesome! Thanks Joe. *Jeremiah Adams*
Senior Software Developer Pearson 2154 East Commons Ave. Suite 400 Centennial, CO 80122 Always Learning Learn more at www.pearson.com On Tue, Jul 28, 2015 at 4:52 PM, Joe Witt <[email protected]> wrote: > You'll want to uncomment both of these lines actually: > > #java.arg.11=-XX:PermSize=128M > #java.arg.12=-XX:MaxPermSize=128M > > And if you like you can make them a bit larger. > > Thanks > Joe > > On Tue, Jul 28, 2015 at 6:52 PM, Joe Witt <[email protected]> wrote: > > Jeremiah, > > > > In this case it is talking about permgen (Java 7 and where classes go). > > > > If you take a look at the './conf/bootstap.conf' file there is a line > that says: > > > > #java.arg.11=-XX:PermSize=128M > > > > Uncomment that and you should be good to go. We need more than the > > typical Java 7 default for PermGen space. > > > > Of note the MergeContent processor is extremely memory efficient and > > pretty much a non-factor on the heap as it never actually loads > > content into memory. Very different than most systems would be. > > > > Thanks > > Joe > > > > On Tue, Jul 28, 2015 at 6:47 PM, Adams, Jeremiah > > <[email protected]> wrote: > >> I am running into memory issues when using MergeContent. I am seeing > >> OutofMemoryErrors. I am using 0.1.0. > >> > >> I am processing a kafka queue via GetKafka, some custom Processors and > >> PutKafka. I have a PutKafka Controller just prior to the MergeContent > >> Processor. I am passing JSON messages through the FlowFiles. > >> > >> I am looking to merge the flow files until reaching a minimum size or a > >> time period elapses, then write them to S3. I get the memory issue with > >> just 15-20 messages. The MergeContent Processor continues to update the > >> Tasks in Tasks/Time even when not sending messages to the flow. > >> > >> Here are my settings for the MergeContent Processor: > >> Merge Strategy: Bin-Packing Algo > >> Merge Format: Binary Concatenation > >> Attribute Strategy: Keep only Common Attributes > >> Correlation Attribute Name: No Value Set > >> Minimum Number of Entries: 1 > >> Maximum Number of Entries: No Value Set > >> Minimum Group Size: 1KB > >> Maximum Group Size: 2 MB > >> Max Bin Age: 10 m > >> Maximum number of Bins: 100 > >> Header File: No Value Set > >> Footer File: No Value Set > >> Demarcator File: No Value Set > >> Compression Level: 1 > >> Keep Path: False > >> > >> 2015-07-28 14:25:17,580 WARN [NiFi Web Server-22] > >> org.eclipse.jetty.servlet.ServletHandler Error for > >> > /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status > >> java.lang.OutOfMemoryError: PermGen space > >> 2015-07-28 14:25:17,811 WARN [NiFi Web Server-22] > >> org.eclipse.jetty.server.HttpChannel > >> > /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status?recursive=false > >> java.lang.OutOfMemoryError: PermGen space > >> 2015-07-28 14:25:18,020 WARN [NiFi Web Server-22] > >> o.e.jetty.util.thread.QueuedThreadPool > >> java.lang.OutOfMemoryError: PermGen space > >> 2015-07-28 14:25:18,020 WARN [NiFi Web Server-22] > >> o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death: > >> org.eclipse.jetty.util.thread.QueuedThreadPool$3@1a4b0747 in NiFi Web > >> Server{STARTED,8<=9<=200,i=0,q=4} > >> 2015-07-28 14:25:18,775 WARN [NiFi Web Server-23] > >> org.eclipse.jetty.servlet.ServletHandler Error for > >> > /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status > >> java.lang.OutOfMemoryError: PermGen space > >> 2015-07-28 14:25:19,005 WARN [NiFi Web Server-23] > >> org.eclipse.jetty.server.HttpChannel > >> > /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status?recursive=false > >> java.lang.OutOfMemoryError: PermGen space > >> 2015-07-28 14:25:19,240 WARN [NiFi Web Server-23] > >> o.e.jetty.util.thread.QueuedThreadPool > >> java.lang.OutOfMemoryError: PermGen space > >> 2015-07-28 14:25:19,240 WARN [NiFi Web Server-23] > >> o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death: > >> org.eclipse.jetty.util.thread.QueuedThreadPool$3@1a4b0747 in NiFi Web > >> Server{STARTED,8<=9<=200,i=0,q=4} > >> > >> *Jeremiah Adams* > >> > >> Senior Software Developer > >> Pearson > >> > >> 2154 East Commons Ave. > >> Suite 400 > >> Centennial, CO 80122 > >> > >> > >> Always Learning > >> Learn more at www.pearson.com >
