Jeremiah, In this case it is talking about permgen (Java 7 and where classes go).
If you take a look at the './conf/bootstap.conf' file there is a line that says: #java.arg.11=-XX:PermSize=128M Uncomment that and you should be good to go. We need more than the typical Java 7 default for PermGen space. Of note the MergeContent processor is extremely memory efficient and pretty much a non-factor on the heap as it never actually loads content into memory. Very different than most systems would be. Thanks Joe On Tue, Jul 28, 2015 at 6:47 PM, Adams, Jeremiah <[email protected]> wrote: > I am running into memory issues when using MergeContent. I am seeing > OutofMemoryErrors. I am using 0.1.0. > > I am processing a kafka queue via GetKafka, some custom Processors and > PutKafka. I have a PutKafka Controller just prior to the MergeContent > Processor. I am passing JSON messages through the FlowFiles. > > I am looking to merge the flow files until reaching a minimum size or a > time period elapses, then write them to S3. I get the memory issue with > just 15-20 messages. The MergeContent Processor continues to update the > Tasks in Tasks/Time even when not sending messages to the flow. > > Here are my settings for the MergeContent Processor: > Merge Strategy: Bin-Packing Algo > Merge Format: Binary Concatenation > Attribute Strategy: Keep only Common Attributes > Correlation Attribute Name: No Value Set > Minimum Number of Entries: 1 > Maximum Number of Entries: No Value Set > Minimum Group Size: 1KB > Maximum Group Size: 2 MB > Max Bin Age: 10 m > Maximum number of Bins: 100 > Header File: No Value Set > Footer File: No Value Set > Demarcator File: No Value Set > Compression Level: 1 > Keep Path: False > > 2015-07-28 14:25:17,580 WARN [NiFi Web Server-22] > org.eclipse.jetty.servlet.ServletHandler Error for > /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status > java.lang.OutOfMemoryError: PermGen space > 2015-07-28 14:25:17,811 WARN [NiFi Web Server-22] > org.eclipse.jetty.server.HttpChannel > /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status?recursive=false > java.lang.OutOfMemoryError: PermGen space > 2015-07-28 14:25:18,020 WARN [NiFi Web Server-22] > o.e.jetty.util.thread.QueuedThreadPool > java.lang.OutOfMemoryError: PermGen space > 2015-07-28 14:25:18,020 WARN [NiFi Web Server-22] > o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death: > org.eclipse.jetty.util.thread.QueuedThreadPool$3@1a4b0747 in NiFi Web > Server{STARTED,8<=9<=200,i=0,q=4} > 2015-07-28 14:25:18,775 WARN [NiFi Web Server-23] > org.eclipse.jetty.servlet.ServletHandler Error for > /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status > java.lang.OutOfMemoryError: PermGen space > 2015-07-28 14:25:19,005 WARN [NiFi Web Server-23] > org.eclipse.jetty.server.HttpChannel > /nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status?recursive=false > java.lang.OutOfMemoryError: PermGen space > 2015-07-28 14:25:19,240 WARN [NiFi Web Server-23] > o.e.jetty.util.thread.QueuedThreadPool > java.lang.OutOfMemoryError: PermGen space > 2015-07-28 14:25:19,240 WARN [NiFi Web Server-23] > o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death: > org.eclipse.jetty.util.thread.QueuedThreadPool$3@1a4b0747 in NiFi Web > Server{STARTED,8<=9<=200,i=0,q=4} > > *Jeremiah Adams* > > Senior Software Developer > Pearson > > 2154 East Commons Ave. > Suite 400 > Centennial, CO 80122 > > > Always Learning > Learn more at www.pearson.com
