I am running into memory issues when using MergeContent. I am seeing
OutofMemoryErrors. I am using 0.1.0.
I am processing a kafka queue via GetKafka, some custom Processors and
PutKafka. I have a PutKafka Controller just prior to the MergeContent
Processor. I am passing JSON messages through the FlowFiles.
I am looking to merge the flow files until reaching a minimum size or a
time period elapses, then write them to S3. I get the memory issue with
just 15-20 messages. The MergeContent Processor continues to update the
Tasks in Tasks/Time even when not sending messages to the flow.
Here are my settings for the MergeContent Processor:
Merge Strategy: Bin-Packing Algo
Merge Format: Binary Concatenation
Attribute Strategy: Keep only Common Attributes
Correlation Attribute Name: No Value Set
Minimum Number of Entries: 1
Maximum Number of Entries: No Value Set
Minimum Group Size: 1KB
Maximum Group Size: 2 MB
Max Bin Age: 10 m
Maximum number of Bins: 100
Header File: No Value Set
Footer File: No Value Set
Demarcator File: No Value Set
Compression Level: 1
Keep Path: False
2015-07-28 14:25:17,580 WARN [NiFi Web Server-22]
org.eclipse.jetty.servlet.ServletHandler Error for
/nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status
java.lang.OutOfMemoryError: PermGen space
2015-07-28 14:25:17,811 WARN [NiFi Web Server-22]
org.eclipse.jetty.server.HttpChannel
/nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status?recursive=false
java.lang.OutOfMemoryError: PermGen space
2015-07-28 14:25:18,020 WARN [NiFi Web Server-22]
o.e.jetty.util.thread.QueuedThreadPool
java.lang.OutOfMemoryError: PermGen space
2015-07-28 14:25:18,020 WARN [NiFi Web Server-22]
o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death:
org.eclipse.jetty.util.thread.QueuedThreadPool$3@1a4b0747 in NiFi Web
Server{STARTED,8<=9<=200,i=0,q=4}
2015-07-28 14:25:18,775 WARN [NiFi Web Server-23]
org.eclipse.jetty.servlet.ServletHandler Error for
/nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status
java.lang.OutOfMemoryError: PermGen space
2015-07-28 14:25:19,005 WARN [NiFi Web Server-23]
org.eclipse.jetty.server.HttpChannel
/nifi-api/controller/process-groups/caffa1c4-ef49-4f0c-b144-0b5d47264481/status?recursive=false
java.lang.OutOfMemoryError: PermGen space
2015-07-28 14:25:19,240 WARN [NiFi Web Server-23]
o.e.jetty.util.thread.QueuedThreadPool
java.lang.OutOfMemoryError: PermGen space
2015-07-28 14:25:19,240 WARN [NiFi Web Server-23]
o.e.jetty.util.thread.QueuedThreadPool Unexpected thread death:
org.eclipse.jetty.util.thread.QueuedThreadPool$3@1a4b0747 in NiFi Web
Server{STARTED,8<=9<=200,i=0,q=4}
*Jeremiah Adams*
Senior Software Developer
Pearson
2154 East Commons Ave.
Suite 400
Centennial, CO 80122
Always Learning
Learn more at www.pearson.com
