[
https://issues.apache.org/jira/browse/FLUME-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559335#comment-13559335
]
Mohit Anchlia commented on FLUME-1850:
--------------------------------------
I am putting these changes tomorrow in the Test environment. I'll keep an eye
on open and close file logs and see how it goes. Would decreasing max open
files create any performance problems? Wondering why the default is 5000?
In Prod it takea about 2 weeks before it runs out of memory.
> OutOfMemory Error
> -----------------
>
> Key: FLUME-1850
> URL: https://issues.apache.org/jira/browse/FLUME-1850
> Project: Flume
> Issue Type: Bug
> Components: Node
> Affects Versions: v1.3.0
> Environment: RHEL 6
> Reporter: Mohit Anchlia
> Attachments: flume-oo.docx, Screen Shot 2013-01-16 at 11.05.55 PM.png
>
>
> We are using flume-1.3.0. After flume is up for a while (30 days+) we get
> OutOfMemory error. Our heap is set to 2G and load on the system is very low.
> Around 50 request/minute. We use AvroClient and long lived connection.
> Below is the stack trace. I don't have the heap dump but I plan to enable
> that for next time.
> 13/01/16 09:09:38 ERROR hdfs.HDFSEventSink: process failed
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:282)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
> at
> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.append(SequenceFile.java:1320)
> at
> org.apache.flume.sink.hdfs.HDFSSequenceFile.append(HDFSSequenceFile.java:72)
> at
> org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:376)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:729)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:727)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:282)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
> at
> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.append(SequenceFile.java:1320)
> at
> org.apache.flume.sink.hdfs.HDFSSequenceFile.append(HDFSSequenceFile.java:72)
> at
> org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:376)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:729)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:727)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
