[
https://issues.apache.org/jira/browse/FLUME-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556912#comment-13556912
]
Connor Woodson commented on FLUME-1850:
---------------------------------------
One solution that involves no code changes is to just reduce hdfs.maxOpenFiles
in your configuration to automatically close your unused BucketWriters (it
should be no more than 2 * the possible number of open files; 2x to avoid
opening/closing many different files around the turn of an hour).
And a list of code modifications that would solve this issue which come to mind:
A simple change to the code would be to not cancel the idleTimeout when the
rollInterval occurs.
And the most complex solution would be to be able to detect if a timestamped
path is outdated/won't occur again (I think that's the most significant cause
of this issue).
Another way is to replace the 'idleTimeoutCallback' with
'removeReferenceCallback' which does the exact same thing; this callback will
get called either when the writer idles out, or when it closes without
reopening.
And one last option I suppose is to automatically close files/destory
bucketwriters after they roll, but that implementation would be tricky and
nuanced (rolling in the middle of the batch).
> OutOfMemory Error
> -----------------
>
> Key: FLUME-1850
> URL: https://issues.apache.org/jira/browse/FLUME-1850
> Project: Flume
> Issue Type: Bug
> Components: Node
> Affects Versions: v1.3.0
> Environment: RHEL 6
> Reporter: Mohit Anchlia
> Attachments: flume-oo.docx, Screen Shot 2013-01-16 at 11.05.55 PM.png
>
>
> We are using flume-1.3.0. After flume is up for a while (30 days+) we get
> OutOfMemory error. Our heap is set to 2G and load on the system is very low.
> Around 50 request/minute. We use AvroClient and long lived connection.
> Below is the stack trace. I don't have the heap dump but I plan to enable
> that for next time.
> 13/01/16 09:09:38 ERROR hdfs.HDFSEventSink: process failed
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:282)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
> at
> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.append(SequenceFile.java:1320)
> at
> org.apache.flume.sink.hdfs.HDFSSequenceFile.append(HDFSSequenceFile.java:72)
> at
> org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:376)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:729)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:727)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor"
> java.lang.OutOfMemoryError: Java heap space
> at java.util.Arrays.copyOf(Arrays.java:2786)
> at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94)
> at java.io.DataOutputStream.write(DataOutputStream.java:90)
> at org.apache.hadoop.io.Text.write(Text.java:282)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:90)
> at
> org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:77)
> at
> org.apache.hadoop.io.SequenceFile$BlockCompressWriter.append(SequenceFile.java:1320)
> at
> org.apache.flume.sink.hdfs.HDFSSequenceFile.append(HDFSSequenceFile.java:72)
> at
> org.apache.flume.sink.hdfs.BucketWriter.append(BucketWriter.java:376)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:729)
> at
> org.apache.flume.sink.hdfs.HDFSEventSink$2.call(HDFSEventSink.java:727)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
