[
https://issues.apache.org/jira/browse/FLUME-734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Luke Forehand updated FLUME-734:
--------------------------------
Comment: was deleted
(was: We are struggling with this same problem, we are seeing the exception
with version 0.9.4
{{{
011-09-19 17:39:42,772 INFO com.cloudera.flume.core.connector.DirectDriver:
Connector logicalNode sink.hdfs-24 exited with error: OutputFormat instance can
only write to the same OutputStream
java.io.IOException: OutputFormat instance can only write to the same
OutputStream
at
com.ni.flume.outputformat.SeqFileJsonOutputFormat.format(SeqFileJsonOutputFormat.java:78)
at
com.cloudera.flume.handlers.hdfs.CustomDfsSink.append(CustomDfsSink.java:80)
at
com.cloudera.flume.handlers.hdfs.EscapedCustomDfsSink.append(EscapedCustomDfsSink.java:123)
at com.cloudera.flume.core.CompositeSink.append(CompositeSink.java:61)
at
com.cloudera.flume.handlers.rolling.RollSink.synchronousAppend(RollSink.java:234)
}}})
> escapedFormatDfs goes into a file creation frenzy
> -------------------------------------------------
>
> Key: FLUME-734
> URL: https://issues.apache.org/jira/browse/FLUME-734
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v0.9.4
> Environment: CentOS 5.6
> Reporter: Eran Kutner
> Assignee: Jonathan Hsieh
> Priority: Critical
> Attachments: FLUME-734-draft.patch, flume.log
>
>
> Using this configuration:
> collectorSource(54001) | collector(600000) {
> escapedFormatDfs("hdfs://hadoop1-m1:8020/raw-events/%Y-%m-%d/",
> "events-%{rolltag}-col1.snappy", seqfile("SnappyCodec")) }
> The expected behavior is to see a new file created every 10 minutes. However,
> once in a while the collector would go into a file creation frenzy, creating
> new files every second.
> The log indicates that writing has failed with error: "OutputFormat instance
> can only write to the same OutputStream" causing the file to be closed a new
> one to be opened just to be closed again.
> Looking at the code I'm not even sure how the output stream could change but
> the behavior I'm seeing feels like some sort of a race condition. It is
> happening much more under heavy load than under low load.
> See attached log excerpt.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
