escapedFormatDfs goes into a file creation frenzy
-------------------------------------------------
Key: FLUME-734
URL: https://issues.apache.org/jira/browse/FLUME-734
Project: Flume
Issue Type: Bug
Components: Sinks+Sources
Affects Versions: v0.9.4
Environment: CentOS 5.6
Reporter: Eran Kutner
Priority: Critical
Attachments: flume.log
Using this configuration:
collectorSource(54001) | collector(600000) {
escapedFormatDfs("hdfs://hadoop1-m1:8020/raw-events/%Y-%m-%d/",
"events-%{rolltag}-col1.snappy", seqfile("SnappyCodec")) }
The expected behavior is to see a new file created every 10 minutes. However,
once in a while the collector would go into a file creation frenzy, creating
new files every second.
The log indicates that writing has failed with error: "OutputFormat instance
can only write to the same OutputStream" causing the file to be closed a new
one to be opened just to be closed again.
Looking at the code I'm not even sure how the output stream could change but
the behavior I'm seeing feels like some sort of a race condition. It is
happening much more under heavy load than under low load.
See attached log excerpt.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
