[
https://issues.apache.org/jira/browse/FLUME-734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13118202#comment-13118202
]
[email protected] commented on FLUME-734:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2129/#review2221
-----------------------------------------------------------
Ship it!
Looks fine to me.
Note that the TestRollSink.java has a new test case getting added as part of
Flume-768. The EscapedCustomDfsSink is called from that new test. We'll need
that minor change in the new test after both patches are committed.
thanx
Prasad
- Prasad
On 2011-09-30 15:09:20, jmhsieh wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/2129/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-30 15:09:20)
bq.
bq.
bq. Review request for Flume, Prasad Mujumdar and Eric Sammer.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. Output formats are now changed into function expressions and a new output
format is created for each customdfssink. (this happens when there are multiple
files being written by a escapedFormatDfs, and there was a shared outputformat
object).
bq.
bq. commit 20a85af7d21a2a33c63903c794b4bd0d3dd2be02
bq. Author: Jonathan Hsieh <[email protected]>
bq. Date: Wed Aug 10 09:18:02 2011 -0700
bq.
bq. FLUME-734: escapedFormatDfs goes into a file creation frenzy
bq.
bq.
bq. This addresses bug flume-734.
bq. https://issues.apache.org/jira/browse/flume-734
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. flume-core/src/main/java/com/cloudera/flume/conf/FlumeConfiguration.java
8ecfed1
bq. flume-core/src/main/java/com/cloudera/flume/conf/SinkBuilderUtil.java
2aaa566
bq.
flume-core/src/main/java/com/cloudera/flume/handlers/hdfs/EscapedCustomDfsSink.java
20ebdfd
bq.
flume-core/src/test/java/com/cloudera/flume/handlers/hdfs/TestEscapedCustomOutputDfs.java
7618acd
bq.
flume-core/src/test/java/com/cloudera/flume/handlers/rolling/TestRollSink.java
1fd788f
bq.
bq. Diff: https://reviews.apache.org/r/2129/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Tests pass.
bq.
bq. From comments on jira, an earlier version of this patch is working for a
few folks already.
bq.
bq.
bq. Thanks,
bq.
bq. jmhsieh
bq.
bq.
> escapedFormatDfs goes into a file creation frenzy
> -------------------------------------------------
>
> Key: FLUME-734
> URL: https://issues.apache.org/jira/browse/FLUME-734
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v0.9.4
> Environment: CentOS 5.6
> Reporter: Eran Kutner
> Assignee: Jonathan Hsieh
> Priority: Critical
> Attachments: FLUME-734-draft.patch, flume.log
>
>
> Using this configuration:
> collectorSource(54001) | collector(600000) {
> escapedFormatDfs("hdfs://hadoop1-m1:8020/raw-events/%Y-%m-%d/",
> "events-%{rolltag}-col1.snappy", seqfile("SnappyCodec")) }
> The expected behavior is to see a new file created every 10 minutes. However,
> once in a while the collector would go into a file creation frenzy, creating
> new files every second.
> The log indicates that writing has failed with error: "OutputFormat instance
> can only write to the same OutputStream" causing the file to be closed a new
> one to be opened just to be closed again.
> Looking at the code I'm not even sure how the output stream could change but
> the behavior I'm seeing feels like some sort of a race condition. It is
> happening much more under heavy load than under low load.
> See attached log excerpt.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
