[
https://issues.apache.org/jira/browse/FLUME-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leslin (Hong Xiang Lin) updated FLUME-1200:
-------------------------------------------
Attachment: FLUME-1200_1.patch
With Mark's comments, I updated patch with:
The part of the message saying "To change fileType if want output compressed"
is a little unclear. Consider using wording like "Please change the fileType if
compressed output is desired"
Please remove the extraneous whitespace at the end of the lines in the patch.
This shows up as red highlighting on the review board.
> HDFSEventSink causes *.snappy file to be created in HDFS even when snappy
> isn't used (due to missing lib)
> ---------------------------------------------------------------------------------------------------------
>
> Key: FLUME-1200
> URL: https://issues.apache.org/jira/browse/FLUME-1200
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Environment: RHEL 6.2 64-bit
> Reporter: Will McQueen
> Assignee: Leslin (Hong Xiang Lin)
> Fix For: v1.2.0
>
> Attachments: FLUME-1200.patch, FLUME-1200_1.patch
>
>
> If I use HDFSEventSink and specify the codec to be snappy, then the sink
> writes data to HDFS with the ".snappy" extension... but the content of those
> HDFS files is not in snappy format when the snappy libs aren't found. The log
> files mention this:
> 2012-05-11 19:38:49,868 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2012-05-11 19:38:49,868 WARN snappy.LoadSnappy: Snappy native library
> not loaded
> ...and I think it should be an error rather than a warning... the sink
> shouldn't write data at all to HDFS if it's not in the format expected by the
> config file (ie, not compressed with snappy). The config file I used is:
> agent.channels = c1
> agent.sources = r1
> agent.sinks = k1
> #
> agent.channels.c1.type = MEMORY
> #
> agent.sources.r1.channels = c1
> agent.sources.r1.type = SEQ
> #
> agent.sinks.k1.channel = c1
> agent.sinks.k1.type = LOGGER
> #
> agent.sinks.k1.channel = c1
> agent.sinks.k1.type = HDFS
> agent.sinks.k1.hdfs.path = hdfs://<host>:<port>:<path>
> agent.sinks.k1.hdfs.fileType = DataStream
> agent.sinks.k1.hdfs.codeC = SnappyCodec
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
