[
https://issues.apache.org/jira/browse/FLUME-1200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13395899#comment-13395899
]
Leslin (Hong Xiang Lin) commented on FLUME-1200:
------------------------------------------------
Posted for code review at https://reviews.apache.org/r/5378/
> HDFSEventSink causes *.snappy file to be created in HDFS even when snappy
> isn't used (due to missing lib)
> ---------------------------------------------------------------------------------------------------------
>
> Key: FLUME-1200
> URL: https://issues.apache.org/jira/browse/FLUME-1200
> Project: Flume
> Issue Type: Bug
> Components: Sinks+Sources
> Affects Versions: v1.2.0
> Environment: RHEL 6.2 64-bit
> Reporter: Will McQueen
> Assignee: Leslin (Hong Xiang Lin)
> Fix For: v1.2.0
>
> Attachments: FLUME-1200.patch
>
>
> If I use HDFSEventSink and specify the codec to be snappy, then the sink
> writes data to HDFS with the ".snappy" extension... but the content of those
> HDFS files is not in snappy format when the snappy libs aren't found. The log
> files mention this:
> 2012-05-11 19:38:49,868 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 2012-05-11 19:38:49,868 WARN snappy.LoadSnappy: Snappy native library
> not loaded
> ...and I think it should be an error rather than a warning... the sink
> shouldn't write data at all to HDFS if it's not in the format expected by the
> config file (ie, not compressed with snappy). The config file I used is:
> agent.channels = c1
> agent.sources = r1
> agent.sinks = k1
> #
> agent.channels.c1.type = MEMORY
> #
> agent.sources.r1.channels = c1
> agent.sources.r1.type = SEQ
> #
> agent.sinks.k1.channel = c1
> agent.sinks.k1.type = LOGGER
> #
> agent.sinks.k1.channel = c1
> agent.sinks.k1.type = HDFS
> agent.sinks.k1.hdfs.path = hdfs://<host>:<port>:<path>
> agent.sinks.k1.hdfs.fileType = DataStream
> agent.sinks.k1.hdfs.codeC = SnappyCodec
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
