[
https://issues.apache.org/jira/browse/FLUME-580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107564#comment-13107564
]
[email protected] commented on FLUME-580:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1949/
-----------------------------------------------------------
(Updated 2011-09-19 00:04:07.721885)
Review request for Flume, Arvind Prabhakar and Roman Shaposhnik.
Changes
-------
added link to jira
Summary
-------
commit b34670902d4c667f98ffe7e619c436148c89e000
Author: Jonathan Hsieh <[email protected]>
Date: Sun Sep 18 11:05:42 2011 -0700
FLUME-580: Flume needs to be consistent with autodiscovery of Hadoop compres
- Adds native paths for hadoop native codec support
This addresses bug FLUME-580.
https://issues.apache.org/jira/browse/FLUME-580
Diffs
-----
bin/flume 183cf70
Diff: https://reviews.apache.org/r/1949/diff
Testing
-------
Built and ran
flume sink 'collectorSource("file:///tmp/snappy", "foo", 10000,
seqfile("snappy")'
typed some stuff at the console, then ^D.
/tmp/snappy dir had files that were snappy codec compressed.
Thanks,
jmhsieh
> Flume needs to be consistent with autodiscovery of Hadoop compression codecs
> ----------------------------------------------------------------------------
>
> Key: FLUME-580
> URL: https://issues.apache.org/jira/browse/FLUME-580
> Project: Flume
> Issue Type: Bug
> Reporter: Disabled imported user
> Assignee: Jonathan Hsieh
>
> Currently, if flume notices that LZO is configured in core-site.xml it tries
> to load it. However, since it doesn't follow the same route that Hadoop does
> for locating the JAR and corresponding so files it fails to do so. Flume has
> to leverage the same logic that hadoop has embedded in bin/hadoop:
> .....
> # setup 'java.library.path' for native-hadoop code if necessary
> if [ -d "${HADOOP_HOME}/build/native" -o -d "${HADOOP_HOME}/lib/native" -o -d
> "${HADOOP_HOME}/sbin" ]; then
> JAVA_PLATFORM=`CLASSPATH=${CLASSPATH} ${JAVA} -Xmx32m
> ${HADOOP_JAVA_PLATFORM_OPTS} org.apache.hadoop.util.PlatformName | sed -e "s/
> /_/g"`
> ...
> so that it can transparently find LZO objects that it tries to autodiscover.
> For the time being, however, the workaround seems to be to tell flume
> explicitly where to look for LZO via
> adding the following to /usr/lib/flume/bin/flume-env.sh:
> export JAVA_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/
> export
> FLUME_CLASSPATH=/usr/lib/hadoop-0.20/lib/hadoop-lzo-20101122174751.20101122171345.552b3f9.jar
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
