[
https://issues.apache.org/jira/browse/FLUME-580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13107916#comment-13107916
]
[email protected] commented on FLUME-580:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1949/#review1963
-----------------------------------------------------------
Ship it!
LGTM.
- Roman
On 2011-09-19 00:04:07, jmhsieh wrote:
bq.
bq. -----------------------------------------------------------
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1949/
bq. -----------------------------------------------------------
bq.
bq. (Updated 2011-09-19 00:04:07)
bq.
bq.
bq. Review request for Flume, Arvind Prabhakar and Roman Shaposhnik.
bq.
bq.
bq. Summary
bq. -------
bq.
bq. commit b34670902d4c667f98ffe7e619c436148c89e000
bq. Author: Jonathan Hsieh <[email protected]>
bq. Date: Sun Sep 18 11:05:42 2011 -0700
bq.
bq. FLUME-580: Flume needs to be consistent with autodiscovery of Hadoop
compres
bq.
bq. - Adds native paths for hadoop native codec support
bq.
bq.
bq. This addresses bug FLUME-580.
bq. https://issues.apache.org/jira/browse/FLUME-580
bq.
bq.
bq. Diffs
bq. -----
bq.
bq. bin/flume 183cf70
bq.
bq. Diff: https://reviews.apache.org/r/1949/diff
bq.
bq.
bq. Testing
bq. -------
bq.
bq. Built and ran
bq.
bq. flume sink 'collectorSource("file:///tmp/snappy", "foo", 10000,
seqfile("snappy")'
bq.
bq. typed some stuff at the console, then ^D.
bq.
bq. /tmp/snappy dir had files that were snappy codec compressed.
bq.
bq.
bq. Thanks,
bq.
bq. jmhsieh
bq.
bq.
> Flume needs to be consistent with autodiscovery of Hadoop compression codecs
> ----------------------------------------------------------------------------
>
> Key: FLUME-580
> URL: https://issues.apache.org/jira/browse/FLUME-580
> Project: Flume
> Issue Type: Bug
> Affects Versions: v0.9.3
> Reporter: Disabled imported user
> Assignee: Jonathan Hsieh
> Fix For: v0.9.5
>
> Attachments:
> 0001-FLUME-580-Flume-needs-to-be-consistent-with-autodisc.patch
>
>
> Currently, if flume notices that LZO is configured in core-site.xml it tries
> to load it. However, since it doesn't follow the same route that Hadoop does
> for locating the JAR and corresponding so files it fails to do so. Flume has
> to leverage the same logic that hadoop has embedded in bin/hadoop:
> .....
> # setup 'java.library.path' for native-hadoop code if necessary
> if [ -d "${HADOOP_HOME}/build/native" -o -d "${HADOOP_HOME}/lib/native" -o -d
> "${HADOOP_HOME}/sbin" ]; then
> JAVA_PLATFORM=`CLASSPATH=${CLASSPATH} ${JAVA} -Xmx32m
> ${HADOOP_JAVA_PLATFORM_OPTS} org.apache.hadoop.util.PlatformName | sed -e "s/
> /_/g"`
> ...
> so that it can transparently find LZO objects that it tries to autodiscover.
> For the time being, however, the workaround seems to be to tell flume
> explicitly where to look for LZO via
> adding the following to /usr/lib/flume/bin/flume-env.sh:
> export JAVA_LIBRARY_PATH=/usr/lib/hadoop-0.20/lib/native/Linux-amd64-64/
> export
> FLUME_CLASSPATH=/usr/lib/hadoop-0.20/lib/hadoop-lzo-20101122174751.20101122171345.552b3f9.jar
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
