hadoop 0.23: native compression libraries not being loaded
----------------------------------------------------------
Key: MAPREDUCE-3515
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3515
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Wing Yew Poon
I installed the hadoop package from the Bigtop hadoop 0.23 branch. Among other
files, the package installs
/usr/lib/hadoop/lib/native
/usr/lib/hadoop/lib/native/libhadoop.a
/usr/lib/hadoop/lib/native/libhadoop.so.1
/usr/lib/hadoop/lib/native/libhadoop.so.1.0.0
/usr/lib/hadoop/lib/native/libhdfs.a
I ran a simple job using compression:
hadoop jar /usr/lib/hadoop/hadoop-mapreduce-examples.jar wordcount -D
mapreduce.output.fileoutputformat.compress=true -D
mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.compress.GzipCodec
examples/text wordcount-gz
I see
11/12/06 13:42:06 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
11/12/06 13:42:06 WARN snappy.LoadSnappy: Snappy native library not loaded
and at the end, I see
-rw-r--r-- 1 root supergroup 0 2011-12-06 13:42 wordcount-gz/_SUCCESS
-rw-r--r-- 1 root supergroup 46228 2011-12-06 13:42
wordcount-gz/part-r-00000.gz
so the output is compressed, but from the log message, I assume that the native
library is not being loaded and that the java gzip is being used.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
