Hi,
I am trying Snappy in Hadoop 2.4.1, here are my steps:
(CentOS 64-bit)
1)
yum install snappy snappy-devel
2)
added the following
(core-site.xml)
<property>
<name>io.compression.codecs</name>
<value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.SnappyCodec</value>
</property>
3)
mapred-site.xml
<property>
<name>mapreduce.admin.map.child.java.opts</name>
<value>-server -XX:NewRatio=8
-Djava.library.path=/usr/lib/hadoop/lib/native/
-Djava.net.preferIPv4Stack=true</value>
<final>true</final>
</property>
<property>
<name>mapreduce.admin.reduce.child.java.opts</name>
<value>-server -XX:NewRatio=8
-Djava.library.path=/usr/lib/hadoop/lib/native/
-Djava.net.preferIPv4Stack=true</value>
<final>true</final>
</property>
4) smoke test
bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar teragen
100000 /tmp/teragenout
I got the following warning, actually there is no any test file created in hdfs:
14/08/19 22:50:10 WARN mapred.YARNRunner: Usage of -Djava.library.path in
mapreduce.admin.map.child.java.opts can cause programs to no longer function if
hadoop native libraries are used. These values should be set as part of the
LD_LIBRARY_PATH in the map JVM env using mapreduce.admin.user.env config
settings.
14/08/19 22:50:10 WARN mapred.YARNRunner: Usage of -Djava.library.path in
mapreduce.admin.reduce.child.java.opts can cause programs to no longer function
if hadoop native libraries are used. These values should be set as part of the
LD_LIBRARY_PATH in the reduce JVM env using mapreduce.admin.user.env config
settings.
Can anyone please advise how to install and enable SNAPPY in Hadoop 2.4.1? or
what would be wrong? or is my new change in mapred-site.xml incorrect?
Regards
Arthur
