Hi, experts.
I meet the following problem when configuring the Snappy lib in Hadoop2.5.2
My snappy installation home is /opt/snappy
My Hadoop installation home is /opt/hadoop/hadoophome
To configure the snappy path, I tried to add the following environment
variables in /etc/profile and hadoop-env.sh :
export JAVA_LIBRARY_PATH=/opt/hadoop/hadoophome/lib/native:/opt/snappy/lib
export LD_LIBRARY_PATH=/opt/hadoop/hadoophome/lib/native:/opt/snappy/lib
After the configuration, I ran the command "hadoop checknative". The result
showed as following which I think means the hadoop can find the snappy lib:
Native library checking:
hadoop: true /opt/hadoop/hadoop-2.5.2/lib/native/libhadoop.so.1.0.0
zlib: true /lib64/libz.so.1
snappy: true /opt/snappy/lib/libsnappy.so.1
lz4: true revision:99
bzip2: false
But when I ran a MapReduce Job, it reported the following error:
Error: java.lang.RuntimeException: native snappy library not available:
SnappyCompressor has not been loaded.
at
org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:69)
at
org.apache.hadoop.io.compress.SnappyCodec.createCompressor(SnappyCodec.java:143)
at
org.apache.hadoop.io.compress.SnappyCodec.createOutputStream(SnappyCodec.java:98)
at
org.apache.hadoop.mapreduce.lib.output.TextOutputFormat.getRecordWriter(TextOutputFormat.java:136)
I also tried to set io.compression.codecs, but it did not work too.
The only way I found worked is to make a soft link as following:
ln -s /opt/snappy/lib/libsnappy.so.1.2.1
/opt/hadoop/hadoophome/lib/native/libsnappy.so.1
I used to config snappy in Hadoop2.4.0 successfully. I remembered that I only
need to config the LD_LIBRARY_PATH in /etc/profile. There is no need to make a
such soft link. Does Hadoop2.5.2 not support this configuration anymore? Or is
there other ways to config in Hadoop2.5.2 which don't require to make links or
have to copy the lib to the hadoop's lib/native directory?
Many Thanks!
