Hi, I found out that copying the files from native folder to /usr/local/hadoop/lib folder solved the problem but the main issue then is why hadoop is not able to pick up the native libraries based on the configuration of environment variables; i.e, why is java.library.path not set from LD_LIBRARY_PATH or JAVA_LIBRARY_PATH which are set to the folder /usr/local/hadoop/lib/native/Linux-i386-32?
Kindly note that I am using Ubuntu 12.04 and there may be a difference in the way it handles the LD_LIBRARY_PATH environment variables but this is not the case since SQOOP is able to use the environment variable LD_LIBRARY_PATH for locating Oracle JDBC jar files. I will be grateful if someone could kindly point out what I am doing wrong here, though there is a work around for now. Thanks and Regards, Gourav On Tue, Jul 24, 2012 at 5:49 PM, Gourav Sengupta <gourav.had...@gmail.com>wrote: > Hi, > > The HADOOP version is 0.20.205 and the jar file is hadoop-lzo-0.4.15.jar. > > I took the following steps in order to enable LZO compression in Ubuntu > HADOOP: > 1. sudo apt-get install liblzo2-dev > > 2. git clone https://github.com/kevinweil/hadoop-lzo.git > > 3. cd hadoop-lzo > > 4. changed the build.xml file and included the line <classpath > refid="classpath"/> in <javah > classpath="${build.classes}".................> tag > > 5. ran the following commands: > export CFLAGS=-m32 > export CXXFLAGS=-m32 > ant compile-native > ant jar > > 6. copied the file hadoop-lzo-0.4.15.jar to /usr/local/hadoop/lib > > 7. tar -cBf - -C ./build/hadoop-lzo-0.4.15/lib/native . | tar -xBvf - -C > $HADOOP/lib/native > > 8. Added the following to core-site.xml: > <property> > <name>io.compression.codecs</name> > <value>org.apache.hadoop.io.compress.GzipCodec,org.apache.hadoop.io.compress.DefaultCodec,org.apache.hadoop.io.compress.BZip2Codec,com.hadoop.compression.lzo.LzoCodec,com.hadoop.compression.lzo.LzopCodec</value> > > </property> > <property> > <name>io.compression.codec.lzo.class</name> > <value>com.hadoop.compression.lzo.LzoCodec</value> > </property> > > 9. Added the following line to mapred-site,xml > <property> > <name>mapred.child.env</name> > <value>JAVA_LIBRARY_PATH=/path/to/your/native/hadoop-lzo/libs</value> > </property> > > 10. Modified $HADOOP/bin/hadoop so that the line JAVA_LIBRARY_PATH='' is > commented > > 11. created the following environmental variables in hadoop-env.sh > export > JAVA_LIBRARY_PATH=/usr/local/hadoop/lib/native/Linux-i386-32:/usr/local/hadoop/lib:/usr/local/hadoop/lib/hadoop-lzo-0.4.15.jar > export > LD_LIBRARY_PATH=/usr/local/oraclejdbcconnector:/usr/local/hadoop/lib/native/Linux-i386-32:/usr/lib/i386-linux-gnu:/usr/local/hadoop/lib/hadoop-lzo-0.4.15.jar > export > HADOOP_CLASSPATH=$HADOOP/lib:/usr/local/hadoop/lib/hadoop-lzo-0.4.15.jar > > 12. restarted HADOOP > > 13. On running the command "hadoop jar > /usr/local/hadoop/lib/hadoop-lzo-0.4.15.jar > com.hadoop.compression.lzo.LzoIndexer /tmp/test.log.lzo" I am getting the > following error: > ERROR lzo.GPLNativeCodeLoader: Could not load native gpl > libraryjava.lang.UnsatisfiedLinkError: no gplcompression in > java.library.path > > > I will be grateful if someone could kindly help me in resolving the issue. > > > Thanks and Regards, > Gourav >
