By default, native libs are expected at $HADOOP_COMMON_HOME/lib/native/ for linux and added to the AM/task launch env via "LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_COMMON_HOME/lib/native/". For Windows the PATH is updated with the location instead of LD_LIBRARY_PATH.
If your native libs are in a different location, then the AM/task env would need to be overridden with the location of the native lib. *From:* Siddharth Seth [mailto:[email protected]] *Sent:* Monday, November 17, 2014 12:15 PM *To:* [email protected] *Subject:* Re: Native Compression Lib loading failed Are these direct Tez jobs ? The task and am env do need to be setup to make use of Snappy. The property that needs to be set depends on the job being run. Tez direct would require tez.am.launch.env and tez.task.launch.env to be set. For Hive, Pig, MR, etc this may be different, depending on how these projects setup the env. On Mon, Nov 17, 2014 at 4:54 AM, Tsuyoshi OZAWA <[email protected]> wrote: Hi Hitesh, I'm facing with same problem in Apache version. Can you run jobs with snappy compression with 0.5.2 release? Thanks, - Tsuyoshi On Mon, Nov 17, 2014 at 9:26 PM, Hitesh Shah <[email protected]> wrote: > Hi Subroto, > > It could be an installer/distro issue. You may need to redirect your question to the Hortonworks forums to get an answer on this. > > But yes, you are right - the env needs to be setup correctly to load the required native libs. Out of curiosity, are you using Ambari to setup your cluster? > > thanks > — Hitesh > > On Nov 17, 2014, at 11:07 AM, Subroto Sanyal <[email protected]> wrote: > >> Hi, >> >> With HDP-2.1.2 we are encountering the problem of: >> 2014-09-11 10:14:42,881 ERROR [TezChild] org.apache.tez.runtime.task.TezTaskRunner: Exception of type Error. Exiting now >> java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z >> at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method) >> at org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63) >> at org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132) >> at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) >> at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) >> at org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1198) >> at org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1091) >> at org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1441) >> at org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:274) >> >> >> >> To workaround the problem we have to set the properties explicitly: >> tez.am.launch.env >> >> tez.task.launch.env >> >> >> >> Is this a problem with HDP installation? >> >> -- >> Cheers, >> Subroto Sanyal > -- - Tsuyoshi -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
