Are these direct Tez jobs ? The task and am env do need to be setup to make use of Snappy. The property that needs to be set depends on the job being run. Tez direct would require tez.am.launch.env and tez.task.launch.env to be set. For Hive, Pig, MR, etc this may be different, depending on how these projects setup the env.
On Mon, Nov 17, 2014 at 4:54 AM, Tsuyoshi OZAWA <[email protected]> wrote: > Hi Hitesh, > > I'm facing with same problem in Apache version. Can you run jobs with > snappy compression with 0.5.2 release? > > Thanks, > - Tsuyoshi > > > On Mon, Nov 17, 2014 at 9:26 PM, Hitesh Shah <[email protected]> wrote: > > Hi Subroto, > > > > It could be an installer/distro issue. You may need to redirect your > question to the Hortonworks forums to get an answer on this. > > > > But yes, you are right - the env needs to be setup correctly to load the > required native libs. Out of curiosity, are you using Ambari to setup your > cluster? > > > > thanks > > — Hitesh > > > > On Nov 17, 2014, at 11:07 AM, Subroto Sanyal <[email protected]> > wrote: > > > >> Hi, > >> > >> With HDP-2.1.2 we are encountering the problem of: > >> 2014-09-11 10:14:42,881 ERROR [TezChild] > org.apache.tez.runtime.task.TezTaskRunner: Exception of type Error. Exiting > now > >> java.lang.UnsatisfiedLinkError: > org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z > >> at > org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method) > >> at > org.apache.hadoop.io.compress.SnappyCodec.checkNativeCodeLoaded(SnappyCodec.java:63) > >> at > org.apache.hadoop.io.compress.SnappyCodec.getCompressorType(SnappyCodec.java:132) > >> at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148) > >> at > org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163) > >> at > org.apache.hadoop.io.SequenceFile$Writer.init(SequenceFile.java:1198) > >> at > org.apache.hadoop.io.SequenceFile$Writer.<init>(SequenceFile.java:1091) > >> at > org.apache.hadoop.io.SequenceFile$BlockCompressWriter.<init>(SequenceFile.java:1441) > >> at > org.apache.hadoop.io.SequenceFile.createWriter(SequenceFile.java:274) > >> > >> > >> > >> To workaround the problem we have to set the properties explicitly: > >> tez.am.launch.env > >> > >> tez.task.launch.env > >> > >> > >> > >> Is this a problem with HDP installation? > >> > >> -- > >> Cheers, > >> Subroto Sanyal > > > > > > -- > - Tsuyoshi >
