Thanks, Sid. Setting this parameter solves this problem. We usually do not care about rolling deploys on our end. In our cloud based model, we just do a rolling restart of the cluster <https://www.qubole.com/blog/product/drag-n-drop-upgrades-of-hadoop-spark-and-presto-clusters/> with the updated libraries -- it has given us a far smoother experience. Since our hadoop and tez libraries are tightly linked in an Amazon AMI, we like to keep the tez library lean.
On 3 February 2016 at 12:27, Siddharth Seth <[email protected]> wrote: > Setting tez.use.cluster.hadoop-libs to true is one option. This will try > picking up the classpath from the cluster itself - I believe a value > configured in yarn-site.xml. The drawback to this though is a that job > which are running during a rolling upgrade of the cluster can run into > issues - 1) Different tasks running with different versions of the > hadoop-client, 2) jars being replaced from under a running process. > Alternately, if you're trying to use a custom version of hadoop-libraries, > these can be place in HDFS and setup via 'tez.aux.uris' > > The suggested deployment is with the regular package though. Is there a > specific reason for not making use of this. If a custom version of hadoop > is required, the package can be built by providing the -Dhadoop.version > flag to maven. > > - Sid > > > > > > On Tue, Feb 2, 2016 at 10:19 PM, Rajat Jain <[email protected]> wrote: > >> In our current hive-on-tez deployment, we use the tez regular package >> which has hadoop-* jars in tez/lib. >> >> When we switched to a tez-minimal and started getting errors while >> launching containers, presumably because hadoop-* jars are not present in >> the container launch classpath. How can this be fixed? >> >> Thanks, >> Rajat >> >> Exception in thread "main" java.lang.NoClassDefFoundError: >> org/apache/hadoop/service/AbstractService >> at java.lang.ClassLoader.defineClass1(Native Method) >> at java.lang.ClassLoader.defineClass(ClassLoader.java:800) >> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) >> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) >> at java.net.URLClassLoader.access$100(URLClassLoader.java:71) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:361) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482) >> Caused by: java.lang.ClassNotFoundException: >> org.apache.hadoop.service.AbstractService >> at java.net.URLClassLoader$1.run(URLClassLoader.java:366) >> at java.net.URLClassLoader$1.run(URLClassLoader.java:355) >> at java.security.AccessController.doPrivileged(Native Method) >> at java.net.URLClassLoader.findClass(URLClassLoader.java:354) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:425) >> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) >> at java.lang.ClassLoader.loadClass(ClassLoader.java:358) >> ... 13 more >> > >
