Setting tez.use.cluster.hadoop-libs to true is one option. This will try picking up the classpath from the cluster itself - I believe a value configured in yarn-site.xml. The drawback to this though is a that job which are running during a rolling upgrade of the cluster can run into issues - 1) Different tasks running with different versions of the hadoop-client, 2) jars being replaced from under a running process. Alternately, if you're trying to use a custom version of hadoop-libraries, these can be place in HDFS and setup via 'tez.aux.uris'
The suggested deployment is with the regular package though. Is there a specific reason for not making use of this. If a custom version of hadoop is required, the package can be built by providing the -Dhadoop.version flag to maven. - Sid On Tue, Feb 2, 2016 at 10:19 PM, Rajat Jain <[email protected]> wrote: > In our current hive-on-tez deployment, we use the tez regular package > which has hadoop-* jars in tez/lib. > > When we switched to a tez-minimal and started getting errors while > launching containers, presumably because hadoop-* jars are not present in > the container launch classpath. How can this be fixed? > > Thanks, > Rajat > > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/hadoop/service/AbstractService > at java.lang.ClassLoader.defineClass1(Native Method) > at java.lang.ClassLoader.defineClass(ClassLoader.java:800) > at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) > at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) > at java.net.URLClassLoader.access$100(URLClassLoader.java:71) > at java.net.URLClassLoader$1.run(URLClassLoader.java:361) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.service.AbstractService > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > ... 13 more >
