Or if this jar is always going to be needed for all jobs, you can configure “tez.aux.uris” to point to the location of the jar on HDFS.
— Hitesh On Sep 1, 2015, at 4:17 AM, Rajesh Balamohan <[email protected]> wrote: > You can use DAG.addTaskLocalFiles() for this. > > Rough example: > > //Get the jar details > Path path = new Path("hdfs://nn:port/tmp/customJar.jar"); <-- Custom jar > path > FileSystem fs = path.getFileSystem(tezConf); > FileStatus fileStatus = fs.getFileStatus(path); > long size = fileStatus.getLen(); > long modTime = fileStatus.getModificationTime(); > > //Create local resource (ConveterUtils should be available in > org.apache.hadoop.yarn.util) > LocalResource localResource = > LocalResource.newInstance(ConverterUtils.getYarnUrlFromPath(path), > LocalResourceType.FILE, LocalResourceVisibility.APPLICATION, size, > modTime); > > //Add resources to DAG > Map<String, LocalResource> additionalResources = Maps.newHashMap(); > additionalResources.put("MyCustom.jar", localResource); > dag.addTaskLocalFiles(additionalResources); > System.out.println("Added jars to DAG"); > > > On Tue, Sep 1, 2015 at 3:13 PM, Sanjay Maurya <[email protected]> wrote: > Hi , > > I am running a DAG , but at run time Vertex is failing because it is not able > to load the class for my custom processor which is present in my jar > > My question is how to load the custom classes in Tez runtime . Is there any > similar API present in TEZ like we have in org.apache.hadoop.mapreduce.Job > (setJarByClass) . > > Below is the exception I am getting: > > 15/09/01 09:26:05 INFO examples.MyAggregator: DAG diagnostics: [Vertex > failed, vertexName=GroupBy, vertexId=vertex_1441084367785_0009_1_00, > diagnostics=[Task failed, taskId=task_1441084367785_0009_1_00_000000, > diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running > task:org.apache.tez.dag.api.TezUncheckedException: Unable to load class: > org.apache.tez.examples.MyAggregator$GroupByProcessor > at > org.apache.tez.common.ReflectionUtils.getClazz(ReflectionUtils.java:45) > at > org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:96) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.createProcessor(LogicalIOProcessorRuntimeTask.java:597) > at > org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.initialize(LogicalIOProcessorRuntimeTask.java:210) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) > at > org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) > at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.ClassNotFoundException: > org.apache.tez.examples.MyAggregator$GroupByProcessor > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:278) > at > org.apache.tez.common.ReflectionUtils.getClazz(ReflectionUtils.java:43) > ... 15 more > >
