Hi ,
I am running a DAG , but at run time Vertex is failing because it is not able
to load the class for my custom processor which is present in my jar
My question is how to load the custom classes in Tez runtime . Is there any
similar API present in TEZ like we have in org.apache.hadoop.mapreduce.Job
(setJarByClass) .
Below is the exception I am getting:
15/09/01 09:26:05 INFO examples.MyAggregator: DAG diagnostics: [Vertex failed,
vertexName=GroupBy, vertexId=vertex_1441084367785_0009_1_00, diagnostics=[Task
failed, taskId=task_1441084367785_0009_1_00_000000, diagnostics=[TaskAttempt 0
failed, info=[Error: Failure while running
task:org.apache.tez.dag.api.TezUncheckedException: Unable to load class:
org.apache.tez.examples.MyAggregator$GroupByProcessor at
org.apache.tez.common.ReflectionUtils.getClazz(ReflectionUtils.java:45)
at
org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:96)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.createProcessor(LogicalIOProcessorRuntimeTask.java:597)
at
org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.initialize(LogicalIOProcessorRuntimeTask.java:210)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:176)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
at java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
at
org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
at java.util.concurrent.FutureTask.run(FutureTask.java:262) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)Caused by:
java.lang.ClassNotFoundException:
org.apache.tez.examples.MyAggregator$GroupByProcessor at
java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at
java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
java.lang.Class.forName0(Native Method) at
java.lang.Class.forName(Class.java:278) at
org.apache.tez.common.ReflectionUtils.getClazz(ReflectionUtils.java:43)
... 15 more
