Have you try using jar rather than tar.gz ?
Best Regard, Jeff Zhang From: Raajay <[email protected]<mailto:[email protected]>> Reply-To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Date: Friday, September 11, 2015 at 3:15 PM To: "[email protected]<mailto:[email protected]>" <[email protected]<mailto:[email protected]>> Subject: Missing libraries. I am running DAGs generated by Hive for Tez in offline mode; as in I store the DAGs to disk and then run them later using my own Tez Client. I have been able to get this setup going in local mode. However, while running on the cluster, I hit Processor class not found exception (snippet below). I figure this is because, custom processor classes defined in Hive (eg: HiveSplitGenerator) is not visible while executing a mapper. I have uploaded, hive exec jar (apache-hive-2.0.0-SNAPSHOT-bin.tar.gz) to HDFS and pointed ${tez.aux.uris} to that location. Not sure what more is needed to make hive Classes visible to tez tasks ? "tar.gz" does not work ? 2015-09-11 00:59:02,973 INFO [Dispatcher thread: Central] impl.VertexImpl: Recovered Vertex State, vertexId=vertex_1441949856963_0006_1_02 [Map 1], state=NEW, numInitedSourceVertices=0, numStartedSourceVertices=0, numRecoveredSourceVertices=0, recoveredEvents=0, tasksIsNull=false, numTasks=0 2015-09-11 00:59:02,974 INFO [Dispatcher thread: Central] impl.VertexImpl: Root Inputs exist for Vertex: Map 4 : {a={InputName=a}, {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInputLegacy, hasPayload=true}, {ControllerDescriptor=ClassName=org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator, hasPayload=false}} 2015-09-11 00:59:02,974 INFO [Dispatcher thread: Central] impl.VertexImpl: Starting root input initializer for input: a, with class: [org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator] 2015-09-11 00:59:02,974 INFO [Dispatcher thread: Central] impl.VertexImpl: Setting vertexManager to RootInputVertexManager for vertex_1441949856963_0006_1_00 [Map 4] 2015-09-11 00:59:02,979 INFO [Dispatcher thread: Central] impl.VertexImpl: Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set #tasks for the vertex vertex_1441949856963_0006_1_00 [Map 4] 2015-09-11 00:59:02,979 INFO [Dispatcher thread: Central] impl.VertexImpl: Vertex will initialize from input initializer. vertex_1441949856963_0006_1_00 [Map 4] 2015-09-11 00:59:02,980 INFO [Dispatcher thread: Central] impl.VertexImpl: Vertex will initialize via inputInitializers vertex_1441949856963_0006_1_00 [Map 4]. Starting root input initializers: 1 2015-09-11 00:59:02,981 ERROR [Dispatcher thread: Central] common.AsyncDispatcher: Error in dispatcher thread org.apache.tez.dag.api.TezUncheckedException: Unable to load class: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator at org.apache.tez.common.ReflectionUtils.getClazz(ReflectionUtils.java:45) at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:96) at org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:137) at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:114)
