“tez.aux.uris” supports a comma separated list of files and directories on HDFS
or any distributed filesystem ( no tar balls, archives, etc and no support for
file:// ). When Hive submits a query to Tez, it adds its hive-exec.jar to the
tez runtime ( similar to MR distributed cache ). If you are hitting a class not
found again, you are missing some jars from the classpath. If you are using
your own InputFormat, try doing an “add jar “ in your hive script or add that
custom jar to tez.aux.uris after uploading it to HDFS.
If you are modifying the hive-exec.jar, you may need to ensure that Hive is
using that jar and not overriding your custom hive-exec.jar.
— Hitesh
On Sep 11, 2015, at 12:56 AM, Raajay <raaja...@gmail.com> wrote:
> Yeah. I added the hive-exec.jar that contains HiveSpltGenerator to HDFS. I
> still hit the exception
>
> On Fri, Sep 11, 2015 at 2:43 AM, Jianfeng (Jeff) Zhang
> <jzh...@hortonworks.com> wrote:
>
> Have you try using jar rather than tar.gz ?
>
>
> Best Regard,
> Jeff Zhang
>
>
> From: Raajay <raaja...@gmail.com>
> Reply-To: "user@tez.apache.org" <user@tez.apache.org>
> Date: Friday, September 11, 2015 at 3:15 PM
> To: "user@tez.apache.org" <user@tez.apache.org>
> Subject: Missing libraries.
>
> I am running DAGs generated by Hive for Tez in offline mode; as in I store
> the DAGs to disk and then run them later using my own Tez Client.
>
> I have been able to get this setup going in local mode. However, while
> running on the cluster, I hit Processor class not found exception (snippet
> below). I figure this is because, custom processor classes defined in Hive
> (eg: HiveSplitGenerator) is not visible while executing a mapper.
>
> I have uploaded, hive exec jar (apache-hive-2.0.0-SNAPSHOT-bin.tar.gz) to
> HDFS and pointed ${tez.aux.uris} to that location. Not sure what more is
> needed to make hive Classes visible to tez tasks ? "tar.gz" does not work ?
>
>
> 2015-09-11 00:59:02,973 INFO [Dispatcher thread: Central] impl.VertexImpl:
> Recovered Vertex State, vertexId=vertex_1441949856963_0006_1_02 [Map 1],
> state=NEW, numInitedSourceVertices=0, numStartedSourceVertices=0,
> numRecoveredSourceVertices=0, recoveredEvents=0, tasksIsNull=false, numTasks=0
> 2015-09-11 00:59:02,974 INFO [Dispatcher thread: Central] impl.VertexImpl:
> Root Inputs exist for Vertex: Map 4 : {a={InputName=a},
> {Descriptor=ClassName=org.apache.tez.mapreduce.input.MRInputLegacy,
> hasPayload=true},
> {ControllerDescriptor=ClassName=org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator,
> hasPayload=false}}
> 2015-09-11 00:59:02,974 INFO [Dispatcher thread: Central] impl.VertexImpl:
> Starting root input initializer for input: a, with class:
> [org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator]
> 2015-09-11 00:59:02,974 INFO [Dispatcher thread: Central] impl.VertexImpl:
> Setting vertexManager to RootInputVertexManager for
> vertex_1441949856963_0006_1_00 [Map 4]
> 2015-09-11 00:59:02,979 INFO [Dispatcher thread: Central] impl.VertexImpl:
> Num tasks is -1. Expecting VertexManager/InputInitializers/1-1 split to set
> #tasks for the vertex vertex_1441949856963_0006_1_00 [Map 4]
> 2015-09-11 00:59:02,979 INFO [Dispatcher thread: Central] impl.VertexImpl:
> Vertex will initialize from input initializer. vertex_1441949856963_0006_1_00
> [Map 4]
> 2015-09-11 00:59:02,980 INFO [Dispatcher thread: Central] impl.VertexImpl:
> Vertex will initialize via inputInitializers vertex_1441949856963_0006_1_00
> [Map 4]. Starting root input initializers: 1
> 2015-09-11 00:59:02,981 ERROR [Dispatcher thread: Central]
> common.AsyncDispatcher: Error in dispatcher thread
> org.apache.tez.dag.api.TezUncheckedException: Unable to load class:
> org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
> at
> org.apache.tez.common.ReflectionUtils.getClazz(ReflectionUtils.java:45)
> at
> org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:96)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:137)
> at
> org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:114)
>
>
>
