Georgi Ivanov created OOZIE-2643: ------------------------------------ Summary: Hive on Tez vai Oozie fails with ClassNotFound Exception against tables with Hcatalog Json Serde Key: OOZIE-2643 URL: https://issues.apache.org/jira/browse/OOZIE-2643 Project: Oozie Issue Type: Bug Reporter: Georgi Ivanov
If we create an oozie workflow that has a hive action using tez execution engine and we reference a table with hcatalog json serde, oozie does not localize properly the sharelib jars for the tez session. It localizes them for the Hive Action but they do not propagate to Tez. Using Hive on MR works fine. This problem is only present with Oozie and Tez. The workflow throws ClassNotFound Exception. 2016-08-18 08:35:00,337 [ERROR] [TezChild] |tez.TezProcessor|: java.lang.RuntimeException: Map operator initialization failed at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:265) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:149) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:139) at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:344) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:181) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:172) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:172) at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:168) at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:347) at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:382) at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:227) ... 15 more Caused by: java.lang.ClassNotFoundException: Class org.apache.hive.hcatalog.data.JsonSerDe not found at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101) at org.apache.hadoop.hive.ql.plan.PartitionDesc.getDeserializer(PartitionDesc.java:143) at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:313) ... 17 more The only workaround that I've found so far is to add the hcatalog sharelib to tez.aux.uris as a configuration parameter inside the workflow.xml <property> <name>tez.aux.uris</name> <value>${nameNode}/user/oozie/share/lib/hcatalog/</value> </property> -- This message was sent by Atlassian JIRA (v6.3.4#6332)