[
https://issues.apache.org/jira/browse/KYLIN-1082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI reopened KYLIN-1082:
---------------------------------
I deployed a latest build from 1.x-staging, create a new cube and then build
it; The job failed at the second step (fact distinct), all mapper has this
error:
{code}
Error: java.lang.ClassNotFoundException:
org.apache.hadoop.hive.ql.metadata.HiveStorageHandler at
java.net.URLClassLoader$1.run(URLClassLoader.java:366) at
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at
java.security.AccessController.doPrivileged(Native Method) at
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at
java.lang.ClassLoader.loadClass(ClassLoader.java:425) at
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at
java.lang.ClassLoader.loadClass(ClassLoader.java:358) at
org.apache.hive.hcatalog.mapreduce.HCatSplit.readFields(HCatSplit.java:139) at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:71)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableDeserializer.deserialize(WritableSerialization.java:42)
at org.apache.hadoop.mapred.MapTask.getSplitDetails(MapTask.java:371) at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:731) at
org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
{code}
It seems some hive dependencies are missed. Yanghong, could you please double
check this? For testing this, sandbox vm is not a good choice as it only has 1
node and all jars exists in that. You'd better selecting a clustered env where
the hadoop nodes don't have hive installed.
> Hive dependencies should be add to tmpjars
> ------------------------------------------
>
> Key: KYLIN-1082
> URL: https://issues.apache.org/jira/browse/KYLIN-1082
> Project: Kylin
> Issue Type: Bug
> Components: Environment , Job Engine
> Reporter: liyang
> Assignee: Zhong Yanghong
> Labels: newbie
> Fix For: v2.1, v1.3
>
> Attachments: auto_hive_tmpjars_1_x_staging.patch,
> auto_hive_tmpjars_2_x_staging.patch
>
>
> Currently kylin assume all data nodes have hive deployment at exact same FS
> location. However, a better position is to think hive as a client side app.
> Then we need to ship hive jar with MR job every time.
> This make deploy kylin a lot easier in cluster that does not have hive on all
> data nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
