Thanks for the information; We will check this and try to fix it in next release, a JIRA is created for tracking this: https://issues.apache.org/jira/browse/KYLIN-1507
ianzeng, please try the workaround that Pierre mentioned, it should work definitely. Thanks Pierre for the sharing, it does save our effort on investigate this issue. 2016-03-18 23:10 GMT+08:00 Pierre Gunet-Caplain <[email protected]>: > Hi, > > I also had this problem on a cloudera distribution. I found out there is a > regex that filters the dependencies to keep only a few of them (hive-exec, > hive-metastore and hive-hcatalog-core if I'm right), and the regex is > filtering out the cloudera jars names. > > So I used another way send the dependencies : in kylin.properties I > added kylin.job.mr.lib.dir=/local/directory/kylin/dep . In this folder I > created symbolic links to the cloudera jars enumerated above. These jars > are shipped to tmpjars at execution, and it works. > > I hope it help, > > Pierre > Data Architect at Deezer > > > > 2016-03-18 15:51 GMT+01:00 ianzeng <[email protected]>: > >> the find-hive-decency result is >> bin/find-hive-dependency.sh >> 16/03/18 18:09:26 WARN conf.HiveConf: DEPRECATED: Configuration property >> hive.metastore.local no longer has any effect. Make sure to provide a valid >> value for hive.metastore.uris if you are connecting to a remote metastore. >> >> Logging initialized using configuration in >> jar:file:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/jars/hive-common-0.13.1-cdh5.3.3.jar!/hive-log4j.properties >> HCAT_HOME not found, try to find hcatalog path from hadoop home >> hive dependency: >> /opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/lib/hive/conf:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-client.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-compress-1.4.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jline-0.9.94.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jpam-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-0.23.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-beeline.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-hwi.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/junit-4.11.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-cli.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-server.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/datanucleus-api-jdo-3.2.6.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/servlet-api-2.5.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-ant-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jersey-server-1.14.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/xz-1.0.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-exec-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/servlet-api-2.5-20081211.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-testutils-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/asm-commons-3.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/asm-3.2.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/htrace-core.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-cli-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-common.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/avro.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/libfb303-0.9.0.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/geronimo-jta_1.1_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-testutils.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-hwi-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-0.23-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/oro-2.0.8.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-httpclient-3.0.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/derby-10.10.1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/groovy-all-2.1.6.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-logging-1.1.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-hbase-handler.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hamcrest-core-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-jdbc-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/datanucleus-rdbms-3.2.9.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-service.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jersey-servlet-1.14.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/libthrift-0.9.0-cdh5-2.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/geronimo-annotation_1.0_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-contrib.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/httpclient-4.2.5.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/asm-tree-3.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-cli-1.2.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/zookeeper.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-jdbc.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-common.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-beeline-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-ant.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/guava-11.0.2.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jdo-api-3.0.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-protocol.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-common.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-metastore.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/activation-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/ant-launcher-1.9.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/paranamer-2.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-serde-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/geronimo-jaspic_1.0_spec-1.0.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jta-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-common-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-service-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/mail-1.4.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/antlr-2.7.7.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-lang-2.6.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-hadoop-compat.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-exec.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-codec-1.4.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hbase-hadoop2-compat.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-scheduler.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/log4j-1.2.16.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/ST4-4.0.4.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/antlr-runtime-3.4.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/stringtemplate-3.2.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/tempus-fugit-1.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/opencsv-2.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/snappy-java-1.0.4.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-common-secure.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jetty-all-7.6.0.v20120127.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-serde.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-metastore-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/stax-api-1.0.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-common-secure-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/velocity-1.5.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-contrib-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jsr305-1.3.9.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-hbase-handler-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/datanucleus-core-3.2.10.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-common-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/hive-shims-scheduler-0.13.1-cdh5.3.3.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/super-csv-2.2.0.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-lang3-3.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/httpcore-4.2.5.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jetty-util-6.1.26.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/commons-io-2.4.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/ant-1.9.1.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/bonecp-0.8.0.RELEASE.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive/lib/jetty-6.1.26.jar:/opt/cloudera/parcels/CDH-5.3.3-1.cdh5.3.3.p0.5/bin/../lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-0.13.1-cdh5.3.3.jar >> >> 在 2016年3月18日，22:33，ShaoFeng Shi <[email protected]> 写道： >> >> Run $KYLIN_HOME/bin/find-hive-dependency.sh to see whether it can locate >> the HCat jar files in the machine that running Kylin, please paste the >> output here for trouble shotting. >> >> 2016-03-18 18:28 GMT+08:00 ianzeng <[email protected]>: >> >>> HI all, >>> >>> >>> I has installed kylin 1.5 on redhead 6.3. I try build sample cube. But >>> got error msg as follow: >>> >>> >>> 2016-03-18 18:18:43,084 WARN [main] org.apache.hadoop.conf.Configuration: >>> job.xml:an attempt to override final parameter: hadoop.ssl.server.conf; >>> Ignoring. >>> 2016-03-18 18:18:43,093 WARN [main] org.apache.hadoop.conf.Configuration: >>> job.xml:an attempt to override final parameter: >>> mapreduce.job.end-notification.max.attempts; Ignoring. >>> 2016-03-18 18:18:43,509 INFO [main] >>> org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated. >>> Instead, use dfs.metrics.session-id >>> 2016-03-18 18:18:43,921 INFO [main] >>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output >>> Committer Algorithm version is 1 >>> 2016-03-18 18:18:43,933 INFO [main] org.apache.hadoop.mapred.Task: Using >>> ResourceCalculatorProcessTree : [ ] >>> 2016-03-18 18:18:44,120 WARN [main] org.apache.hadoop.mapred.YarnChild: >>> Exception running child : java.lang.RuntimeException: >>> java.lang.ClassNotFoundException: Class >>> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found >>> at >>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2047) >>> at >>> org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:184) >>> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:746) >>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) >>> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:415) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642) >>> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) >>> Caused by: java.lang.ClassNotFoundException: Class >>> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found >>> at >>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1953) >>> at >>> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2045) >>> ... 8 more >>> >>> >>> >>> What should I do to fix this problem ? >>> >> >> >> >> -- >> Best regards, >> >> Shaofeng Shi >> >> >> > -- Best regards, Shaofeng Shi
