Shaofeng SHI created KYLIN-2020:
-----------------------------------
Summary: NoClassDefFoundError:
org/apache/hive/hcatalog/mapreduce/HCatInputFormat on EMR 5.0.0
Key: KYLIN-2020
URL: https://issues.apache.org/jira/browse/KYLIN-2020
Project: Kylin
Issue Type: Bug
Components: Environment
Affects Versions: v1.5.4
Reporter: Shaofeng SHI
Assignee: Shaofeng SHI
Failed to build sample cube on a EMR 5.0 client node; classpath looks good, the
hive-hcatalog-core.jar can be found, but still got ClassNotFound error:
{code}
2016-09-16 14:37:34,944 INFO [pool-8-thread-1] common.MapReduceExecutable:113
: -conf /usr/local/apache-kylin-1.5.4-HBase1.x-bin/conf/kylin_job_conf.xml
-cubename kylin_sales_cube -output
/kylin/kylin_metadata/kylin-c3b85960-8a47-4b67-a7a5-145084f3d0cb/kylin_sales_cube/fact_distinct_columns
-segmentid 84761f2f-d69e-4810-b482-d87764d9b9db -statisticsenabled true
-statisticsoutput
/kylin/kylin_metadata/kylin-c3b85960-8a47-4b67-a7a5-145084f3d0cb/kylin_sales_cube/statistics
-statisticssamplingpercent 100 -jobname
Kylin_Fact_Distinct_Columns_kylin_sales_cube_Step -cubingJobId
c3b85960-8a47-4b67-a7a5-145084f3d0cb
2016-09-16 14:37:34,997 INFO [pool-8-thread-1] steps.FactDistinctColumnsJob:86
: Starting: Kylin_Fact_Distinct_Columns_kylin_sales_cube_Step
2016-09-16 14:37:34,997 INFO [pool-8-thread-1] common.AbstractHadoopJob:164 :
append job jar:
/usr/local/apache-kylin-1.5.4-HBase1.x-bin/lib/kylin-job-1.5.4.jar
2016-09-16 14:37:34,997 INFO [pool-8-thread-1] common.AbstractHadoopJob:172 :
append kylin.hbase.dependency: /usr/lib/hbase/bin/../lib/hbase-common-1.2.2.jar
to mapreduce.application.classpath
2016-09-16 14:37:34,997 INFO [pool-8-thread-1] common.AbstractHadoopJob:189 :
Hadoop job classpath is:
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*,
$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*,
/usr/lib/hadoop-lzo/lib/*,
/usr/share/aws/emr/emrfs/conf,
/usr/share/aws/emr/emrfs/lib/*,
/usr/share/aws/emr/emrfs/auxlib/*,
/usr/share/aws/emr/lib/*,
/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar,
/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar,
/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar,
/usr/share/aws/emr/cloudwatch-sink/lib/*
,/usr/lib/hbase/bin/../lib/hbase-common-1.2.2.jar
2016-09-16 14:37:34,998 INFO [pool-8-thread-1] common.AbstractHadoopJob:201 :
Hive Dependencies Before Filtered:
/usr/lib/hive/conf,/usr/lib/hive/lib/snappy-0.2.jar,/usr/lib/hive/lib/groovy-all-2.4.4.jar,/usr/lib/hive/lib/hive-llap-tez-2.1.0-amzn-0.jar,/usr/lib/hive/lib/jasper-compiler-5.5.23.jar,/usr/lib/hive/lib/guice-3.0.jar,/usr/lib/hive/lib/hive-service.jar,/usr/lib/hive/lib/jetty-sslengine-6.1.26.jar,/usr/lib/hive/lib/jersey-server-1.14.jar,/usr/lib/hive/lib/opencsv-2.3.jar,/usr/lib/hive/lib/commons-math-2.2.jar,/usr/lib/hive/lib/hive-shims-common.jar,/usr/lib/hive/lib/commons-lang3-3.1.jar,/usr/lib/hive/lib/hive-testutils.jar,/usr/lib/hive/lib/hive-jdbc-2.1.0-amzn-0.jar,/usr/lib/hive/lib/datanucleus-rdbms-4.1.7.jar,/usr/lib/hive/lib/junit-4.11.jar,/usr/lib/hive/lib/batik-ext-1.8.jar,/usr/lib/hive/lib/hive-accumulo-handler-2.1.0-amzn-0.jar,/usr/lib/hive/lib/antlr4-runtime-4.5.jar,/usr/lib/hive/lib/hive-service-rpc-2.1.0-amzn-0.jar,/usr/lib/hive/lib/twill-discovery-api-0.6.0-incubating.jar,/usr/lib/hive/lib/avro-1.7.7.jar,/usr/lib/hive/lib/bsh-core-2.0b4.jar,/usr/lib/hive/lib/datanucleus-api-jdo-4.2.1.jar,/usr/lib/hive/lib/hive-cli.jar,/usr/lib/hive/lib/hive-llap-server.jar,/usr/lib/hive/lib/log4j-slf4j-impl-2.4.1.jar,/usr/lib/hive/lib/javax.jdo-3.2.0-m3.jar,/usr/lib/hive/lib/twill-common-0.6.0-incubating.jar,/usr/lib/hive/lib/joni-2.1.2.jar,/usr/lib/hive/lib/maven-scm-provider-svn-commons-1.4.jar,/usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,/usr/lib/hive/lib/hive-beeline-2.1.0-amzn-0.jar,/usr/lib/hive/lib/accumulo-fate-1.6.0.jar,/usr/lib/hive/lib/tephra-hbase-compat-1.0-0.6.0.jar,/usr/lib/hive/lib/libfb303-0.9.3.jar,/usr/lib/hive/lib/hive-jdbc.jar,/usr/lib/hive/lib/disruptor-3.3.0.jar,/usr/lib/hive/lib/stringtemplate-3.2.1.jar,/usr/lib/hive/lib/jcodings-1.0.8.jar,/usr/lib/hive/lib/janino-2.7.6.jar,/usr/lib/hive/lib/commons-fileupload-1.3.1.jar,/usr/lib/hive/lib/plexus-utils-1.5.6.jar,/usr/lib/hive/lib/antlr-2.7.7.jar,/usr/lib/hive/lib/guava-14.0.1.jar,/usr/lib/hive/lib/hive-cli-2.1.0-amzn-0.jar,/usr/lib/hive/lib/metrics-core-3.1.0.jar,/usr/lib/hive/lib/jetty-all-7.6.0.v20120127.jar,/usr/lib/hive/lib/activation-1.1.jar,/usr/lib/hive/lib/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar,/usr/lib/hive/lib/parquet-hadoop-bundle-1.8.1.jar,/usr/lib/hive/lib/jackson-xc-1.9.2.jar,/usr/lib/hive/lib/guice-assistedinject-3.0.jar,/usr/lib/hive/lib/jdo-api-3.0.1.jar,/usr/lib/hive/lib/regexp-1.3.jar,/usr/lib/hive/lib/json-20090211.jar,/usr/lib/hive/lib/commons-codec-1.4.jar,/usr/lib/hive/lib/metrics-core-2.2.0.jar,/usr/lib/hive/lib/hive-storage-api-2.1.0-amzn-0.jar,/usr/lib/hive/lib/jackson-databind-2.4.2.jar,/usr/lib/hive/lib/datanucleus-core-4.1.6.jar,/usr/lib/hive/lib/ivy-2.4.0.jar,/usr/lib/hive/lib/xom-1.2.5.jar,/usr/lib/hive/lib/guice-servlet-3.0.jar,/usr/lib/hive/lib/curator-client-2.6.0.jar,/usr/lib/hive/lib/antisamy-1.5.3.jar,/usr/lib/hive/lib/hive-orc.jar,/usr/lib/hive/lib/jackson-jaxrs-1.9.2.jar,/usr/lib/hive/lib/ant-launcher-1.9.1.jar,/usr/lib/hive/lib/hive-orc-2.1.0-amzn-0.jar,/usr/lib/hive/lib/hive-llap-common.jar,/usr/lib/hive/lib/hive-shims.jar,/usr/lib/hive/lib/tempus-fugit-1.1.jar,/usr/lib/hive/lib/hive-hplsql-2.1.0-amzn-0.jar,/usr/lib/hive/lib/dropwizard-metrics-hadoop-metrics2-reporter-0.1.0.jar,/usr/lib/hive/lib/httpcore-4.4.jar,/usr/lib/hive/lib/servlet-api-2.5-6.1.14.jar,/usr/lib/hive/lib/gson-2.2.4.jar,/usr/lib/hive/lib/hive-service-rpc.jar,/usr/lib/hive/lib/hive-hbase-handler.jar,/usr/lib/hive/lib/hive-shims-common-2.1.0-amzn-0.jar,/usr/lib/hive/lib/geronimo-jaspic_1.0_spec-1.0.jar,/usr/lib/hive/lib/hive-accumulo-handler.jar,/usr/lib/hive/lib/jpam-1.1.jar,/usr/lib/hive/lib/hive-shims-scheduler-2.1.0-amzn-0.jar,/usr/lib/hive/lib/fastutil-6.5.6.jar,/usr/lib/hive/lib/commons-compiler-2.7.6.jar,/usr/lib/hive/lib/hive-serde.jar,/usr/lib/hive/lib/hive-hbase-handler-2.1.0-amzn-0.jar,/usr/lib/hive/lib/httpclient-4.4.jar,/usr/lib/hive/lib/hive-ant-2.1.0-amzn-0.jar,/usr/lib/hive/lib/zookeeper.jar,/usr/lib/hive/lib/accumulo-trace-1.6.0.jar,/usr/lib/hive/lib/hive-hplsql.jar,/usr/lib/hive/lib/geronimo-annotation_1.0_spec-1.1.1.jar,/usr/lib/hive/lib/bonecp-0.8.0.RELEASE.jar,/usr/lib/hive/lib/ST4-4.0.4.jar,/usr/lib/hive/lib/xml-apis-ext-1.3.04.jar,/usr/lib/hive/lib/xalan-2.7.0.jar,/usr/lib/hive/lib/jsp-2.1-6.1.14.jar,/usr/lib/hive/lib/jetty-6.1.26.jar,/usr/lib/hive/lib/antlr-runtime-3.4.jar,/usr/lib/hive/lib/metrics-json-3.1.0.jar,/usr/lib/hive/lib/ant-1.6.5.jar,/usr/lib/hive/lib/curator-framework-2.6.0.jar,/usr/lib/hive/lib/commons-io-2.4.jar,/usr/lib/hive/lib/twill-api-0.6.0-incubating.jar,/usr/lib/hive/lib/commons-cli-1.2.jar,/usr/lib/hive/lib/jetty-all-server-7.6.0.v20120127.jar,/usr/lib/hive/lib/libthrift-0.9.3.jar,/usr/lib/hive/lib/commons-httpclient-3.0.1.jar,/usr/lib/hive/lib/commons-logging-1.2.jar,/usr/lib/hive/lib/twill-core-0.6.0-incubating.jar,/usr/lib/hive/lib/log4j-core-2.4.1.jar,/usr/lib/hive/lib/jsp-api-2.1.jar,/usr/lib/hive/lib/hive-shims-scheduler.jar,/usr/lib/hive/lib/hive-shims-2.1.0-amzn-0.jar,/usr/lib/hive/lib/hive-common-2.1.0-amzn-0.jar,/usr/lib/hive/lib/commons-dbcp-1.4.jar,/usr/lib/hive/lib/hive-hwi.jar,/usr/lib/hive/lib/hive-llap-client-2.1.0-amzn-0.jar,/usr/lib/hive/lib/super-csv-2.2.0.jar,/usr/lib/hive/lib/maven-scm-api-1.4.jar,/usr/lib/hive/lib/jasper-runtime-5.5.23.jar,/usr/lib/hive/lib/jsp-api-2.1-6.1.14.jar,/usr/lib/hive/lib/velocity-1.5.jar,/usr/lib/hive/lib/hive-contrib-2.1.0-amzn-0.jar,/usr/lib/hive/lib/hive-llap-tez.jar,/usr/lib/hive/lib/jcommander-1.32.jar,/usr/lib/hive/lib/javax.servlet-3.0.0.v201112011016.jar,/usr/lib/hive/lib/esapi-2.1.0.1.jar,/usr/lib/hive/lib/jackson-annotations-2.4.0.jar,/usr/lib/hive/lib/mail-1.4.1.jar,/usr/lib/hive/lib/hive-llap-ext-client.jar,/usr/lib/hive/lib/hive-llap-server-2.1.0-amzn-0.jar,/usr/lib/hive/lib/twill-discovery-core-0.6.0-incubating.jar,/usr/lib/hive/lib/ant-1.9.1.jar,/usr/lib/hive/lib/jackson-core-2.4.2.jar,/usr/lib/hive/lib/nekohtml-1.9.16.jar,/usr/lib/hive/lib/joda-time-2.5.jar,/usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,/usr/lib/hive/lib/hive-llap-ext-client-2.1.0-amzn-0.jar,/usr/lib/hive/lib/hive-hwi-2.1.0-amzn-0.jar,/usr/lib/hive/lib/netty-all-4.0.23.Final.jar,/usr/lib/hive/lib/slider-core-0.90.2-incubating.jar,/usr/lib/hive/lib/derby-10.10.2.0.jar,/usr/lib/hive/lib/batik-util-1.8.jar,/usr/lib/hive/lib/log4j-web-2.4.1.jar,/usr/lib/hive/lib/batik-css-1.8.jar,/usr/lib/hive/lib/commons-compress-1.9.jar,/usr/lib/hive/lib/jamon-runtime-2.3.1.jar,/usr/lib/hive/lib/org.abego.treelayout.core-1.0.1.jar,/usr/lib/hive/lib/jta-1.1.jar,/usr/lib/hive/lib/javolution-5.5.1.jar,/usr/lib/hive/lib/accumulo-start-1.6.0.jar,/usr/lib/hive/lib/mariadb-connector-java.jar,/usr/lib/hive/lib/tephra-api-0.6.0.jar,/usr/lib/hive/lib/transaction-api-1.1.jar,/usr/lib/hive/lib/asm-commons-3.1.jar,/usr/lib/hive/lib/jline-2.12.jar,/usr/lib/hive/lib/jsr305-3.0.0.jar,/usr/lib/hive/lib/commons-vfs2-2.0.jar,/usr/lib/hive/lib/asm-tree-3.1.jar,/usr/lib/hive/lib/hive-exec.jar,/usr/lib/hive/lib/hive-service-2.1.0-amzn-0.jar,/usr/lib/hive/lib/hive-contrib.jar,/usr/lib/hive/lib/geronimo-jta_1.1_spec-1.1.1.jar,/usr/lib/hive/lib/hive-storage-api.jar,/usr/lib/hive/lib/jsp-api-2.0.jar,/usr/lib/hive/lib/hive-testutils-2.1.0-amzn-0.jar,/usr/lib/hive/lib/hamcrest-core-1.3.jar,/usr/lib/hive/lib/twill-zookeeper-0.6.0-incubating.jar,/usr/lib/hive/lib/maven-scm-provider-svnexe-1.4.jar,/usr/lib/hive/lib/protobuf-java-2.5.0.jar,/usr/lib/hive/lib/log4j-1.2-api-2.4.1.jar,/usr/lib/hive/lib/hive-ant.jar,/usr/lib/hive/lib/servlet-api-2.4.jar,/usr/lib/hive/lib/jetty-util-6.1.26.jar,/usr/lib/hive/lib/findbugs-annotations-1.3.9-1.jar,/usr/lib/hive/lib/hive-metastore.jar,/usr/lib/hive/lib/javax.inject-1.jar,/usr/lib/hive/lib/jersey-client-1.9.jar,/usr/lib/hive/lib/commons-pool-1.5.4.jar,/usr/lib/hive/lib/hive-beeline.jar,/usr/lib/hive/lib/htrace-core-3.1.0-incubating.jar,/usr/lib/hive/lib/snappy-java-1.0.5.jar,/usr/lib/hive/lib/hive-serde-2.1.0-amzn-0.jar,/usr/lib/hive/lib/hive-common.jar,/usr/lib/hive/lib/commons-collections-3.2.2.jar,/usr/lib/hive/lib/asm-3.1.jar,/usr/lib/hive/lib/commons-el-1.0.jar,/usr/lib/hive/lib/curator-recipes-2.6.0.jar,/usr/lib/hive/lib/paranamer-2.3.jar,/usr/lib/hive/lib/metrics-jvm-3.1.0.jar,/usr/lib/hive/lib/hive-llap-common-2.1.0-amzn-0.jar,/usr/lib/hive/lib/accumulo-core-1.6.0.jar,/usr/lib/hive/lib/stax-api-1.0.1.jar,/usr/lib/hive/lib/log4j-api-2.4.1.jar,/usr/lib/hive/lib/tephra-core-0.6.0.jar,/usr/lib/hive/lib/eigenbase-properties-1.1.5.jar,/usr/lib/hive/lib/aopalliance-1.0.jar,/usr/lib/hive/lib/commons-beanutils-core-1.8.3.jar,/usr/lib/hive/lib/commons-lang-2.6.jar,/usr/lib/hive/lib/hive-llap-client.jar,/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
2016-09-16 14:37:35,019 INFO [pool-8-thread-1] common.AbstractHadoopJob:203 :
Hive Dependencies After Filtered:
/usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,/usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
2016-09-16 14:37:35,019 INFO [pool-8-thread-1] common.AbstractHadoopJob:240 :
No Kafka dependency jars set in the environment, will find them from jvm:
2016-09-16 14:37:35,020 ERROR [pool-8-thread-1] common.AbstractHadoopJob:248 :
Cannot found kafka dependency jars: java.lang.ClassNotFoundException:
org.apache.kafka.clients.consumer.KafkaConsumer
2016-09-16 14:37:35,021 INFO [pool-8-thread-1] common.AbstractHadoopJob:371 :
Job 'tmpjars' updated --
file:/usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,file:/usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,file:/usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
2016-09-16 14:37:35,022 INFO [pool-8-thread-1]
steps.FactDistinctColumnsJob:100 : Found segment:
kylin_sales_cube[20120101000000_20160829000000]
2016-09-16 14:37:35,028 INFO [pool-8-thread-1] common.AbstractHadoopJob:510 :
tempMetaFileString is : null
2016-09-16 14:37:35,030 ERROR [pool-8-thread-1]
execution.AbstractExecutable:114 : error running Executable:
MapReduceExecutable{id=c3b85960-8a47-4b67-a7a5-145084f3d0cb-02, name=Extract
Fact Table Distinct Columns, state=RUNNING}
2016-09-16 14:37:35,034 DEBUG [pool-8-thread-1] hbase.HBaseResourceStore:262 :
Update row /execute_output/c3b85960-8a47-4b67-a7a5-145084f3d0cb-02 from oldTs:
1474036654866, to newTs: 1474036655032, operation result: true
2016-09-16 14:37:35,037 DEBUG [pool-8-thread-1] hbase.HBaseResourceStore:262 :
Update row /execute_output/c3b85960-8a47-4b67-a7a5-145084f3d0cb-02 from oldTs:
1474036655032, to newTs: 1474036655036, operation result: true
2016-09-16 14:37:35,037 INFO [pool-8-thread-1] manager.ExecutableManager:292 :
job id:c3b85960-8a47-4b67-a7a5-145084f3d0cb-02 from RUNNING to ERROR
2016-09-16 14:37:35,039 ERROR [pool-8-thread-1]
execution.AbstractExecutable:114 : error running Executable:
CubingJob{id=c3b85960-8a47-4b67-a7a5-145084f3d0cb, name=kylin_sales_cube -
20120101000000_20160829000000 - BUILD - GMT+08:00 2016-09-16 22:07:02,
state=RUNNING}
2016-09-16 14:37:35,043 DEBUG [pool-8-thread-1] hbase.HBaseResourceStore:262 :
Update row /execute_output/c3b85960-8a47-4b67-a7a5-145084f3d0cb from oldTs:
1474036654855, to newTs: 1474036655041, operation result: true
2016-09-16 14:37:35,046 DEBUG [pool-8-thread-1] hbase.HBaseResourceStore:262 :
Update row /execute_output/c3b85960-8a47-4b67-a7a5-145084f3d0cb from oldTs:
1474036655041, to newTs: 1474036655044, operation result: true
2016-09-16 14:37:35,046 INFO [pool-8-thread-1] manager.ExecutableManager:292 :
job id:c3b85960-8a47-4b67-a7a5-145084f3d0cb from RUNNING to ERROR
2016-09-16 14:37:35,046 WARN [pool-8-thread-1]
execution.AbstractExecutable:222 : no need to send email, user list is empty
2016-09-16 14:37:35,047 ERROR [pool-8-thread-1] threadpool.DefaultScheduler:140
: ExecuteException job:c3b85960-8a47-4b67-a7a5-145084f3d0cb
org.apache.kylin.job.exception.ExecuteException:
org.apache.kylin.job.exception.ExecuteException:
java.lang.NoClassDefFoundError:
org/apache/hive/hcatalog/mapreduce/HCatInputFormat
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:122)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.kylin.job.exception.ExecuteException:
java.lang.NoClassDefFoundError:
org/apache/hive/hcatalog/mapreduce/HCatInputFormat
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:122)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:112)
... 4 more
Caused by: java.lang.NoClassDefFoundError:
org/apache/hive/hcatalog/mapreduce/HCatInputFormat
at
org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:89)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:123)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:103)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:88)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:120)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:112)
... 6 more
Caused by: java.lang.ClassNotFoundException:
org.apache.hive.hcatalog.mapreduce.HCatInputFormat
at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1858)
at
org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1701)
... 12 more
2016-09-16 14:38:34,845 INFO [pool-7-thread-1] threadpool.DefaultScheduler:118
: Job Fetcher: 0 should running, 0 actual running, 0 ready, 0 already succeed,
1 error, 0 discarded, 0 others
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
