You're correct; this issue was also reported by another user in a couple
days ago, the fix will be released in next version. You can find a
workaround in the JIRA: https://issues.apache.org/jira/browse/KYLIN-1507
2016-03-24 11:12 GMT+08:00 sdangi <sda...@datalenz.com>:
> */private static final String KYLIN_HIVE_DEPENDENCY_JARS =
>
> "[^,]*hive-exec[0-9.-]+\\.jar|[^,]*hive-metastore[0-9.-]+\\.jar|[^,]*hive-hcatalog-core[0-9.-]+\\.jar";/
> *
> String filterKylinHiveDependency(String kylinHiveDependency) {
> if (StringUtils.isBlank(kylinHiveDependency))
> return "";
>
> StringBuilder jarList = new StringBuilder();
>
> Pattern hivePattern = Pattern.compile(KYLIN_HIVE_DEPENDENCY_JARS);
> Matcher matcher = hivePattern.matcher(kylinHiveDependency);
>
> while (matcher.find()) {
> if (jarList.length() > 0)
> jarList.append(",");
> jarList.append(matcher.group());
> }
>
> This logic doesn't work for CDH5.5.2 as the reg ex not is not matched.
> Hive Dependencies Before Filtered:
>
> /opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/lib/hive/conf:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-shims-scheduler.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-metastore.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-serde-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jcommander-1.32.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-testutils-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-hbase-handler-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-beeline-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jackson-core-2.2.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hamcrest-core-1.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/htrace-core.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/maven-scm-api-1.4.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-service-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/parquet-hadoop-bundle.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/accumulo-start-1.6.0.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-shims-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-ant-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jta-1.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-testutils.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/log4j-1.2.16.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/guava-14.0.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/libthrift-0.9.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-shims-0.23.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/velocity-1.5.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jackson-jaxrs-1.9.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-contrib.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-math-2.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/paranamer-2.3.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/maven-scm-provider-svnexe-1.4.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hbase-client.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/high-scale-lib-1.1.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-cli-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-accumulo-handler-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-compiler-2.7.6.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/avro.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/xz-1.0.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-service.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/httpclient-4.2.5.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/metrics-core-3.0.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/curator-client-2.6.0.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/curator-framework-2.6.0.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-shims-scheduler-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/oro-2.0.8.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/asm-3.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jetty-all-7.6.0.v20120127.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-shims-common-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jackson-databind-2.2.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/datanucleus-core-3.2.10.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jetty-all-server-7.6.0.v20120127.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-exec.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-dbcp-1.4.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/geronimo-jaspic_1.0_spec-1.0.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-ant.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-lang-2.6.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-cli.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/accumulo-core-1.6.0.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/bonecp-0.8.0.RELEASE.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-cli-1.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/stringtemplate-3.2.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jline-2.12.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/activation-1.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jsr305-3.0.0.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/servlet-api-2.5.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hbase-common.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/httpcore-4.2.5.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-httpclient-3.0.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hbase-hadoop2-compat.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jersey-server-1.14.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/eigenbase-properties-1.1.4.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-contrib-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jersey-servlet-1.14.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/tempus-fugit-1.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-hwi-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-common.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/antlr-2.7.7.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-beeline.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-logging-1.1.3.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jdo-api-3.0.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-vfs2-2.0.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-compress-1.4.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-jdbc.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/groovy-all-2.4.4.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hbase-hadoop-compat.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/libfb303-0.9.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/geronimo-jta_1.1_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/datanucleus-rdbms-3.2.9.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/plexus-utils-1.5.6.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-shims-0.23-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-digester-1.8.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/maven-scm-provider-svn-commons-1.4.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-io-2.4.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-beanutils-1.7.0.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/ant-launcher-1.9.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-configuration-1.6.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-hbase-handler.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-beanutils-core-1.8.0.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/gson-2.2.4.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jackson-xc-1.9.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/snappy-java-1.0.4.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/apache-log4j-extras-1.2.17.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-serde.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-jdbc-standalone.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/ST4-4.0.4.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-accumulo-handler.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/geronimo-annotation_1.0_spec-1.1.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-pool-1.5.4.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/accumulo-trace-1.6.0.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/regexp-1.3.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-metastore-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/mail-1.4.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/accumulo-fate-1.6.0.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-jdbc-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/janino-2.7.6.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/derby-10.11.1.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/super-csv-2.2.0.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hbase-protocol.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-shims.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/asm-commons-3.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jackson-annotations-2.2.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hbase-server.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-exec-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/stax-api-1.0.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/ant-1.9.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-shims-common.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/logredactor-1.0.3.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/opencsv-2.3.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-common-1.1.0-cdh5.5.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/zookeeper.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/junit-4.11.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-jdbc-1.1.0-cdh5.5.2-standalone.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/commons-codec-1.4.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/metrics-json-3.0.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/antlr-runtime-3.4.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/datanucleus-api-jdo-3.2.6.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/asm-tree-3.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/hive-hwi.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/metrics-jvm-3.0.2.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive/lib/jpam-1.1.jar:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/bin/../lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-1.1.0-cdh5.5.2.jar
> kylin.log:2016-03-23 14:26:29,165 INFO [pool-5-thread-2]
> common.AbstractHadoopJob:215 : Hive Dependencies After Filtered:
>
> This caused ClassNotFoundErrors:
> User: hdfs
> Name: Kylin_Fact_Distinct_Columns_test_Step
> Application Type: MAPREDUCE
> Application Tags:
> State: FINISHED
> FinalStatus: FAILED
>
>
> 2016-03-23 14:26:50,070 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1458582499641_0019_m_000000_2: Error:
> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199)
> at
>
> org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:184)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
> at
>
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)
> ... 8 more
>
> 2016-03-23 14:26:50,072 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1458582499641_0019_m_000000_2 TaskAttempt Transitioned from RUNNING
> to FAIL_FINISHING_CONTAINER
> 2016-03-23 14:26:50,073 INFO [Thread-51]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: 2 failures on
> node sd-ae49-cf90.nam.nsroot.net
> 2016-03-23 14:26:50,074 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1458582499641_0019_m_000000_3 TaskAttempt Transitioned from NEW to
> UNASSIGNED
> 2016-03-23 14:26:50,074 INFO [Thread-51]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Added
> attempt_1458582499641_0019_m_000000_3 to list of failed maps
> 2016-03-23 14:26:50,459 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Before
> Scheduling: PendingReds:2 ScheduledMaps:1 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:3 ContRel:0
> HostLocal:1 RackLocal:0
> 2016-03-23 14:26:50,462 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1458582499641_0019: ask=1 release= 0 newContainers=0
> finishedContainers=0 resourcelimit=<memory:223232, vCores:21> knownNMs=4
> 2016-03-23 14:26:50,462 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=<memory:223232, vCores:21>
> 2016-03-23 14:26:50,462 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow
> start threshold not met. completedMapsForReduceSlowstart 1
> 2016-03-23 14:26:51,467 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Received
> completed container container_1458582499641_0019_01_000004
> 2016-03-23 14:26:51,467 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Got allocated
> containers 1
> 2016-03-23 14:26:51,467 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigning
> container Container: [ContainerId: container_1458582499641_0019_01_000005,
> NodeId: sd-d7e7-2a0a.nam.nsroot.net:8041, NodeHttpAddress:
> sd-d7e7-2a0a.nam.nsroot.net:8042, Resource: <memory:4096, vCores:6>,
> Priority: 5, Token: Token { kind: ContainerToken, service:
> 169.172.134.175:8041 }, ] to fast fail map
> 2016-03-23 14:26:51,467 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned from
> earlierFailedMaps
> 2016-03-23 14:26:51,467 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1458582499641_0019_m_000000_2 TaskAttempt Transitioned from
> FAIL_FINISHING_CONTAINER to FAILED
> 2016-03-23 14:26:51,467 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Assigned
> container container_1458582499641_0019_01_000005 to
> attempt_1458582499641_0019_m_000000_3
> 2016-03-23 14:26:51,467 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Recalculating
> schedule, headroom=<memory:223232, vCores:21>
> 2016-03-23 14:26:51,467 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1458582499641_0019_m_000000_2:
> 2016-03-23 14:26:51,467 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Reduce slow
> start threshold not met. completedMapsForReduceSlowstart 1
> 2016-03-23 14:26:51,468 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: After
> Scheduling: PendingReds:2 ScheduledMaps:0 ScheduledReds:0 AssignedMaps:1
> AssignedReds:0 CompletedMaps:0 CompletedReds:0 ContAlloc:4 ContRel:0
> HostLocal:1 RackLocal:0
> 2016-03-23 14:26:51,468 INFO [ContainerLauncher #5]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Processing the event EventType: CONTAINER_COMPLETED for container
> container_1458582499641_0019_01_000004 taskAttempt
> attempt_1458582499641_0019_m_000000_2
> 2016-03-23 14:26:51,470 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1458582499641_0019_m_000000_3 TaskAttempt Transitioned from
> UNASSIGNED to ASSIGNED
> 2016-03-23 14:26:51,470 INFO [ContainerLauncher #6]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Processing the event EventType: CONTAINER_REMOTE_LAUNCH for container
> container_1458582499641_0019_01_000005 taskAttempt
> attempt_1458582499641_0019_m_000000_3
> 2016-03-23 14:26:51,470 INFO [ContainerLauncher #6]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl:
> Launching
> attempt_1458582499641_0019_m_000000_3
> 2016-03-23 14:26:51,470 INFO [ContainerLauncher #6]
> org.apache.hadoop.yarn.client.api.impl.ContainerManagementProtocolProxy:
> Opening proxy : sd-d7e7-2a0a.nam.nsroot.net:8041
> 2016-03-23 14:26:51,480 INFO [ContainerLauncher #6]
> org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncherImpl: Shuffle
> port returned by ContainerManager for attempt_1458582499641_0019_m_000000_3
> : 13562
> 2016-03-23 14:26:51,481 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: TaskAttempt:
> [attempt_1458582499641_0019_m_000000_3] using containerId:
> [container_1458582499641_0019_01_000005 on NM:
> [sd-d7e7-2a0a.nam.nsroot.net:8041]
> 2016-03-23 14:26:51,481 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl:
> attempt_1458582499641_0019_m_000000_3 TaskAttempt Transitioned from
> ASSIGNED
> to RUNNING
> 2016-03-23 14:26:52,471 INFO [RMCommunicator Allocator]
> org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor: getResources()
> for application_1458582499641_0019: ask=1 release= 0 newContainers=0
> finishedContainers=0 resourcelimit=<memory:223232, vCores:21> knownNMs=4
> 2016-03-23 14:26:52,841 INFO [Socket Reader #1 for port 59475]
> SecurityLogger.org.apache.hadoop.ipc.Server: Auth successful for
> job_1458582499641_0019 (auth:SIMPLE)
> 2016-03-23 14:26:52,854 INFO [IPC Server handler 3 on 59475]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID :
> jvm_1458582499641_0019_m_000005 asked for a task
> 2016-03-23 14:26:52,854 INFO [IPC Server handler 3 on 59475]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: JVM with ID:
> jvm_1458582499641_0019_m_000005 given task:
> attempt_1458582499641_0019_m_000000_3
> 2016-03-23 14:26:53,874 INFO [IPC Server handler 4 on 59475]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Progress of TaskAttempt
> attempt_1458582499641_0019_m_000000_3 is : 0.0
> 2016-03-23 14:26:53,892 ERROR [IPC Server handler 5 on 59475]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Task:
> attempt_1458582499641_0019_m_000000_3 - exited :
> java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199)
> at
>
> org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:184)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
> at
>
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)
> ... 8 more
>
> 2016-03-23 14:26:53,892 INFO [IPC Server handler 5 on 59475]
> org.apache.hadoop.mapred.TaskAttemptListenerImpl: Diagnostics report from
> attempt_1458582499641_0019_m_000000_3: Error: java.lang.RuntimeException:
> java.lang.ClassNotFoundException: Class
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199)
> at
>
> org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:184)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
> at
>
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)
> ... 8 more
>
> 2016-03-23 14:26:53,893 INFO [AsyncDispatcher event handler]
> org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics
> report from attempt_1458582499641_0019_m_000000_3: Error:
> java.lang.RuntimeException: java.lang.ClassNotFoundException: Class
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199)
> at
>
> org.apache.hadoop.mapreduce.task.JobContextImpl.getInputFormatClass(JobContextImpl.java:184)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> *Caused by: java.lang.ClassNotFoundException: Class
> org.apache.hive.hcatalog.mapreduce.HCatInputFormat not found*
> at
>
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
> at
> org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)
> ... 8 more
>
>
>
>
> --
> View this message in context:
> http://apache-kylin.74782.x6.nabble.com/AbstractHadoopJob-java-and-KYLIN-HIVE-DEPENDENCY-JARS-in-1-3-1-5-versions-tp3951.html
> Sent from the Apache Kylin mailing list archive at Nabble.com.
>
--
Best regards,
Shaofeng Shi
