?????? java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat

2016-10-19 Thread ???? 
find-hive-dependency.sh is ok
I say find-hbase-dependency.sh this script,
when i mkdir -p /usr/lib/oozie/lib, the variable HBASE_ENV_INIT is set true
i do not edit /usr/lib/hbase/conf/hbase-env.sh and  Change the first line from
export HBASE_CLASSPATH=/etc/hadoop/conf
to
export HBASE_CLASSPATH=$HBASE_CLASSPATH:/etc/hadoop/conf




--  --
??: "ShaoFeng Shi";<shaofeng...@apache.org>;
: 2016??10??19??(??) 5:38
??: "dev"<dev@kylin.apache.org>; 

: Re: java.lang.NoClassDefFoundError: 
org/apache/hive/hcatalog/mapreduce/HCatInputFormat



In 1.5.4 kylin change to load hcat from /usr/lib/hive-hcatalog, you can try
to modify the file to
https://github.com/apache/kylin/blob/kylin-1.5.4.1/build/bin/find-hive-dependency.sh#L89

2016-10-19 15:48 GMT+08:00  <121762...@qq.com>:

> it is ok,thanks
> if my emr cluster has a directory /usr/lib/oozie/lib??HBASE_ENV_INIT??true,
> i do not change /usr/lib/hbase/conf/hbase-env.sh, i do not konw why you
> judge by /usr/lib/oozie/lib directory
>
>
> if [ -n is_aws ] && [ -d "/usr/lib/oozie/lib" ]; then
> export HBASE_ENV_INIT="true"
> fi
>
>
>
>
>
> --  --
> ??: "liuyiming.vip";<liuyiming@gmail.com>;
> ????: 2016??10??19??(??) 2:47
> ??: "dev"<dev@kylin.apache.org>;
>
> : Re: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
>
>
>
> Edit /usr/lib/hbase/conf/hbase-env.sh
> Change the first line from
> export HBASE_CLASSPATH=/etc/hadoop/conf
>
> to
> export HBASE_CLASSPATH=$HBASE_CLASSPATH:/etc/hadoop/conf
>
> 2016-10-19 14:24 GMT+08:00  <121762...@qq.com>:
>
> > Hi   when i build the sample kylin_sales_cube, in the kylin.log,i find
> > java.lang.NoClassDefFoundError. but in the log i find some messages, i
> do
> > not konw why?
> >
> >
> > /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.
> 1.0-amzn-0.jar
> > is loaded, why class is not foung?
> >
> >
> > 016-10-18 23:17:59,902 INFO  [pool-8-thread-3]
> > common.AbstractHadoopJob:203 : Hive Dependencies After Filtered:
> > /usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,/
> > usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,/usr/lib/hive-
> > hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
> >
> >
> > 2016-10-18 23:17:59,907 INFO  [pool-8-thread-3]
> > common.AbstractHadoopJob:371 : Job 'tmpjars' updated --
> > file:/usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,
> > file:/usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,file:/
> > usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
> >
> >
> > 2016-10-18 23:17:59,934 ERROR [pool-8-thread-3]
> > threadpool.DefaultScheduler:140 : ExecuteException
> > job:845380a7-63b7-4315-8d71-6bb290018790
> > org.apache.kylin.job.exception.ExecuteException: org.apache.kylin.job.
> exception.ExecuteException:
> > java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> > mapreduce/HCatInputFormat
> > at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:123)
> > at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> > JobRunner.run(DefaultScheduler.java:136)
> > at java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > Caused by: org.apache.kylin.job.exception.ExecuteException: java.lang.
> NoClassDefFoundError:
> > org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> > at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:123)
> > at org.apache.kylin.job.execution.DefaultChainedExecutable.
> doWork(
> > DefaultChainedExecutable.java:57)
> > at org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:113)
> > ... 4 more
> > Caused by: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> > mapreduce/HCatInputFormat
> > at org.apache.kylin.source.hive.HiveMRInput$
> HiveTableInputFormat.
> > configureJob(HiveMRInput.java:89)
> > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> > setupMapper(FactDistinctColumnsJob.java:123)
> > at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> > run(FactDistinct

?????? java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat

2016-10-19 Thread ???? 
it is ok,thanks
if my emr cluster has a directory /usr/lib/oozie/lib??HBASE_ENV_INIT??true, i 
do not change /usr/lib/hbase/conf/hbase-env.sh, i do not konw why you judge by 
/usr/lib/oozie/lib directory


if [ -n is_aws ] && [ -d "/usr/lib/oozie/lib" ]; then
export HBASE_ENV_INIT="true"
fi





--  --
??: "liuyiming.vip";<liuyiming@gmail.com>;
: 2016??10??19??(??) 2:47
??: "dev"<dev@kylin.apache.org>; 

: Re: java.lang.NoClassDefFoundError: 
org/apache/hive/hcatalog/mapreduce/HCatInputFormat



Edit /usr/lib/hbase/conf/hbase-env.sh
Change the first line from
export HBASE_CLASSPATH=/etc/hadoop/conf

to
export HBASE_CLASSPATH=$HBASE_CLASSPATH:/etc/hadoop/conf

2016-10-19 14:24 GMT+08:00  <121762...@qq.com>:

> Hi   when i build the sample kylin_sales_cube, in the kylin.log,i find
> java.lang.NoClassDefFoundError. but in the log i find some messages, i do
> not konw why?
>
>
> /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
> is loaded, why class is not foung?
>
>
> 016-10-18 23:17:59,902 INFO  [pool-8-thread-3]
> common.AbstractHadoopJob:203 : Hive Dependencies After Filtered:
> /usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,/
> usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,/usr/lib/hive-
> hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
>
>
> 2016-10-18 23:17:59,907 INFO  [pool-8-thread-3]
> common.AbstractHadoopJob:371 : Job 'tmpjars' updated --
> file:/usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,
> file:/usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,file:/
> usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
>
>
> 2016-10-18 23:17:59,934 ERROR [pool-8-thread-3]
> threadpool.DefaultScheduler:140 : ExecuteException
> job:845380a7-63b7-4315-8d71-6bb290018790
> org.apache.kylin.job.exception.ExecuteException: 
> org.apache.kylin.job.exception.ExecuteException:
> java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:123)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> JobRunner.run(DefaultScheduler.java:136)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kylin.job.exception.ExecuteException: 
> java.lang.NoClassDefFoundError:
> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:123)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:57)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
> ... 4 more
> Caused by: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
> at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.
> configureJob(HiveMRInput.java:89)
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> setupMapper(FactDistinctColumnsJob.java:123)
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> run(FactDistinctColumnsJob.java:103)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:88)
> at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:120)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
> ... 6 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.
> mapreduce.HCatInputFormat
> at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> WebappClassLoaderBase.java:1858)
> at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> WebappClassLoaderBase.java:1701)
> ... 12 more




-- 
With Warm regards

Yiming Liu (??)

Re: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat

2016-10-19 Thread Billy(Yiming) Liu 
Edit /usr/lib/hbase/conf/hbase-env.sh
Change the first line from
export HBASE_CLASSPATH=/etc/hadoop/conf

to
export HBASE_CLASSPATH=$HBASE_CLASSPATH:/etc/hadoop/conf

2016-10-19 14:24 GMT+08:00 张磊 <121762...@qq.com>:

> Hi   when i build the sample kylin_sales_cube, in the kylin.log,i find
> java.lang.NoClassDefFoundError. but in the log i find some messages, i do
> not konw why?
>
>
> /usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
> is loaded, why class is not foung?
>
>
> 016-10-18 23:17:59,902 INFO  [pool-8-thread-3]
> common.AbstractHadoopJob:203 : Hive Dependencies After Filtered:
> /usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,/
> usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,/usr/lib/hive-
> hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
>
>
> 2016-10-18 23:17:59,907 INFO  [pool-8-thread-3]
> common.AbstractHadoopJob:371 : Job 'tmpjars' updated --
> file:/usr/lib/hive/lib/hive-metastore-2.1.0-amzn-0.jar,
> file:/usr/lib/hive/lib/hive-exec-2.1.0-amzn-0.jar,file:/
> usr/lib/hive-hcatalog/share/hcatalog/hive-hcatalog-core-2.1.0-amzn-0.jar
>
>
> 2016-10-18 23:17:59,934 ERROR [pool-8-thread-3]
> threadpool.DefaultScheduler:140 : ExecuteException
> job:845380a7-63b7-4315-8d71-6bb290018790
> org.apache.kylin.job.exception.ExecuteException: 
> org.apache.kylin.job.exception.ExecuteException:
> java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:123)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$
> JobRunner.run(DefaultScheduler.java:136)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kylin.job.exception.ExecuteException: 
> java.lang.NoClassDefFoundError:
> org/apache/hive/hcatalog/mapreduce/HCatInputFormat
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:123)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:57)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
> ... 4 more
> Caused by: java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/
> mapreduce/HCatInputFormat
> at org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.
> configureJob(HiveMRInput.java:89)
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> setupMapper(FactDistinctColumnsJob.java:123)
> at org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> run(FactDistinctColumnsJob.java:103)
> at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:88)
> at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:120)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:113)
> ... 6 more
> Caused by: java.lang.ClassNotFoundException: org.apache.hive.hcatalog.
> mapreduce.HCatInputFormat
> at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> WebappClassLoaderBase.java:1858)
> at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(
> WebappClassLoaderBase.java:1701)
> ... 12 more




-- 
With Warm regards

Yiming Liu (刘一鸣)

[jira] [Created] (KYLIN-1842) java.lang.NoClassDefFoundError: org/apache/hive/hcatalog/mapreduce/HCatInputFormat

2016-06-30 Thread raphael5200 (JIRA) 
raphael5200 created KYLIN-1842:
--

 Summary: java.lang.NoClassDefFoundError: 
org/apache/hive/hcatalog/mapreduce/HCatInputFormat
 Key: KYLIN-1842
 URL: https://issues.apache.org/jira/browse/KYLIN-1842
 Project: Kylin
  Issue Type: Bug
  Components: Job Engine
Affects Versions: v1.5.0
 Environment: Zookeeper-3.4
Hadoop-2.5
hbase-1.1.3
hive-1.2.1
kylin-1.5.0
Reporter: raphael5200
Assignee: Dong Li
 Fix For: 1.x-HBase1.1.3


i have a error in kylin;
Error message:
2016-07-01 12:30:07,431 ERROR [pool-5-thread-2] 
execution.AbstractExecutable:116 : error running Executable
org.apache.kylin.job.exception.ExecuteException: 
java.lang.NoClassDefFoundError: 
org/apache/hive/hcatalog/mapreduce/HCatInputFormat
at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at 
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
at 
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:124)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NoClassDefFoundError: 
org/apache/hive/hcatalog/mapreduce/HCatInputFormat
at 
org.apache.kylin.source.hive.HiveMRInput$HiveTableInputFormat.configureJob(HiveMRInput.java:81)
at 
org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.setupMapper(FactDistinctColumnsJob.java:111)
at 
org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:91)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at 
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:118)
at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:114)
... 6 more
Caused by: java.lang.ClassNotFoundException: 
org.apache.hive.hcatalog.mapreduce.HCatInputFormat
at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1720)
at 
org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1571)
... 13 more

my conf:
.bash_profile:
PATH=$PATH:$HOME/bin
JAVA_HOME=/usr/java/jdk1.7.0_79
HADOOP_HOME=/usr/local/hadoop-2.5.1
ZOOKEEPER_HOME=/usr/local/zookeeper-3.4.6
HBASE_HOME=/usr/local/hbase-1.1.3
HIVE_HOME=/usr/local/hive-1.2.1


HCAT_HOME=$HIVE_HOME/hcatalog
KYLIN_HOME=/usr/local/kylin-1.5.0-HBase1.1.3
CATALINA_HOME=$KYLIN_HOME/tomcat
hive_dependency=$HIVE_HOME/conf:$HIVE_HOME/lib/*:$HCAT_HOME/share/hcatalog/hive-hcatalog-core-1.2.1.jar

PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$ZOOKEEPER_HOME/bin:$HBASE_HOME/bin:$HIVE_HOME/bin:$HCAT_HOME/bin:$CATALINA_HOME/bin:$KYLIN_HOME/bin
export JAVA_HOME
export HADOOP_HOME
export ZOOKEEPER_HOME
export HBASE_HOME
export HIVE_HOME

export HCAT_HOME
export CATALINA_HOME
export KYLIN_HOME
export PATH

kylin.sh:
export KYLIN_HOME=/usr/local/kylin-1.5.0-HBase1.1.3
export 
HBASE_CLASSPATH_PREFIX=${tomcat_root}/bin/bootstrap.jar:${tomcat_root}/bin/tomcat-juli.jar:${tomcat_root}/lib/*:$hive_dependency:$HBASE_CLASSPATH_PREFIX

kylin.properties:
kylin.rest.servers=master01:7070
kylin.job.jar=/tmp/kylin/kylin-job-latest.jar
kylin.coprocessor.local.jar=/tmp/kylin/kylin-coprocessor-latest.jar

please help me 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)