Re: kylin 无法加载hive classpath 内得lib

[Li Yang]

> The Exception threw out from the kylin process, and the EMR version is
5.0.0

Then you could analyze the "bin/find-hive-dependency.sh" to see if it
catches the right hive classpath in your env.

Cheers
Yang


On Tue, Mar 21, 2017 at 11:17 AM, wu lihu  wrote:

> one thing needs to be reminded, if I use Hive CLI, then everything is
> great.
>
> 2017-03-21 11:13 GMT+08:00 wu lihu :
> > The whole stacktrace like below :
> > Caused by: MetaException(message:java.lang.ClassNotFoundException
> > Class org.openx.data.jsonserde.JsonSerDe not found)
> > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_
> fields_result$get_fields_resultStandardScheme.read(ThriftHive
> > Metastore.java:35482)
> > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_
> fields_result$get_fields_resultStandardScheme.read(ThriftHive
> > Metastore.java:35449)
> > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_
> fields_result.read(ThriftHiveMetastore.java:35375)
> > at org.apache.thrift.TServiceClient.receiveBase(
> TServiceClient.java:86)
> > at org.apache.hadoop.hive.metastore.api.
> ThriftHiveMetastore$Client.recv_get_fields(ThriftHiveMetastore.java:1017)
> > at org.apache.hadoop.hive.metastore.api.
> ThriftHiveMetastore$Client.get_fields(ThriftHiveMetastore.java:1003)
> > at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.
> getFields(HiveMetaStoreClient.java:1497)
> > at org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(
> CLIHiveClient.java:80)
> > at org.apache.kylin.source.hive.HiveSourceTableLoader.
> extractHiveTables(HiveSourceTableLoader.java:86)
> > ... 84 more
> >
> > seems like it caused by the hive thrift service read the wrong
> > metastore? or the state of the  metastore has incorrect reference 
> >
> > 2017-03-21 11:04 GMT+08:00 wu lihu :
> >> Hi Li Yang & ShaoFeng
> >> The Exception threw out from the kylin process, and the EMR
> >> version is  5.0.0
> >>
> >> 2017-03-21 2:59 GMT+08:00 Li Yang :
> >>> Also is the ClassNotFoundException from Kylin process or from MR
> process?
> >>> That makes a difference.
> >>>
> >>> On Mon, Mar 20, 2017 at 4:24 PM, ShaoFeng Shi 
> >>> wrote:
> >>>
>  Which version of EMR are you running? In my experience if using EMR,
> there
>  is no additional configuration for Kylin; just download and run
> "kylin.sh
>  start" will work.
> 
>  2017-03-17 16:38 GMT+08:00 wu lihu :
> 
>  > Hi All
>  > 我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
>  > find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta  报错
>  >  Caused by: MetaException(message:java.lang.ClassNotFoundException
>  > Class org.openx.data.jsonserde.JsonSerDe not found)
>  >
>  > 但是我在 System tab 里面可以看到
>  > /usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-
> dependencies.jar
>  > 在classpath 里面。 请问这个怎么解决？
>  >
> 
> 
> 
>  --
>  Best regards,
> 
>  Shaofeng Shi 史少锋
> 
>

Re: Re: Help needed - Issue when build the cube

[Li Yang]

Interesting... Any one caught a reproducible steps, please let us know.

On Tue, Mar 21, 2017 at 11:14 AM, 赵天烁  wrote:

> I had same problem like yejun,but I'm in kylin 2.0,samething happen when I
> discard current job and retry.but when I clone the cube to another name,and
> build with the same time range.it finish successfully.
>
> 
> 赵天烁
> Kevin Zhao
> zhaotians...@meizu.com
>
> 珠海市魅族科技有限公司
> MEIZU Technology Co., Ltd.
> 广东省珠海市科技创新海岸魅族科技楼
> MEIZU Tech Bldg., Technology & Innovation Coast
> Zhuhai, 519085, Guangdong, China
> meizu.com
>
> From: ShaoFeng Shi
> Date: 2017-02-16 17:40
> To: dev
> Subject: Re: Help needed - Issue when build the cube
>
> Assuming you're running with Kylin 1.6; from the source code, it indicates
> that the segment wasn't found, this shoudln't happen usually; You can
> discard current job, and then retry the build:
> https://github.com/apache/kylin/blob/kylin-1.6.0/engine-
> mr/src/main/java/org/apache/kylin/engine/mr/steps/
> FactDistinctColumnsJob.java#L98,
> it
>
> 2017-02-14 13:57 GMT+08:00 xingyejun :
>
> > Hi,
> >
> > Does anyone met the following issues  when building a cube?
> >
> > java.lang.IllegalStateException
> > at
> > org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.
> > run(FactDistinctColumnsJob.java:98)
> > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:88)
> > at
> > org.apache.kylin.engine.mr.common.MapReduceExecutable.
> > doWork(MapReduceExecutable.java:120)
> > at
> > org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:113)
> > at
> > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> > DefaultChainedExecutable.java:57)
> > at
> > org.apache.kylin.job.execution.AbstractExecutable.
> > execute(AbstractExecutable.java:113)
> > at
> > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(
> > DefaultScheduler.java:136)
> > at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > result code:2
> >
> > --
> > View this message in context: http://apache-kylin.74782.x6.
> > nabble.com/Help-needed-Issue-when-build-the-cube-tp7191.html
> > Sent from the Apache Kylin mailing list archive at Nabble.com.
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>

Re: 遇到Failed to parse hive conf file这个错误

[ShaoFeng Shi]

hi, what's the detail error of "Failed to parse hive conf file"? did you
identify the root cause of this error?

Besides, please discard the error job before submitting a new job for the
same time range. When discard a job, its associated segment will be dropped
automatically.

在 2017年3月23日 下午4:24，35925138 <35925...@qq.com>写道：

> 你好dev:
> 我建立好cube后，进行build，遇到Failed to parse hive conf file这个错误，
> 再次build，就提示我 Segments overlap: user_a
> ction[20170301010500_20170322183000] and user_action[20170301010500_
> 20170322183000]
>
>
> 可是我是初次build，不应该会有重复的segment吧，而且我看了一下，这两个segment是同一个，
> kylin版本是1.5.4
> hive版本是1.2.1




-- 
Best regards,

Shaofeng Shi 史少锋

????Failed to parse hive conf file????????

[35925138]

dev:
cubebuild??Failed to parse hive conf file??
build?? Segments overlap: user_a
ction[20170301010500_20170322183000] and 
user_action[20170301010500_20170322183000]


build??segment??segment??
kylin??1.5.4
hive??1.2.1