Re: kylin 无法加载hive classpath 内得lib

[Li Yang]

> The Exception threw out from the kylin process, and the EMR version is
5.0.0

Then you could analyze the "bin/find-hive-dependency.sh" to see if it
catches the right hive classpath in your env.

Cheers
Yang


On Tue, Mar 21, 2017 at 11:17 AM, wu lihu  wrote:

> one thing needs to be reminded, if I use Hive CLI, then everything is
> great.
>
> 2017-03-21 11:13 GMT+08:00 wu lihu :
> > The whole stacktrace like below :
> > Caused by: MetaException(message:java.lang.ClassNotFoundException
> > Class org.openx.data.jsonserde.JsonSerDe not found)
> > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_
> fields_result$get_fields_resultStandardScheme.read(ThriftHive
> > Metastore.java:35482)
> > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_
> fields_result$get_fields_resultStandardScheme.read(ThriftHive
> > Metastore.java:35449)
> > at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_
> fields_result.read(ThriftHiveMetastore.java:35375)
> > at org.apache.thrift.TServiceClient.receiveBase(
> TServiceClient.java:86)
> > at org.apache.hadoop.hive.metastore.api.
> ThriftHiveMetastore$Client.recv_get_fields(ThriftHiveMetastore.java:1017)
> > at org.apache.hadoop.hive.metastore.api.
> ThriftHiveMetastore$Client.get_fields(ThriftHiveMetastore.java:1003)
> > at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.
> getFields(HiveMetaStoreClient.java:1497)
> > at org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(
> CLIHiveClient.java:80)
> > at org.apache.kylin.source.hive.HiveSourceTableLoader.
> extractHiveTables(HiveSourceTableLoader.java:86)
> > ... 84 more
> >
> > seems like it caused by the hive thrift service read the wrong
> > metastore? or the state of the  metastore has incorrect reference 
> >
> > 2017-03-21 11:04 GMT+08:00 wu lihu :
> >> Hi Li Yang & ShaoFeng
> >> The Exception threw out from the kylin process, and the EMR
> >> version is  5.0.0
> >>
> >> 2017-03-21 2:59 GMT+08:00 Li Yang :
> >>> Also is the ClassNotFoundException from Kylin process or from MR
> process?
> >>> That makes a difference.
> >>>
> >>> On Mon, Mar 20, 2017 at 4:24 PM, ShaoFeng Shi 
> >>> wrote:
> >>>
>  Which version of EMR are you running? In my experience if using EMR,
> there
>  is no additional configuration for Kylin; just download and run
> "kylin.sh
>  start" will work.
> 
>  2017-03-17 16:38 GMT+08:00 wu lihu :
> 
>  > Hi All
>  > 我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
>  > find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta  报错
>  >  Caused by: MetaException(message:java.lang.ClassNotFoundException
>  > Class org.openx.data.jsonserde.JsonSerDe not found)
>  >
>  > 但是我在 System tab 里面可以看到
>  > /usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-
> dependencies.jar
>  > 在classpath 里面。 请问这个怎么解决？
>  >
> 
> 
> 
>  --
>  Best regards,
> 
>  Shaofeng Shi 史少锋
> 
>

Re: kylin 无法加载hive classpath 内得lib

[wu lihu]

one thing needs to be reminded, if I use Hive CLI, then everything is great.

2017-03-21 11:13 GMT+08:00 wu lihu :
> The whole stacktrace like below :
> Caused by: MetaException(message:java.lang.ClassNotFoundException
> Class org.openx.data.jsonserde.JsonSerDe not found)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHive
> Metastore.java:35482)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHive
> Metastore.java:35449)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result.read(ThriftHiveMetastore.java:35375)
> at 
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_fields(ThriftHiveMetastore.java:1017)
> at 
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_fields(ThriftHiveMetastore.java:1003)
> at 
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:1497)
> at 
> org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:80)
> at 
> org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:86)
> ... 84 more
>
> seems like it caused by the hive thrift service read the wrong
> metastore? or the state of the  metastore has incorrect reference 
>
> 2017-03-21 11:04 GMT+08:00 wu lihu :
>> Hi Li Yang & ShaoFeng
>> The Exception threw out from the kylin process, and the EMR
>> version is  5.0.0
>>
>> 2017-03-21 2:59 GMT+08:00 Li Yang :
>>> Also is the ClassNotFoundException from Kylin process or from MR process?
>>> That makes a difference.
>>>
>>> On Mon, Mar 20, 2017 at 4:24 PM, ShaoFeng Shi 
>>> wrote:
>>>
 Which version of EMR are you running? In my experience if using EMR, there
 is no additional configuration for Kylin; just download and run "kylin.sh
 start" will work.

 2017-03-17 16:38 GMT+08:00 wu lihu :

 > Hi All
 > 我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
 > find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta  报错
 >  Caused by: MetaException(message:java.lang.ClassNotFoundException
 > Class org.openx.data.jsonserde.JsonSerDe not found)
 >
 > 但是我在 System tab 里面可以看到
 > /usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
 > 在classpath 里面。 请问这个怎么解决？
 >



 --
 Best regards,

 Shaofeng Shi 史少锋

Re: kylin 无法加载hive classpath 内得lib

[wu lihu]

The whole stacktrace like below :
Caused by: MetaException(message:java.lang.ClassNotFoundException
Class org.openx.data.jsonserde.JsonSerDe not found)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHive
Metastore.java:35482)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result$get_fields_resultStandardScheme.read(ThriftHive
Metastore.java:35449)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_fields_result.read(ThriftHiveMetastore.java:35375)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:86)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_fields(ThriftHiveMetastore.java:1017)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_fields(ThriftHiveMetastore.java:1003)
at 
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getFields(HiveMetaStoreClient.java:1497)
at 
org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:80)
at 
org.apache.kylin.source.hive.HiveSourceTableLoader.extractHiveTables(HiveSourceTableLoader.java:86)
... 84 more

seems like it caused by the hive thrift service read the wrong
metastore? or the state of the  metastore has incorrect reference 

2017-03-21 11:04 GMT+08:00 wu lihu :
> Hi Li Yang & ShaoFeng
> The Exception threw out from the kylin process, and the EMR
> version is  5.0.0
>
> 2017-03-21 2:59 GMT+08:00 Li Yang :
>> Also is the ClassNotFoundException from Kylin process or from MR process?
>> That makes a difference.
>>
>> On Mon, Mar 20, 2017 at 4:24 PM, ShaoFeng Shi 
>> wrote:
>>
>>> Which version of EMR are you running? In my experience if using EMR, there
>>> is no additional configuration for Kylin; just download and run "kylin.sh
>>> start" will work.
>>>
>>> 2017-03-17 16:38 GMT+08:00 wu lihu :
>>>
>>> > Hi All
>>> > 我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
>>> > find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta  报错
>>> >  Caused by: MetaException(message:java.lang.ClassNotFoundException
>>> > Class org.openx.data.jsonserde.JsonSerDe not found)
>>> >
>>> > 但是我在 System tab 里面可以看到
>>> > /usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
>>> > 在classpath 里面。 请问这个怎么解决？
>>> >
>>>
>>>
>>>
>>> --
>>> Best regards,
>>>
>>> Shaofeng Shi 史少锋
>>>

Re: kylin 无法加载hive classpath 内得lib

[wu lihu]

Hi Li Yang & ShaoFeng
The Exception threw out from the kylin process, and the EMR
version is  5.0.0

2017-03-21 2:59 GMT+08:00 Li Yang :
> Also is the ClassNotFoundException from Kylin process or from MR process?
> That makes a difference.
>
> On Mon, Mar 20, 2017 at 4:24 PM, ShaoFeng Shi 
> wrote:
>
>> Which version of EMR are you running? In my experience if using EMR, there
>> is no additional configuration for Kylin; just download and run "kylin.sh
>> start" will work.
>>
>> 2017-03-17 16:38 GMT+08:00 wu lihu :
>>
>> > Hi All
>> > 我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
>> > find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta  报错
>> >  Caused by: MetaException(message:java.lang.ClassNotFoundException
>> > Class org.openx.data.jsonserde.JsonSerDe not found)
>> >
>> > 但是我在 System tab 里面可以看到
>> > /usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
>> > 在classpath 里面。 请问这个怎么解决？
>> >
>>
>>
>>
>> --
>> Best regards,
>>
>> Shaofeng Shi 史少锋
>>

Re: kylin 无法加载hive classpath 内得lib

[Li Yang]

Also is the ClassNotFoundException from Kylin process or from MR process?
That makes a difference.

On Mon, Mar 20, 2017 at 4:24 PM, ShaoFeng Shi 
wrote:

> Which version of EMR are you running? In my experience if using EMR, there
> is no additional configuration for Kylin; just download and run "kylin.sh
> start" will work.
>
> 2017-03-17 16:38 GMT+08:00 wu lihu :
>
> > Hi All
> > 我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
> > find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta  报错
> >  Caused by: MetaException(message:java.lang.ClassNotFoundException
> > Class org.openx.data.jsonserde.JsonSerDe not found)
> >
> > 但是我在 System tab 里面可以看到
> > /usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
> > 在classpath 里面。 请问这个怎么解决？
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi 史少锋
>

Re: kylin 无法加载hive classpath 内得lib

[ShaoFeng Shi]

Which version of EMR are you running? In my experience if using EMR, there
is no additional configuration for Kylin; just download and run "kylin.sh
start" will work.

2017-03-17 16:38 GMT+08:00 wu lihu :

> Hi All
> 我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
> find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta  报错
>  Caused by: MetaException(message:java.lang.ClassNotFoundException
> Class org.openx.data.jsonserde.JsonSerDe not found)
>
> 但是我在 System tab 里面可以看到
> /usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
> 在classpath 里面。 请问这个怎么解决？
>



-- 
Best regards,

Shaofeng Shi 史少锋

kylin 无法加载hive classpath 内得lib

[wu lihu]

Hi All
我是用得 AWS EMR， 我把JSON-SERDE 得 jar 加入了 hive 得 classpath 里面了，修改了
find-hive-dependency.sh ，把这个JAR 得路径也加入进去了。可是还是加载不了 hivemeta  报错
 Caused by: MetaException(message:java.lang.ClassNotFoundException
Class org.openx.data.jsonserde.JsonSerDe not found)

但是我在 System tab 里面可以看到
/usr/lib/hive/lib/json-serde-1.3.8-SNAPSHOT-jar-with-dependencies.jar
在classpath 里面。 请问这个怎么解决？