Hi Tanmay, Thank you for your update, and I am glad to hear that you have finally fixed your issue.
---------------- Best wishes, Xiaoxiang Yu 发件人: Tanmay Movva <[email protected]> 日期: 2019年12月6日 星期五 01:51 收件人: Xiaoxiang Yu <[email protected]> 主题: Re: Error on EMR Hey Xiaoxiang, Thank you so much. This worked for me. Also there is one mistake in your export hive_dependency, spelling error, it should be HBASE instead of HBSE. Probably while putting it on github. Haha. Thanks On Wed, Dec 4, 2019 at 10:27 AM Xiaoxiang Yu <[email protected]<mailto:[email protected]>> wrote: This is my install with some additional steps, please check this: https://github.com/hit-lacus/hit-lacus.github.io/issues/76#issuecomment-548255402 , and I didn't met the same problem as you, so I never copy any hive jar into Kylin from EMR env. Besides, I am using emr-5.27 in region cn-northwest-1. If you use a different version, maybe the problem you faced cannot fixed by my steps. ---------------- Best wishes, Xiaoxiang Yu 发件人: Tanmay Movva <[email protected]<mailto:[email protected]>> 日期: 2019年12月4日 星期三 12:25 收件人: Xiaoxiang Yu <[email protected]<mailto:[email protected]>> 主题: Re: Error on EMR Hi, Can you share your hive conf and kylin conf changes that you have made. I was able to install and setup kylin and run some .sh files. But I get the class not found error at stage 2 while building sample cube. I probably am missing some hive jar in classpath, but then I haven't made any significant changes to conf. So not able to debug. On Tue, Dec 3, 2019 at 8:11 PM Xiaoxiang Yu <[email protected]<mailto:[email protected]>> wrote: Hi, I have successfully deployed latest version of Kylin(3.0.beta) on AWS EMR 5.27 and build a few cubes successfully, maybe you can have a try? The cluster is created by CLI looks like this, and I deployed Kylin on MASTER node: aws emr create-cluster --applications Name=Hadoop Name=Hive Name=Pig Name=Spark Name=Sqoop Name=Tez Name=Zeppelin Name=ZooKeeper Name=Ganglia\ --release-label emr-5.27.0 \ --instance-groups '[{"InstanceCount":4,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":200,"VolumeType":"gp2"},"VolumesPerInstance":1}]},"InstanceGroupType":"CORE","InstanceType":"m4.2xlarge","Name":"Worker Cluster"},{"InstanceCount":1,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":100,"VolumeType":"gp2"},"VolumesPerInstance":1}]},"InstanceGroupType":"MASTER","InstanceType":"c4.4xlarge","Name":"MasterQuery"}]' \ --configurations '[{"Classification":"hdfs-site","Properties":{"dfs.replication":"2"}}]' \ --ebs-root-volume-size 100 \ --enable-debugging \ --name 'BenchmarkCluster' \ --scale-down-behavior TERMINATE_AT_TASK_COMPLETION \ --region cn-northwest-1 ---------------- Best wishes, Xiaoxiang Yu 在 2019/12/2 20:38，“Tanmay Movva”<[email protected]> 写入: Hello, We have installed kylin on our EMR master along with hbase, hadoop and hive. Using download-spark.sh from KYLIN_HOME/bin I have installed spark. As mentioned in "Install KYLIN on AWS EMR" guide we have followed the steps to configure Kylin working dir and hbase storage as S3 and also made the necessary zkquorum changes. When we run the sample.sh or check-env.sh we don't get any errors. But when we run the cube build job from UI, the job fails at stage-2 "Redistribute Flat Hive Tables". As the job "Create Intermediate Hive tables" has been completed successfully I don't think there has been any error with Hive. Can anyone help us with this? Thank You. java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf at org.apache.kylin.source.hive.CLIHiveClient.<init>(CLIHiveClient.java:47) at org.apache.kylin.source.hive.HiveClientFactory.getHiveClient(HiveClientFactory.java:27) at org.apache.kylin.source.hive.RedistributeFlatHiveTableStep.computeRowCount(RedistributeFlatHiveTableStep.java:40) at org.apache.kylin.source.hive.RedistributeFlatHiveTableStep.doWork(RedistributeFlatHiveTableStep.java:91) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) at org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hive.conf.HiveConf at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928) at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771) ... 11 more -- Regards, Tanmay Krishna Movva Razorpay -- Regards, Tanmay Krishna Movva Razorpay -- Regards, Tanmay Krishna Movva Razorpay
