Hi Tanmay, Could you share how did you fix the problem? We can update the document if it missed something. Thanks!
Best regards, Shaofeng Shi 史少锋 Apache Kylin PMC Email: [email protected] Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html Join Kylin user mail group: [email protected] Join Kylin dev mail group: [email protected] Xiaoxiang Yu <[email protected]> 于2019年12月6日周五 上午10:28写道： > Hi Tanmay, > > Thank you for your update, and I am glad to hear that you have finally > fixed your issue. > > > > ---------------- > > Best wishes, > > Xiaoxiang Yu > > > > > > *发件人**: *Tanmay Movva <[email protected]> > *日期**: *2019年12月6日 星期五 01:51 > *收件人**: *Xiaoxiang Yu <[email protected]> > *主题**: *Re: Error on EMR > > > > Hey Xiaoxiang, > > > > Thank you so much. This worked for me. Also there is one mistake in your > export hive_dependency, spelling error, it should be HBASE instead of HBSE. > Probably while putting it on github. Haha. Thanks > > > > On Wed, Dec 4, 2019 at 10:27 AM Xiaoxiang Yu <[email protected]> > wrote: > > This is my install with some additional steps, please check this: > https://github.com/hit-lacus/hit-lacus.github.io/issues/76#issuecomment-548255402 > , and I didn't met the same problem as you, so I never copy any hive jar > into Kylin from EMR env. > > > > Besides, I am using emr-5.27 in region cn-northwest-1. If you use a > different version, maybe the problem you faced cannot fixed by my steps. > > > > ---------------- > > Best wishes, > > Xiaoxiang Yu > > > > > > *发件人**: *Tanmay Movva <[email protected]> > *日期**: *2019年12月4日 星期三 12:25 > *收件人**: *Xiaoxiang Yu <[email protected]> > *主题**: *Re: Error on EMR > > > > Hi, > > > > Can you share your hive conf and kylin conf changes that you have made. I > was able to install and setup kylin and run some .sh files. But I get the > class not found error at stage 2 while building sample cube. I probably am > missing some hive jar in classpath, but then I haven't made any significant > changes to conf. So not able to debug. > > > > On Tue, Dec 3, 2019 at 8:11 PM Xiaoxiang Yu <[email protected]> > wrote: > > Hi, > I have successfully deployed latest version of Kylin(3.0.beta) on AWS > EMR 5.27 and build a few cubes successfully, maybe you can have a try? > The cluster is created by CLI looks like this, and I deployed Kylin on > MASTER node: > > aws emr create-cluster --applications Name=Hadoop Name=Hive Name=Pig > Name=Spark Name=Sqoop Name=Tez Name=Zeppelin Name=ZooKeeper Name=Ganglia\ > --release-label emr-5.27.0 \ > --instance-groups > '[{"InstanceCount":4,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":200,"VolumeType":"gp2"},"VolumesPerInstance":1}]},"InstanceGroupType":"CORE","InstanceType":"m4.2xlarge","Name":"Worker > Cluster"},{"InstanceCount":1,"EbsConfiguration":{"EbsBlockDeviceConfigs":[{"VolumeSpecification":{"SizeInGB":100,"VolumeType":"gp2"},"VolumesPerInstance":1}]},"InstanceGroupType":"MASTER","InstanceType":"c4.4xlarge","Name":"MasterQuery"}]' > \ > --configurations > '[{"Classification":"hdfs-site","Properties":{"dfs.replication":"2"}}]' \ > --ebs-root-volume-size 100 \ --enable-debugging \ > --name 'BenchmarkCluster' \ > --scale-down-behavior TERMINATE_AT_TASK_COMPLETION \ > --region cn-northwest-1 > > ---------------- > Best wishes, > Xiaoxiang Yu > > > 在 2019/12/2 20:38，“Tanmay Movva”<[email protected]> 写入: > > Hello, > > We have installed kylin on our EMR master along with hbase, hadoop and > hive. Using download-spark.sh from KYLIN_HOME/bin I have installed > spark. > As mentioned in "Install KYLIN on AWS EMR" guide we have followed the > steps > to configure Kylin working dir and hbase storage as S3 and also made > the > necessary zkquorum changes. > > When we run the sample.sh or check-env.sh we don't get any errors. But > when > we run the cube build job from UI, the job fails at stage-2 > "Redistribute > Flat Hive Tables". As the job "Create Intermediate Hive tables" has > been > completed successfully I don't think there has been any error with > Hive. > > Can anyone help us with this? Thank You. > > > java.lang.NoClassDefFoundError: org/apache/hadoop/hive/conf/HiveConf > at > org.apache.kylin.source.hive.CLIHiveClient.<init>(CLIHiveClient.java:47) > at > org.apache.kylin.source.hive.HiveClientFactory.getHiveClient(HiveClientFactory.java:27) > at > org.apache.kylin.source.hive.RedistributeFlatHiveTableStep.computeRowCount(RedistributeFlatHiveTableStep.java:40) > at > org.apache.kylin.source.hive.RedistributeFlatHiveTableStep.doWork(RedistributeFlatHiveTableStep.java:91) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:167) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748) > Caused by: java.lang.ClassNotFoundException: > org.apache.hadoop.hive.conf.HiveConf > at > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928) > at > org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771) > ... 11 more > > -- > Regards, > Tanmay Krishna Movva > Razorpay > > > > > -- > > Regards, > > Tanmay Krishna Movva > > Razorpay > > > > > -- > > Regards, > > Tanmay Krishna Movva > > Razorpay >
