https://cwiki.apache.org/confluence/display/Hive/HiveDeveloperFAQ#HiveDeveloperFAQ-Howtobuildallsource ?
On Tue, Dec 22, 2015 at 4:34 PM, hongbin ma <[email protected]> wrote: > btw, please prefer to use English, apache is a global community > > On Tue, Dec 22, 2015 at 4:33 PM, hongbin ma <[email protected]> wrote: > >> hi, how's your hive built? is it compiled against hadoop 1.x? JobContext >> was a class in hadoop 1.x, but became an interface in hadoop 2.x >> >> On Tue, Dec 22, 2015 at 4:23 PM, 胡志华(万里通科技及数据中心商务智能团队数据分析组) < >> [email protected]> wrote: >> >>> 是的，我的很有可能是版本问题 >>> >>> -----邮件原件----- >>> 发件人: TTS2沉默天使 [mailto:[email protected]] >>> 发送时间: 2015年12月22日 16:20 >>> 收件人: dev >>> 主题: 回复：答复: building cube , stop at step 2,help >>> >>> 看了下错误 你的错误和我的不一样 但是都是第二步 >>> >>> >>> >>> >>> ------------------ 原始邮件 ------------------ >>> 发件人: " 胡志华(万里通科技及数据中心商务智能团队数据分析组)";<[email protected]>; >>> 发送时间: 2015年12月22日(星期二) 下午4:17 >>> 收件人: "'[email protected]'"<[email protected]>; >>> >>> 主题: 答复: building cube , stop at step 2,help >>> >>> >>> >>> Thanks, you mean my hive version is incompatible with hadoop2.7.1 >>> >>> My hive is pre-build binary package, so I will check the hadoop dependecy >>> >>> >>> -- >>> ---邮件原件----- >>> 发件人: [email protected] [mailto:[email protected]] >>> 发送时间: 2015年12月22日 16:02 >>> 收件人: dev >>> 主题: Re: building cube , stop at step 2,help >>> >>> Hi, >>> I think there could be a hadoop version mismatch in your binary package. >>> If you use of hive and hbase, maybe the default hadoop dependency is not >>> as latest as your current use 2.7.1 , thus may lead to the >>> IncompatibleClassChangeError. >>> >>> One possible workaround is to recompile and package your hive and hbase >>> from souce code with targeting hadoop version to 2.7.1. >>> >>> >>> >>> [email protected] >>> >>> From: 胡志华(万里通科技及数据中心商务智能团队数据分析组) >>> Date: 2015-12-22 15:54 >>> To: '[email protected]' >>> Subject: building cube , stop at step 2,help Hi, >>> >>> I downloaded “apache-kylin-1.1.1-incubating-bin”, and use >>> sample.sh to build a cube “kylin_sales_cube”, when I started the job , it >>> stoped at step 2, with error >>> >>> java.lang.IncompatibleClassChangeError: Found interface >>> org.apache.hadoop.mapreduce.JobContext, but class was expected >>> at >>> org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:102) >>> at >>> org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301) >>> at >>> org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318) >>> at >>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196) >>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) >>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) >>> at java.security.AccessController.doPrivileged(Native Method) >>> at javax.security.auth.Subject.doAs(Subject.java:415) >>> at >>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) >>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) >>> at >>> org.apache.kylin.job.hadoop.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:120) >>> at >>> org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:83) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) >>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) >>> at >>> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113) >>> at >>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) >>> at >>> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51) >>> at >>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) >>> at >>> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> My environment lists as following >>> >>> Hadoop2.7.1 >>> Hbase0.98.13 >>> Hive0.13.1 >>> >>> So ,from the error code above, my hive environment is >>> incompatible? And what does the parameter mean at step 2? >>> >>> -conf >>> /home/hadoop/apache-kylin-1.1.1-incubating-bin/conf/kylin_job_conf.xml >>> -cubename kylin_sales_cube -output >>> /kylin/kylin_metadata/kylin-79ec4e09-9128-42bd-a455-9dcf3c16a42a/kylin_sales_cube/fact_distinct_columns >>> -jobname Kylin_Fact_Distinct_Columns_kylin_sales_cube_Step -tablename >>> default.kylin_intermediate_kylin_sales_cube_desc_19700101000000_20151217000000_79ec4e09_9128_42bd_a455_9dcf3c16a42a >>> >>> >>> -----邮件原件----- >>> 发件人: Shi, Shaofeng [mailto:[email protected]] >>> 发送时间: 2015年12月22日 15:39 >>> 收件人: [email protected] >>> 主题: Re: about assigning different mr job queue to different user groups >>> inside one kylin instances >>> >>> Thanks Dong; I put a comment in your JIRA: >>> https://issues.apache.org/jira/browse/KYLIN-1241, we can discuss there. >>> >>> On 12/21/15, 10:29 AM, "dong wang" <[email protected]> wrote: >>> >>> >I have added the logic ( https://github.com/apache/kylin/pull/17 ) to >>> >auto detect the kylin_job_conf_medium/small/large_projectname.xml >>> >first, then kylin_job_conf_projectname.xml, then >>> >kylin_job_conf_medium/small/large.xml, >>> >finally kylin_job_conf.xml, if still not found, throws an exception >>> > >>> >2015-12-14 16:22 GMT+08:00 yu feng <[email protected]>: >>> > >>> >> Maybe a quick solution is just create a config file for every project >>> >>named ''kylin_job_conf_${projectname}.xml", it will just modify code >>> >>that selecting config file for a MR job, and you can control every MR >>> >>config property in project level. >>> >> >>> >> 2015-12-11 22:00 GMT+08:00 Shi, Shaofeng <[email protected]>: >>> >> >>> >> > This is a valid scenario; So far Kylin doesn¹t have project or cube >>> >>level >>> >> > job configurations, if it be implemented, you problem will be >>> solved. >>> >> > >>> >> > On 12/11/15, 7:09 PM, "dong wang" <[email protected]> wrote: >>> >> > >>> >> > >Currently, we have different business groups, we want to assign >>> >> different >>> >> > >mr job queue to different businiess user groups inside ONLY ONE >>> >> > >kylin instance, do we have this feature? and as searched, I find >>> >> > >the >>> >> following >>> >> > >piece: >>> >> > > >>> >> > >public static final String KYLIN_MAP_JOB_QUEUE = >>> >>"mapred.job.queue.name >>> >> "; >>> >> > > >>> >> > > >>> >> > >If we can pass a parameter to indicate different users to use >>> >>different >>> >> mr >>> >> > >job queue when building their own jobs with updating the codes, a >>> >> question >>> >> > >is that is there any potential problems to conduct the action? >>> >> > >>> >> > >>> >> >>> >>> >>> >>> >>> >>> >>> ******************************************************************************************************************************** >>> The information in this email is confidential and may be legally >>> privileged. If you have received this email in error or are not the >>> intended recipient, please immediately notify the sender and delete this >>> message from your computer. Any use, distribution, or copying of this email >>> other than by the intended recipient is strictly prohibited. All messages >>> sent to and from us may be monitored to ensure compliance with internal >>> policies and to protect our business. >>> Emails are not secure and cannot be guaranteed to be error free as they >>> can be intercepted, amended, lost or destroyed, or contain viruses. Anyone >>> who communicates with us by email is taken to accept these risks. >>> >>> 收发邮件者请注意： >>> 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。 >>> 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。 >>> >>> ******************************************************************************************************************************** >>> >>> >>> >>> >>> >>> ******************************************************************************************************************************** >>> The information in this email is confidential and may be legally >>> privileged. If you have received this email in error or are not the >>> intended recipient, please immediately notify the sender and delete this >>> message from your computer. Any use, distribution, or copying of this email >>> other than by the intended recipient is strictly prohibited. All messages >>> sent to and from us may be monitored to ensure compliance with internal >>> policies and to protect our business. >>> Emails are not secure and cannot be guaranteed to be error free as they >>> can be intercepted, amended, lost or destroyed, or contain viruses. Anyone >>> who communicates with us by email is taken to accept these risks. >>> >>> 收发邮件者请注意： >>> 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。 >>> 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。 >>> >>> ******************************************************************************************************************************** >>> >>> >>> >>> >>> >>> ******************************************************************************************************************************** >>> The information in this email is confidential and may be legally >>> privileged. If you have received this email in error or are not the >>> intended recipient, please immediately notify the sender and delete this >>> message from your computer. Any use, distribution, or copying of this email >>> other than by the intended recipient is strictly prohibited. All messages >>> sent to and from us may be monitored to ensure compliance with internal >>> policies and to protect our business. >>> Emails are not secure and cannot be guaranteed to be error free as they >>> can be intercepted, amended, lost or destroyed, or contain viruses. Anyone >>> who communicates with us by email is taken to accept these risks. >>> >>> 收发邮件者请注意： >>> 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。 >>> 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。 >>> >>> ******************************************************************************************************************************** >>> >> >> >> >> -- >> Regards, >> >> *Bin Mahone | 马洪宾* >> Apache Kylin: http://kylin.io >> Github: https://github.com/binmahone >> > > > > -- > Regards, > > *Bin Mahone | 马洪宾* > Apache Kylin: http://kylin.io > Github: https://github.com/binmahone > -- Regards, *Bin Mahone | 马洪宾* Apache Kylin: http://kylin.io Github: https://github.com/binmahone
