hi, how's your hive built? is it compiled against hadoop 1.x? JobContext was a class in hadoop 1.x, but became an interface in hadoop 2.x
On Tue, Dec 22, 2015 at 4:23 PM, 胡志华(万里通科技及数据中心商务智能团队数据分析组) < [email protected]> wrote: > 是的，我的很有可能是版本问题 > > -----邮件原件----- > 发件人: TTS2沉默天使 [mailto:[email protected]] > 发送时间: 2015年12月22日 16:20 > 收件人: dev > 主题: 回复：答复: building cube , stop at step 2,help > > 看了下错误 你的错误和我的不一样 但是都是第二步 > > > > > ------------------ 原始邮件 ------------------ > 发件人: " 胡志华(万里通科技及数据中心商务智能团队数据分析组)";<[email protected]>; > 发送时间: 2015年12月22日(星期二) 下午4:17 > 收件人: "'[email protected]'"<[email protected]>; > > 主题: 答复: building cube , stop at step 2,help > > > > Thanks, you mean my hive version is incompatible with hadoop2.7.1 > > My hive is pre-build binary package, so I will check the hadoop dependecy > > > -- > ---邮件原件----- > 发件人: [email protected] [mailto:[email protected]] > 发送时间: 2015年12月22日 16:02 > 收件人: dev > 主题: Re: building cube , stop at step 2,help > > Hi, > I think there could be a hadoop version mismatch in your binary package. > If you use of hive and hbase, maybe the default hadoop dependency is not > as latest as your current use 2.7.1 , thus may lead to the > IncompatibleClassChangeError. > > One possible workaround is to recompile and package your hive and hbase > from souce code with targeting hadoop version to 2.7.1. > > > > [email protected] > > From: 胡志华(万里通科技及数据中心商务智能团队数据分析组) > Date: 2015-12-22 15:54 > To: '[email protected]' > Subject: building cube , stop at step 2,help Hi, > > I downloaded “apache-kylin-1.1.1-incubating-bin”, and use sample.sh > to build a cube “kylin_sales_cube”, when I started the job , it stoped at > step 2, with error > > java.lang.IncompatibleClassChangeError: Found interface > org.apache.hadoop.mapreduce.JobContext, but class was expected > at > org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:102) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301) > at > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318) > at > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) > at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) > at > org.apache.kylin.job.hadoop.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:120) > at > org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:83) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > at > org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > > My environment lists as following > > Hadoop2.7.1 > Hbase0.98.13 > Hive0.13.1 > > So ,from the error code above, my hive environment is incompatible? > And what does the parameter mean at step 2? > > -conf > /home/hadoop/apache-kylin-1.1.1-incubating-bin/conf/kylin_job_conf.xml > -cubename kylin_sales_cube -output > /kylin/kylin_metadata/kylin-79ec4e09-9128-42bd-a455-9dcf3c16a42a/kylin_sales_cube/fact_distinct_columns > -jobname Kylin_Fact_Distinct_Columns_kylin_sales_cube_Step -tablename > default.kylin_intermediate_kylin_sales_cube_desc_19700101000000_20151217000000_79ec4e09_9128_42bd_a455_9dcf3c16a42a > > > -----邮件原件----- > 发件人: Shi, Shaofeng [mailto:[email protected]] > 发送时间: 2015年12月22日 15:39 > 收件人: [email protected] > 主题: Re: about assigning different mr job queue to different user groups > inside one kylin instances > > Thanks Dong; I put a comment in your JIRA: > https://issues.apache.org/jira/browse/KYLIN-1241, we can discuss there. > > On 12/21/15, 10:29 AM, "dong wang" <[email protected]> wrote: > > >I have added the logic ( https://github.com/apache/kylin/pull/17 ) to > >auto detect the kylin_job_conf_medium/small/large_projectname.xml > >first, then kylin_job_conf_projectname.xml, then > >kylin_job_conf_medium/small/large.xml, > >finally kylin_job_conf.xml, if still not found, throws an exception > > > >2015-12-14 16:22 GMT+08:00 yu feng <[email protected]>: > > > >> Maybe a quick solution is just create a config file for every project > >>named ''kylin_job_conf_${projectname}.xml", it will just modify code > >>that selecting config file for a MR job, and you can control every MR > >>config property in project level. > >> > >> 2015-12-11 22:00 GMT+08:00 Shi, Shaofeng <[email protected]>: > >> > >> > This is a valid scenario; So far Kylin doesn¹t have project or cube > >>level > >> > job configurations, if it be implemented, you problem will be solved. > >> > > >> > On 12/11/15, 7:09 PM, "dong wang" <[email protected]> wrote: > >> > > >> > >Currently, we have different business groups, we want to assign > >> different > >> > >mr job queue to different businiess user groups inside ONLY ONE > >> > >kylin instance, do we have this feature? and as searched, I find > >> > >the > >> following > >> > >piece: > >> > > > >> > >public static final String KYLIN_MAP_JOB_QUEUE = > >>"mapred.job.queue.name > >> "; > >> > > > >> > > > >> > >If we can pass a parameter to indicate different users to use > >>different > >> mr > >> > >job queue when building their own jobs with updating the codes, a > >> question > >> > >is that is there any potential problems to conduct the action? > >> > > >> > > >> > > > > > > > ******************************************************************************************************************************** > The information in this email is confidential and may be legally > privileged. If you have received this email in error or are not the > intended recipient, please immediately notify the sender and delete this > message from your computer. Any use, distribution, or copying of this email > other than by the intended recipient is strictly prohibited. All messages > sent to and from us may be monitored to ensure compliance with internal > policies and to protect our business. > Emails are not secure and cannot be guaranteed to be error free as they > can be intercepted, amended, lost or destroyed, or contain viruses. Anyone > who communicates with us by email is taken to accept these risks. > > 收发邮件者请注意： > 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。 > 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。 > > ******************************************************************************************************************************** > > > > > > ******************************************************************************************************************************** > The information in this email is confidential and may be legally > privileged. If you have received this email in error or are not the > intended recipient, please immediately notify the sender and delete this > message from your computer. Any use, distribution, or copying of this email > other than by the intended recipient is strictly prohibited. All messages > sent to and from us may be monitored to ensure compliance with internal > policies and to protect our business. > Emails are not secure and cannot be guaranteed to be error free as they > can be intercepted, amended, lost or destroyed, or contain viruses. Anyone > who communicates with us by email is taken to accept these risks. > > 收发邮件者请注意： > 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。 > 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。 > > ******************************************************************************************************************************** > > > > > > ******************************************************************************************************************************** > The information in this email is confidential and may be legally > privileged. If you have received this email in error or are not the > intended recipient, please immediately notify the sender and delete this > message from your computer. Any use, distribution, or copying of this email > other than by the intended recipient is strictly prohibited. All messages > sent to and from us may be monitored to ensure compliance with internal > policies and to protect our business. > Emails are not secure and cannot be guaranteed to be error free as they > can be intercepted, amended, lost or destroyed, or contain viruses. Anyone > who communicates with us by email is taken to accept these risks. > > 收发邮件者请注意： > 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。 > 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。 > > ******************************************************************************************************************************** > -- Regards, *Bin Mahone | 马洪宾* Apache Kylin: http://kylin.io Github: https://github.com/binmahone
