why your error at step 2 changed from IncompatibleClassChangeError to ClassNotFoundException? please provide as much info as possible, otherwise we won't be able to help much.
btw, how's your hive compiled? it can compile against both hadoop1 and hadoop2 On Thu, Dec 24, 2015 at 9:12 AM, 胡志华(万里通科技及数据中心商务智能团队数据分析组) < [email protected]> wrote: > So, where can I find the action at step 2? > > > From kylin.job , I just find below > > > /***************************************/ > INSERT OVERWRITE TABLE > kylin_intermediate_kylin_sales_cube_desc_19700101000000_20151216000000_683cd765_3c9f_40e2_9130_a1017b80b400 > SELECT > KYLIN_SALES.PART_DT > ,KYLIN_SALES.LEAF_CATEG_ID > ,KYLIN_SALES.LSTG_SITE_ID > ,KYLIN_CATEGORY_GROUPINGS.META_CATEG_NAME > ,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL2_NAME > ,KYLIN_CATEGORY_GROUPINGS.CATEG_LVL3_NAME > ,KYLIN_SALES.LSTG_FORMAT_NAME > ,KYLIN_SALES.PRICE > ,KYLIN_SALES.SELLER_ID > FROM DEFAULT.KYLIN_SALES as KYLIN_SALES > INNER JOIN DEFAULT.KYLIN_CAL_DT as KYLIN_CAL_DT > ON KYLIN_SALES.PART_DT = KYLIN_CAL_DT.CAL_DT > INNER JOIN DEFAULT.KYLIN_CATEGORY_GROUPINGS as KYLIN_CATEGORY_GROUPINGS > ON KYLIN_SALES.LEAF_CATEG_ID = KYLIN_CATEGORY_GROUPINGS.LEAF_CATEG_ID AND > KYLIN_SALES.LSTG_SITE_ID = KYLIN_CATEGORY_GROUPINGS.SITE_ID > WHERE (KYLIN_SALES.PART_DT < '2015-12-16') > ; > > " > [pool-6-thread-1]:[2015-12-23 > 16:58:43,089][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:112)] > - Job Fetcher: 0 running, 1 actual running, 1 ready, 11 others > [http-bio-7070-exec-3]:[2015-12-23 > 16:58:56,093][DEBUG][org.apache.kylin.rest.service.AdminService.getConfigAsString(AdminService.java:90)] > - Get Kylin Runtime Config > [http-bio-7070-exec-5]:[2015-12-23 > 16:58:56,095][DEBUG][org.apache.kylin.rest.filter.KylinApiFilter.logRequest(KylinApiFilter.java:120)] > - REQUEST: REQUESTER=ADMIN;REQ_TIME=GMT-08:00 2015-12-23 > 00:58:56;URI=/kylin/api/projects;METHOD=GET;QUERY_STRING=null;PAYLOAD=;RESP_STATUS=200; > [http-bio-7070-exec-2]:[2015-12-23 > 16:58:56,099][DEBUG][org.apache.kylin.rest.filter.KylinApiFilter.logRequest(KylinApiFilter.java:120)] > - REQUEST: REQUESTER=ADMIN;REQ_TIME=GMT-08:00 2015-12-23 > 00:58:56;URI=/kylin/api/user/authentication;METHOD=GET;QUERY_STRING=null;PAYLOAD=;RESP_STATUS=200; > [http-bio-7070-exec-3]:[2015-12-23 > 16:58:56,099][DEBUG][org.apache.kylin.rest.filter.KylinApiFilter.logRequest(KylinApiFilter.java:120)] > - REQUEST: REQUESTER=ADMIN;REQ_TIME=GMT-08:00 2015-12-23 > 00:58:56;URI=/kylin/api/admin/config;METHOD=GET;QUERY_STRING=null;PAYLOAD=;RESP_STATUS=200; > [http-bio-7070-exec-2]:[2015-12-23 > 16:58:56,357][DEBUG][org.apache.kylin.rest.filter.KylinApiFilter.logRequest(KylinApiFilter.java:120)] > - REQUEST: REQUESTER=ADMIN;REQ_TIME=GMT-08:00 2015-12-23 > 00:58:56;URI=/kylin/api/jobs;METHOD=GET;QUERY_STRING=limit=15&offset=0&projectName=learn_kylin;PAYLOAD=;RESP_STATUS=200; > [pool-7-thread-1]:[2015-12-23 > 16:59:23,032][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)] > - Saving resource /execute_output/683cd765-3c9f-40e2-9130-a1017b80b400-00 > (Store kylin_metadata@hbase) > [pool-7-thread-1]:[2015-12-23 > 16:59:23,044][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)] > - Saving resource /execute_output/683cd765-3c9f-40e2-9130-a1017b80b400-00 > (Store kylin_metadata@hbase) > > > /***************************************/ > > > But hadoop mapreduce failed as below > > Job Name: Kylin_Fact_Distinct_Columns_kylin_sales_cube_Step > User Name: root > Queue: root.default > State: FAILED > Uberized: false > Submitted: Wed Dec 23 16:59:29 CST 2015 > Started: Wed Dec 23 16:59:34 CST 2015 > Finished: Wed Dec 23 16:59:55 CST 2015 > Elapsed: 20sec > Diagnostics: > Task failed task_1449675076057_0024_m_000000 > Job failed as tasks failed. failedMaps:1 failedReduces:0 > ApplicationMaster > Attempt Number Start Time Node Logs > 1 Wed Dec 23 16:59:31 CST 2015 dw-server20-34.wlt:8042 logs > > > > > > > > -----邮件原件----- > 发件人: hongbin ma [mailto:[email protected]] > 发送时间: 2015年12月22日 16:34 > 收件人: [email protected] > 主题: Re: 答复: 回复：答复: building cube , stop at step 2,help > > btw, please prefer to use English, apache is a global community > > On Tue, Dec 22, 2015 at 4:33 PM, hongbin ma <[email protected]> wrote: > > > hi, how's your hive built? is it compiled against hadoop 1.x? > > JobContext was a class in hadoop 1.x, but became an interface in > > hadoop 2.x > > > > On Tue, Dec 22, 2015 at 4:23 PM, 胡志华(万里通科技及数据中心商务智能团队数据分析组) < > > [email protected]> wrote: > > > >> 是的，我的很有可能是版本问题 > >> > >> -----邮件原件----- > >> 发件人: TTS2沉默天使 [mailto:[email protected]] > >> 发送时间: 2015年12月22日 16:20 > >> 收件人: dev > >> 主题: 回复：答复: building cube , stop at step 2,help > >> > >> 看了下错误 你的错误和我的不一样 但是都是第二步 > >> > >> > >> > >> > >> ------------------ 原始邮件 ------------------ > >> 发件人: " 胡志华(万里通科技及数据中心商务智能团队数据分析组)";<[email protected]>; > >> 发送时间: 2015年12月22日(星期二) 下午4:17 > >> 收件人: "'[email protected]'"<[email protected]>; > >> > >> 主题: 答复: building cube , stop at step 2,help > >> > >> > >> > >> Thanks, you mean my hive version is incompatible with hadoop2.7.1 > >> > >> My hive is pre-build binary package, so I will check the hadoop > >> dependecy > >> > >> > >> -- > >> ---邮件原件----- > >> 发件人: [email protected] [mailto:[email protected]] > >> 发送时间: 2015年12月22日 16:02 > >> 收件人: dev > >> 主题: Re: building cube , stop at step 2,help > >> > >> Hi, > >> I think there could be a hadoop version mismatch in your binary package. > >> If you use of hive and hbase, maybe the default hadoop dependency is > >> not as latest as your current use 2.7.1 , thus may lead to the > >> IncompatibleClassChangeError. > >> > >> One possible workaround is to recompile and package your hive and > >> hbase from souce code with targeting hadoop version to 2.7.1. > >> > >> > >> > >> [email protected] > >> > >> From: 胡志华(万里通科技及数据中心商务智能团队数据分析组) > >> Date: 2015-12-22 15:54 > >> To: '[email protected]' > >> Subject: building cube , stop at step 2,help Hi, > >> > >> I downloaded “apache-kylin-1.1.1-incubating-bin”, and use > >> sample.sh to build a cube “kylin_sales_cube”, when I started the job > >> , it stoped at step 2, with error > >> > >> java.lang.IncompatibleClassChangeError: Found interface > >> org.apache.hadoop.mapreduce.JobContext, but class was expected > >> at > >> > org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:102) > >> at > >> > org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301) > >> at > >> > org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318) > >> at > >> > org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196) > >> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) > >> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) > >> at java.security.AccessController.doPrivileged(Native Method) > >> at javax.security.auth.Subject.doAs(Subject.java:415) > >> at > >> > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) > >> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) > >> at > >> > org.apache.kylin.job.hadoop.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:120) > >> at > >> > org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:83) > >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) > >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) > >> at > >> > org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113) > >> at > >> > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) > >> at > >> > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51) > >> at > >> > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107) > >> at > >> > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130) > >> at > >> > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > >> at > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > >> at java.lang.Thread.run(Thread.java:745) > >> > >> My environment lists as following > >> > >> Hadoop2.7.1 > >> Hbase0.98.13 > >> Hive0.13.1 > >> > >> So ,from the error code above, my hive environment is > >> incompatible? And what does the parameter mean at step 2? > >> > >> -conf > >> /home/hadoop/apache-kylin-1.1.1-incubating-bin/conf/kylin_job_conf.xm > >> l > >> -cubename kylin_sales_cube -output > >> /kylin/kylin_metadata/kylin-79ec4e09-9128-42bd-a455-9dcf3c16a42a/kyli > >> n_sales_cube/fact_distinct_columns > >> -jobname Kylin_Fact_Distinct_Columns_kylin_sales_cube_Step -tablename > >> default.kylin_intermediate_kylin_sales_cube_desc_19700101000000_20151 > >> 217000000_79ec4e09_9128_42bd_a455_9dcf3c16a42a > >> > >> > >> -----邮件原件----- > >> 发件人: Shi, Shaofeng [mailto:[email protected]] > >> 发送时间: 2015年12月22日 15:39 > >> 收件人: [email protected] > >> 主题: Re: about assigning different mr job queue to different user > >> groups inside one kylin instances > >> > >> Thanks Dong; I put a comment in your JIRA: > >> https://issues.apache.org/jira/browse/KYLIN-1241, we can discuss there. > >> > >> On 12/21/15, 10:29 AM, "dong wang" <[email protected]> wrote: > >> > >> >I have added the logic ( https://github.com/apache/kylin/pull/17 ) > >> >to auto detect the kylin_job_conf_medium/small/large_projectname.xml > >> >first, then kylin_job_conf_projectname.xml, then > >> >kylin_job_conf_medium/small/large.xml, > >> >finally kylin_job_conf.xml, if still not found, throws an exception > >> > > >> >2015-12-14 16:22 GMT+08:00 yu feng <[email protected]>: > >> > > >> >> Maybe a quick solution is just create a config file for every > >> >>project named ''kylin_job_conf_${projectname}.xml", it will just > >> >>modify code that selecting config file for a MR job, and you can > >> >>control every MR config property in project level. > >> >> > >> >> 2015-12-11 22:00 GMT+08:00 Shi, Shaofeng <[email protected]>: > >> >> > >> >> > This is a valid scenario; So far Kylin doesn¹t have project or > >> >> > cube > >> >>level > >> >> > job configurations, if it be implemented, you problem will be > solved. > >> >> > > >> >> > On 12/11/15, 7:09 PM, "dong wang" <[email protected]> wrote: > >> >> > > >> >> > >Currently, we have different business groups, we want to assign > >> >> different > >> >> > >mr job queue to different businiess user groups inside ONLY ONE > >> >> > >kylin instance, do we have this feature? and as searched, I > >> >> > >find the > >> >> following > >> >> > >piece: > >> >> > > > >> >> > >public static final String KYLIN_MAP_JOB_QUEUE = > >> >>"mapred.job.queue.name > >> >> "; > >> >> > > > >> >> > > > >> >> > >If we can pass a parameter to indicate different users to use > >> >>different > >> >> mr > >> >> > >job queue when building their own jobs with updating the codes, > >> >> > >a > >> >> question > >> >> > >is that is there any potential problems to conduct the action? > >> >> > > >> >> > > >> >> > >> > >> > >> > >> > >> > >> > >> ********************************************************************* > >> *********************************************************** > >> The information in this email is confidential and may be legally > >> privileged. If you have received this email in error or are not the > >> intended recipient, please immediately notify the sender and delete > >> this message from your computer. Any use, distribution, or copying of > >> this email other than by the intended recipient is strictly > >> prohibited. All messages sent to and from us may be monitored to > >> ensure compliance with internal policies and to protect our business. > >> Emails are not secure and cannot be guaranteed to be error free as > >> they can be intercepted, amended, lost or destroyed, or contain > >> viruses. Anyone who communicates with us by email is taken to accept > these risks. > >> > >> 收发邮件者请注意： > >> 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。 > >> 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。 > >> > >> ********************************************************************* > >> *********************************************************** > >> > >> > >> > >> > >> > >> ********************************************************************* > >> *********************************************************** > >> The information in this email is confidential and may be legally > >> privileged. If you have received this email in error or are not the > >> intended recipient, please immediately notify the sender and delete > >> this message from your computer. Any use, distribution, or copying of > >> this email other than by the intended recipient is strictly > >> prohibited. All messages sent to and from us may be monitored to > >> ensure compliance with internal policies and to protect our business. > >> Emails are not secure and cannot be guaranteed to be error free as > >> they can be intercepted, amended, lost or destroyed, or contain > >> viruses. Anyone who communicates with us by email is taken to accept > these risks. > >> > >> 收发邮件者请注意： > >> 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。 > >> 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。 > >> > >> ********************************************************************* > >> *********************************************************** > >> > >> > >> > >> > >> > >> ********************************************************************* > >> *********************************************************** > >> The information in this email is confidential and may be legally > >> privileged. If you have received this email in error or are not the > >> intended recipient, please immediately notify the sender and delete > >> this message from your computer. Any use, distribution, or copying of > >> this email other than by the intended recipient is strictly > >> prohibited. All messages sent to and from us may be monitored to > >> ensure compliance with internal policies and to protect our business. > >> Emails are not secure and cannot be guaranteed to be error free as > >> they can be intercepted, amended, lost or destroyed, or contain > >> viruses. Anyone who communicates with us by email is taken to accept > these risks. > >> > >> 收发邮件者请注意： > >> 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。 > >> 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。 > >> > >> ********************************************************************* > >> *********************************************************** > >> > > > > > > > > -- > > Regards, > > > > *Bin Mahone | 马洪宾* > > Apache Kylin: http://kylin.io > > Github: https://github.com/binmahone > > > > > > -- > Regards, > > *Bin Mahone | 马洪宾* > Apache Kylin: http://kylin.io > Github: https://github.com/binmahone > > > > > > ******************************************************************************************************************************** > The information in this email is confidential and may be legally > privileged. If you have received this email in error or are not the > intended recipient, please immediately notify the sender and delete this > message from your computer. Any use, distribution, or copying of this email > other than by the intended recipient is strictly prohibited. All messages > sent to and from us may be monitored to ensure compliance with internal > policies and to protect our business. > Emails are not secure and cannot be guaranteed to be error free as they > can be intercepted, amended, lost or destroyed, or contain viruses. Anyone > who communicates with us by email is taken to accept these risks. > > 收发邮件者请注意： > 本邮件含保密信息，若误收本邮件，请务必通知发送人并直接删去，不得使用、传播或复制本邮件。 > 进出邮件均受到本公司合规监控。邮件可能发生被截留、被修改、丢失、被破坏或包含计算机病毒等不安全情况。 > > ******************************************************************************************************************************** > -- Regards, *Bin Mahone | 马洪宾* Apache Kylin: http://kylin.io Github: https://github.com/binmahone
