Thanks for the details. I will put the yarn queue setting in kylin_job_conf.xml and test if it works.
@Hongbin, yes, I think it works for me as we normally configure only one dedicated queue for all jobs of a Kylin instance. Thanks, Hua > -----邮件原件----- > 发件人: dev-return-3632- > [email protected] [mailto:dev-return- > [email protected]] 代表 hongbin > ma > 发送时间: 2015年8月12日 14:50 > 收件人: dev > 主题: Re: Consider adding yarn queue option > > This is a feature not very well documented, and it is a global solution(you > cannot specify different queue for different jobs within the same Kylin > instance) > > @Hua, does it solve you problem? > > On Wed, Aug 12, 2015 at 2:42 PM, Shi, Shaofeng <[email protected]> wrote: > > > Hi Hua, > > > > We have the same requirement as you in eBay internal deployment, and > > our solution is adding such hadoop property in the > > $KYLIN_HOME/conf/kylin_job_conf.xml, like: > > > > <property> > > <name>mapreduce.job.queuename</name> > > <value>queue-name</value> > > <description>Job queue</description> > > </property> > > > > The properties in this xml will be applied when running hive command > > and MR jobs; > > > > > > From 0.8, we separate hive related to another file called > > ³kylin_hive_conf.xml², which will only be applied when running hive > > command: > > https://github.com/apache/incubator-kylin/blob/0.8/conf/kylin_hive_con > > f.xml > > > > > > Basically, we don¹t want to add such hadoop configurations to > > kylin.properties; kylin.properties is for Kylin specific; > > > > Just let me know if it answers your question; > > > > > > On 8/12/15, 2:25 PM, "Huang Hua" <[email protected]> wrote: > > > > >Our hadoop cluster has multiple YARN execution queues for running > > >Hadoop jobs(like MR, SPARK) at different resource capacity. > > > > > > > > > > > >But the current implementation of IntermediateHiveTableStep doesn't > > >have option for users to specify the YARN queue, > > > > > >which basically runs the "hive -e" command in the *DEFAULT* queue. > > >Unfortunately, *DEFAULT* queue might not have enough resource > configured. > > > > > > > > > > > >I think it would be great to allow user specify the running queue for > > >KYLIN jobs, and as far as I know it can be accomplished easily: > > > > > >1. In kylin.properties, specify the MR arugment like > > >"kylin.job.cmd.extra.args=-D > mapreduce.job.queuename=your_yarn_queue" > > > > > >2. Modify the KylinConfig to add an option of YARN queue > > > > > >3. Modify the createIntermediateHiveTableStep method of > > >AbstractJobBuilder to append "SET > mapreduce.job.queuename=your_yarn_queue" to the "hive -e" > > >command > > > > > >For step 2 & 3, it only needs a little bit of coding. > > > > > > > > > > > >I am not sure if the above approach is the best way of doing it, so I > > >would like to hear the opinions from KYLIN community. > > > > > > > > > > > > > > >Thanks, > > > > > >Hua > > > > > > > > > > -- > Regards, > > *Bin Mahone | 马洪宾* > Apache Kylin: http://kylin.io > Github: https://github.com/binmahone
