This is a feature not very well documented, and it is a global solution(you cannot specify different queue for different jobs within the same Kylin instance)
@Hua, does it solve you problem? On Wed, Aug 12, 2015 at 2:42 PM, Shi, Shaofeng <[email protected]> wrote: > Hi Hua, > > We have the same requirement as you in eBay internal deployment, and our > solution is adding such hadoop property in the > $KYLIN_HOME/conf/kylin_job_conf.xml, like: > > <property> > <name>mapreduce.job.queuename</name> > <value>queue-name</value> > <description>Job queue</description> > </property> > > The properties in this xml will be applied when running hive command and > MR jobs; > > > From 0.8, we separate hive related to another file called > ³kylin_hive_conf.xml², which will only be applied when running hive > command: > https://github.com/apache/incubator-kylin/blob/0.8/conf/kylin_hive_conf.xml > > > Basically, we don¹t want to add such hadoop configurations to > kylin.properties; kylin.properties is for Kylin specific; > > Just let me know if it answers your question; > > > On 8/12/15, 2:25 PM, "Huang Hua" <[email protected]> wrote: > > >Our hadoop cluster has multiple YARN execution queues for running Hadoop > >jobs(like MR, SPARK) at different resource capacity. > > > > > > > >But the current implementation of IntermediateHiveTableStep doesn't have > >option for users to specify the YARN queue, > > > >which basically runs the "hive -e" command in the *DEFAULT* queue. > >Unfortunately, *DEFAULT* queue might not have enough resource configured. > > > > > > > >I think it would be great to allow user specify the running queue for > >KYLIN > >jobs, and as far as I know it can be accomplished easily: > > > >1. In kylin.properties, specify the MR arugment like > >"kylin.job.cmd.extra.args=-D mapreduce.job.queuename=your_yarn_queue" > > > >2. Modify the KylinConfig to add an option of YARN queue > > > >3. Modify the createIntermediateHiveTableStep method of AbstractJobBuilder > >to append "SET mapreduce.job.queuename=your_yarn_queue" to the "hive -e" > >command > > > >For step 2 & 3, it only needs a little bit of coding. > > > > > > > >I am not sure if the above approach is the best way of doing it, so I > >would > >like to hear the opinions from KYLIN community. > > > > > > > > > >Thanks, > > > >Hua > > > > -- Regards, *Bin Mahone | 马洪宾* Apache Kylin: http://kylin.io Github: https://github.com/binmahone
