Thanks Harish, that makes sense. I will give it a try. Thanks, Dharmesh
On Tue, Dec 20, 2016 at 10:23 PM, Harish JP <[email protected]> wrote: > Dharmesh, > > The code Rajesh pointed out was to show how to use a custom scheduler. But > that is not a configuration option but something which has to be part of > code which creates the Tez client. You’ll not be able to customize the > scheduler without a code change in hive or tez. I would recommend you to > try to modify hive source and plugin your scheduler or modify the tez > source and use your class: https://github.com/apache/tez/ > blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/ > rm/TaskSchedulerManager.java#L550. But test it out and see if what you > asked for works and we can figure out integration: modify existing > scheduler, add configurable alternative to yarn scheduler, add configurable > default scheduler, etc ... > > — > Thanks, > Harish > > On 21-Dec-2016, at 9:33 AM, Dharmesh Kakadia <[email protected]<mailto: > [email protected]>> wrote: > > Thanks Harish. > I understand YARN allocates container, I am interested in what tasks to run > on those containers once it is available to tez task scheduler. > > @Rajesh I am interested in task scheduling on non-LLAP hive cluster. Is > this currently not possible ? > > Thanks, > Dharmesh > > On Tue, Dec 20, 2016 at 6:38 PM, Rajesh Balamohan <[email protected] > <mailto:[email protected]>> > wrote: > > E.g task scheduler in hive is llap. > > https://github.com/apache/hive/blob/master/llap-tez/src/ > java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java > > https://github.com/apache/hive/blob/master/ql/src/java/ > org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L289 > > https://github.com/apache/hive/blob/master/ql/src/java/ > org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L309 > (This is for Harish's point) > > ~Rajesh.B > > On Wed, Dec 21, 2016 at 7:46 AM, Harish JP <[email protected]> wrote: > > Hi Dharmesh, > > TaskScheduler is not picked up via config. It has to be setup in the > TezClient using setServicePluginDescriptor. And dag/vertex execution > context should be configured to use this task scheduler. > > What you ask is very different from what TaskScheduler in Tez does. The > YarnTaskSchedulerService does not allocate containers, this is actually > done by the Yarn RM. The yarn task scheduler only allocates the Tez task > to > these containers and also tries to reuse containers when possible. > > — > Thanks, > Harish > > On 21-Dec-2016, at 6:38 AM, Dharmesh Kakadia <[email protected]> > wrote: > > Hi, > > I am trying to implement a new tez task scheduler which tries to allocate > tasks uniformly across nodes. As part of this, I am trying to understand > how does tez decide which scheduler to use. I could not find any > configuration, My current understanding is if its running on yarntez > mode, > it will use YarnTaskSchedulerService, and if its uber mode it will use > LocalTaskSchedulerService. I m interested in using it with hive on tez. > > Please let me know what is the configuration for using a new scheduler. > > Thanks, > Dharmesh >
