Dharmesh, The code Rajesh pointed out was to show how to use a custom scheduler. But that is not a configuration option but something which has to be part of code which creates the Tez client. You’ll not be able to customize the scheduler without a code change in hive or tez. I would recommend you to try to modify hive source and plugin your scheduler or modify the tez source and use your class: https://github.com/apache/tez/blob/master/tez-dag/src/main/java/org/apache/tez/dag/app/rm/TaskSchedulerManager.java#L550. But test it out and see if what you asked for works and we can figure out integration: modify existing scheduler, add configurable alternative to yarn scheduler, add configurable default scheduler, etc ...
— Thanks, Harish On 21-Dec-2016, at 9:33 AM, Dharmesh Kakadia <[email protected]<mailto:[email protected]>> wrote: Thanks Harish. I understand YARN allocates container, I am interested in what tasks to run on those containers once it is available to tez task scheduler. @Rajesh I am interested in task scheduling on non-LLAP hive cluster. Is this currently not possible ? Thanks, Dharmesh On Tue, Dec 20, 2016 at 6:38 PM, Rajesh Balamohan <[email protected]<mailto:[email protected]>> wrote: E.g task scheduler in hive is llap. https://github.com/apache/hive/blob/master/llap-tez/src/ java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskSchedulerService.java https://github.com/apache/hive/blob/master/ql/src/java/ org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L289 https://github.com/apache/hive/blob/master/ql/src/java/ org/apache/hadoop/hive/ql/exec/tez/TezSessionState.java#L309 (This is for Harish's point) ~Rajesh.B On Wed, Dec 21, 2016 at 7:46 AM, Harish JP <[email protected]> wrote: Hi Dharmesh, TaskScheduler is not picked up via config. It has to be setup in the TezClient using setServicePluginDescriptor. And dag/vertex execution context should be configured to use this task scheduler. What you ask is very different from what TaskScheduler in Tez does. The YarnTaskSchedulerService does not allocate containers, this is actually done by the Yarn RM. The yarn task scheduler only allocates the Tez task to these containers and also tries to reuse containers when possible. — Thanks, Harish On 21-Dec-2016, at 6:38 AM, Dharmesh Kakadia <[email protected]> wrote: Hi, I am trying to implement a new tez task scheduler which tries to allocate tasks uniformly across nodes. As part of this, I am trying to understand how does tez decide which scheduler to use. I could not find any configuration, My current understanding is if its running on yarntez mode, it will use YarnTaskSchedulerService, and if its uber mode it will use LocalTaskSchedulerService. I m interested in using it with hive on tez. Please let me know what is the configuration for using a new scheduler. Thanks, Dharmesh
