Re: Yarn / mapreduce scheduling

2016-01-26 Thread Sultan
Brad Childs writes: > > Sorry if this is the wrong list, i am looking for deep technical/hadoop source help :) > > How does job scheduling work on yarn framework for map reduce jobs? I see the yarn scheduler discussed here: >

Re: Yarn / mapreduce scheduling

2016-01-26 Thread Chen He
missing another call/method for determining node location for > > block locality? I haven't located the spot in source that reads the > block > > location file for scheduler is why i ask. > > > > -bc > > > > > > - Original Message ----- > > > Fro

Re: Yarn / mapreduce scheduling

2016-01-26 Thread Chen He
Hi Brad IMHO, here is how hadoop scheduler works For Hadoop 1.x In one heartbeat interval, Fifo Scheduler keep strict FIFO order and assign a node as many as possible local task and only one remote task. For Fair scheduler, if you turn on delay algorithm, it also assign local task as many as

Yarn / mapreduce scheduling

2014-04-03 Thread Brad Childs
Sorry if this is the wrong list, i am looking for deep technical/hadoop source help :) How does job scheduling work on yarn framework for map reduce jobs? I see the yarn scheduler discussed here: https://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/YARN.html which leads me to

Re: Yarn / mapreduce scheduling

2014-04-03 Thread Shekhar Gupta
Hi Brad, YARN scheduling does take care of data locality. In YARN, tasks are not assigned based on capacity. Actually certain number of containers are allocated on every node based on node's capacity. Tasks are executed on those containers. While scheduling tasks on containers YARN scheduler

Re: Yarn / mapreduce scheduling

2014-04-03 Thread Sandy Ryza
The equivalent code in the Fair Scheduler is in AppSchedulable.java, under assignContainer(FSSchedulerNode node, boolean reserved). YARN uses delay scheduling ( http://people.csail.mit.edu/matei/papers/2010/eurosys_delay_scheduling.pdf) for achieving data-locality. -Sandy On Thu, Apr 3, 2014

Re: Yarn / mapreduce scheduling

2014-04-03 Thread Brad Childs
in source that reads the block location file for scheduler is why i ask. -bc - Original Message - From: Sandy Ryza sandy.r...@cloudera.com To: common-dev@hadoop.apache.org Sent: Thursday, April 3, 2014 2:38:42 PM Subject: Re: Yarn / mapreduce scheduling The equivalent code

Re: Yarn / mapreduce scheduling

2014-04-03 Thread Sandy Ryza
- Original Message - From: Sandy Ryza sandy.r...@cloudera.com To: common-dev@hadoop.apache.org Sent: Thursday, April 3, 2014 2:38:42 PM Subject: Re: Yarn / mapreduce scheduling The equivalent code in the Fair Scheduler is in AppSchedulable.java, under assignContainer(FSSchedulerNode node