Hi Brad
IMHO, here is how hadoop scheduler works
For Hadoop 1.x
In one heartbeat interval, Fifo Scheduler keep strict FIFO order and assign
a node as many as possible local task and only one remote task. For Fair
scheduler, if you turn on delay algorithm, it also assign local task as
many as pos
cation for
> > block locality? I haven't located the spot in source that reads the
> block
> > location file for scheduler is why i ask.
> >
> > -bc
> >
> >
> > ----- Original Message -
> > > From: "Sandy Ryza"
> >
Brad Childs writes:
>
> Sorry if this is the wrong list, i am looking for deep technical/hadoop
source help :)
>
> How does job scheduling work on yarn framework for map reduce jobs? I see
the yarn scheduler discussed here:
>
https://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/
block
> location file for scheduler is why i ask.
>
> -bc
>
>
> - Original Message -
> > From: "Sandy Ryza"
> > To: common-dev@hadoop.apache.org
> > Sent: Thursday, April 3, 2014 2:38:42 PM
> > Subject: Re: Yarn / mapreduce scheduling
> >
>
d the spot in source that reads the block location
file for scheduler is why i ask.
-bc
- Original Message -
> From: "Sandy Ryza"
> To: common-dev@hadoop.apache.org
> Sent: Thursday, April 3, 2014 2:38:42 PM
> Subject: Re: Yarn / mapreduce scheduling
>
> The
The equivalent code in the Fair Scheduler is in AppSchedulable.java,
under assignContainer(FSSchedulerNode node, boolean reserved).
YARN uses delay scheduling (
http://people.csail.mit.edu/matei/papers/2010/eurosys_delay_scheduling.pdf)
for achieving data-locality.
-Sandy
On Thu, Apr 3, 2014 at
Hi Brad,
YARN scheduling does take care of data locality. In YARN, tasks are not
assigned based on capacity. Actually certain number of containers are
allocated on every node based on node's capacity. Tasks are executed on
those containers. While scheduling tasks on containers YARN scheduler
s
Sorry if this is the wrong list, i am looking for deep technical/hadoop source
help :)
How does job scheduling work on yarn framework for map reduce jobs? I see the
yarn scheduler discussed here:
https://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/YARN.html
which leads me to