Brad Childs writes:
>
> Sorry if this is the wrong list, i am looking for deep technical/hadoop
source help :)
>
> How does job scheduling work on yarn framework for map reduce jobs? I see
the yarn scheduler discussed here:
>
missing another call/method for determining node location for
> > block locality? I haven't located the spot in source that reads the
> block
> > location file for scheduler is why i ask.
> >
> > -bc
> >
> >
> > - Original Message -----
> > > Fro
Hi Brad
IMHO, here is how hadoop scheduler works
For Hadoop 1.x
In one heartbeat interval, Fifo Scheduler keep strict FIFO order and assign
a node as many as possible local task and only one remote task. For Fair
scheduler, if you turn on delay algorithm, it also assign local task as
many as
Sorry if this is the wrong list, i am looking for deep technical/hadoop source
help :)
How does job scheduling work on yarn framework for map reduce jobs? I see the
yarn scheduler discussed here:
https://hadoop.apache.org/docs/r2.2.0/hadoop-yarn/hadoop-yarn-site/YARN.html
which leads me to
Hi Brad,
YARN scheduling does take care of data locality. In YARN, tasks are not
assigned based on capacity. Actually certain number of containers are
allocated on every node based on node's capacity. Tasks are executed on
those containers. While scheduling tasks on containers YARN scheduler
The equivalent code in the Fair Scheduler is in AppSchedulable.java,
under assignContainer(FSSchedulerNode node, boolean reserved).
YARN uses delay scheduling (
http://people.csail.mit.edu/matei/papers/2010/eurosys_delay_scheduling.pdf)
for achieving data-locality.
-Sandy
On Thu, Apr 3, 2014
in source that reads the block location
file for scheduler is why i ask.
-bc
- Original Message -
From: Sandy Ryza sandy.r...@cloudera.com
To: common-dev@hadoop.apache.org
Sent: Thursday, April 3, 2014 2:38:42 PM
Subject: Re: Yarn / mapreduce scheduling
The equivalent code
- Original Message -
From: Sandy Ryza sandy.r...@cloudera.com
To: common-dev@hadoop.apache.org
Sent: Thursday, April 3, 2014 2:38:42 PM
Subject: Re: Yarn / mapreduce scheduling
The equivalent code in the Fair Scheduler is in AppSchedulable.java,
under assignContainer(FSSchedulerNode node