Hi Venu, The paper compares against a naive implementation of fair scheduling, not against the Hadoop Fair Scheduler. You can find out about the data locality algorithms used in the Fair Scheduler in http://www.cs.berkeley.edu/~matei/papers/2010/eurosys_delay_scheduling.pdf. These are available in 0.21+, and they have also been in various distributions of Hadoop (e.g. Cloudera's and Facebook's) for a while. I don't think they got backported into 0.20.203 though and I'm not sure what scheduling algorithm the next-gen team is doing for locality.
Matei On Jun 17, 2011, at 6:46 AM, Venu Gopala Rao wrote: > Hi All, > > > > I have come across a Fair Scheduler published by Microsoft known as > Quincy Fair SCheduler. In this they compare Hadoop Fair Scheduler with > Quincy and say the Hadoop Scheduler has the following problems > > > > 1) Sticky Slots - One drawback of simple fair scheduling is that it is > damaging to locality. Consider the steady state in which each job is > occupying exactly its allocated quota of computers. Whenever a task from job > j completes on computer m, j becomes unblocked but all of the other jobs in > the system remain blocked. Consequently m will be assigned to one of j's > tasks again: this is referred to as the "sticky slot" problem. > > > > 2) Fair Scheduler may not be able to utilize the Data locality to maximum > possible extent. > > > > Does these problems get solved in the 0.21 or Next Gen Map Reduce? > > > > Regards > > Venu > > > > > > >
