Thanks Sandy, this worked. 2012/12/14 Sandy Ryza <[email protected]>
> Hi Lohit, > > Setting yarn.scheduler.fair.assignmultiple to false should give a more even > distribution of tasks. > > -Sandy > > On Fri, Dec 14, 2012 at 10:22 AM, lohit <[email protected]> > wrote: > > > I should have mentioned. I was trying this on FairScheduler. > > I can try CapacityScheduler and see if that works. > > But this is pretty easy to reproduce as I described in the problem. > > > > 2012/12/14 Harsh J <[email protected]> > > > > > Perhaps you can try YARN-80's > > > yarn.scheduler.capacity.node-locality-delay set to 1 for your > > > CapacityScheduler (are you using this or the Fifo)? > > > > > > On Fri, Dec 14, 2012 at 1:37 AM, lohit <[email protected]> > > wrote: > > > > I am running hadoop-2.0.3 alpha branch. > > > > Right now I am trying TestDFSIO, where I would like to have even > > > > distribution of tasks. > > > > As soon as I submit job RM tends to allocate as many containers as > > > possible > > > > from subset of nodes. > > > > > > > > 2012/12/13 Thomas Graves <[email protected]> > > > > > > > >> What kind of job are you running? What version of YARN? > > > >> > > > >> Tom > > > >> > > > >> On 12/13/12 12:24 AM, "lohit" <[email protected]> wrote: > > > >> > > > >> >Hi Devs, > > > >> > > > > >> >I am running few experiments on YARN. On a 50 node cluster I run > only > > > one > > > >> >job. > > > >> >If I kick off job with 150 tasks, I would expect on an average 3 > > tasks > > > in > > > >> >containers per node. > > > >> >But I see many nodes end up having 29-30 containers spawned and > many > > > >> >without any tasks at all. > > > >> >Do I have to use some kind of parameter (delayed scheduling) to > > achieve > > > >> >even distribution? > > > >> > > > > >> >-- > > > >> >Have a Nice Day! > > > >> >Lohit > > > >> > > > >> > > > > > > > > > > > > -- > > > > Have a Nice Day! > > > > Lohit > > > > > > > > > > > > -- > > > Harsh J > > > > > > > > > > > -- > > Have a Nice Day! > > Lohit > > > -- Have a Nice Day! Lohit
