Sorry I am a bit behind on some of the changes that have happened to the scheduler. And I guess I am also behind on what has happened to the MR AM. I just looked at the MR AM code and it goes only off of priority when assigning containers. It also does a sanity check that the memory allocated is large enough to meet the needs of the given task.
So I am happy to say I was wrong on this one :) Sorry I wasted your time, and thanks for the help. --Bobby On 6/18/13 12:59 PM, "Alejandro Abdelnur" <[email protected]> wrote: >Bobby, > >With MAPREDUCE-5310 we removed normalization of resource request on the >MRAM side. This was done because the normalization is an implementation >detail of the RM scheduler. > >IMO, if this is a problem for the MRAM as you suggest, then we should fix >the MRAM logic. > >Note this may happen only the MR job specifies memory requirements for its >tasks that do not much with its normalize value. > >Thanks. > > > >On Tue, Jun 18, 2013 at 10:45 AM, Robert Evans <[email protected]> >wrote: > >> Even returning an over sized container can be very confusing for an >> application. The MR AM will not handle it correctly. If it sees a >> container returned that does not match exactly the priority and size it >> expects, I believe that container is thrown away. We had deadlocks in >>the >> past where it somehow used a reducer container for a mapper and then >>never >> updated the reducer count to request a new one. It is best for now to >>not >> mix the two, and we need to lock down/fix the semantics of what happens >>in >> those situations for a scheduler. >> >> --Bobby >> >> On 6/18/13 12:13 AM, "Bikas Saha" <[email protected]> wrote: >> >> >I think the API allows different size requests at the same priority. >>The >> >implementation of the scheduler drops the size information and uses the >> >last value set. We should probably at least change it to use the >>largest >> >value used so that users don't get containers that are too small for >>them. >> >YARN-847 tracks this. >> > >> >Bikas >> > >> >-----Original Message----- >> >From: Robert Evans [mailto:[email protected]] >> >Sent: Friday, June 14, 2013 7:09 AM >> >To: [email protected] >> >Subject: Re: Container size configuration >> > >> >Is this specifically for YARN? If so, yes you can do this, MR does >>this >> >for Maps vs Reduces. The API right now requires that the different >>sized >> >containers have a different priority. >> > >> > >> >>http://hadoop.apache.org/docs/r2.0.5-alpha/hadoop-yarn/hadoop-yarn-site/W >>r >> >i >> >tingYarnApplications.html >> > >> >Shows how to make a resource request. It also shows how to make a >> >AllocateRequest. If you put in multiple ResourceRequests into the >> >AllocateRequest it will allocate both of them. But remember that that >>the >> >priority needs to be different, and the priority determines the order >>in >> >which the containers will be allocated to your application. >> > >> >--Bobby >> > >> >On 6/13/13 10:41 AM, "Yuzhang Han" <[email protected]> wrote: >> > >> >>Hi, >> >> >> >>I am wondering if I can allocate different size of containers to the >> >>tasks in a job. For example: Job = <Task1, Task2, Task3>, Task1 = >>Task2 >> >>= 1024MB, Task3 = 2048MB. How can I achieve this? Many thanks. >> >> >> >>Yuzhang >> >> > > >-- >Alejandro
