Bobby - MR AM also checks the priority of the allocated container… so we should be safe.
On Jun 19, 2013, at 8:31 AM, Robert Evans <[email protected]> wrote: > Sorry I am a bit behind on some of the changes that have happened to the > scheduler. And I guess I am also behind on what has happened to the MR > AM. I just looked at the MR AM code and it goes only off of priority when > assigning containers. It also does a sanity check that the memory > allocated is large enough to meet the needs of the given task. > > So I am happy to say I was wrong on this one :) > > Sorry I wasted your time, and thanks for the help. > > --Bobby > > On 6/18/13 12:59 PM, "Alejandro Abdelnur" <[email protected]> wrote: > >> Bobby, >> >> With MAPREDUCE-5310 we removed normalization of resource request on the >> MRAM side. This was done because the normalization is an implementation >> detail of the RM scheduler. >> >> IMO, if this is a problem for the MRAM as you suggest, then we should fix >> the MRAM logic. >> >> Note this may happen only the MR job specifies memory requirements for its >> tasks that do not much with its normalize value. >> >> Thanks. >> >> >> >> On Tue, Jun 18, 2013 at 10:45 AM, Robert Evans <[email protected]> >> wrote: >> >>> Even returning an over sized container can be very confusing for an >>> application. The MR AM will not handle it correctly. If it sees a >>> container returned that does not match exactly the priority and size it >>> expects, I believe that container is thrown away. We had deadlocks in >>> the >>> past where it somehow used a reducer container for a mapper and then >>> never >>> updated the reducer count to request a new one. It is best for now to >>> not >>> mix the two, and we need to lock down/fix the semantics of what happens >>> in >>> those situations for a scheduler. >>> >>> --Bobby >>> >>> On 6/18/13 12:13 AM, "Bikas Saha" <[email protected]> wrote: >>> >>>> I think the API allows different size requests at the same priority. >>> The >>>> implementation of the scheduler drops the size information and uses the >>>> last value set. We should probably at least change it to use the >>> largest >>>> value used so that users don't get containers that are too small for >>> them. >>>> YARN-847 tracks this. >>>> >>>> Bikas >>>> >>>> -----Original Message----- >>>> From: Robert Evans [mailto:[email protected]] >>>> Sent: Friday, June 14, 2013 7:09 AM >>>> To: [email protected] >>>> Subject: Re: Container size configuration >>>> >>>> Is this specifically for YARN? If so, yes you can do this, MR does >>> this >>>> for Maps vs Reduces. The API right now requires that the different >>> sized >>>> containers have a different priority. >>>> >>>> >>> >>> http://hadoop.apache.org/docs/r2.0.5-alpha/hadoop-yarn/hadoop-yarn-site/W >>> r >>>> i >>>> tingYarnApplications.html >>>> >>>> Shows how to make a resource request. It also shows how to make a >>>> AllocateRequest. If you put in multiple ResourceRequests into the >>>> AllocateRequest it will allocate both of them. But remember that that >>> the >>>> priority needs to be different, and the priority determines the order >>> in >>>> which the containers will be allocated to your application. >>>> >>>> --Bobby >>>> >>>> On 6/13/13 10:41 AM, "Yuzhang Han" <[email protected]> wrote: >>>> >>>>> Hi, >>>>> >>>>> I am wondering if I can allocate different size of containers to the >>>>> tasks in a job. For example: Job = <Task1, Task2, Task3>, Task1 = >>> Task2 >>>>> = 1024MB, Task3 = 2048MB. How can I achieve this? Many thanks. >>>>> >>>>> Yuzhang >>> >>> >> >> >> -- >> Alejandro > -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/
