Thanks for all the replies. I think I have found the relevant codes that I would like to modify. That said, a project that I'm doing now requires containers to have network bandwidth as one of its resources (In Resource.java: it currently only models memory).
Since I'm planning to implement it anyway, I hope to be able to help Hadoop's development. However, I could not find the relevant JIRA for this. If you know of an existing ticket that is relevant to the aforementioned issue, let me know. If there is none, should I make my changes first (as listed http://wiki.apache.org/hadoop/HowToContribute) and get back after I'm done with my code? Thanks in advance. On Fri, Sep 6, 2013 at 6:37 AM, Steve Loughran <[email protected]>wrote: > worth adding is that this can generate a bias towards affinitive > assignment of an apps containers; for the YARN-896 service we've put > anti-affinity as a subtask, along with having AM opt to get notifications > if assignments can't be met in a bounded period (or it could just examine > its queue of outstanding requests and reach the same conclusion based on > when the requests were submitted) > > > On 6 September 2013 07:43, Sandy Ryza <[email protected]> wrote: > > > That's right. Nodes keep checking in and, when they do, the > > ResourceManager looks for outstanding requests. This means that > assignment > > of containers to nodes depends on the order that they heartbeat in. If > > container requests come in for specific nodes locality is achieved > through > > delay scheduling - the ResourceManager will wait for a configurable > number > > of heartbeats before assigning a container to a non-local node. If > strict > > locality is turned on, the ResourceManager will wait indefinitely for a > > local node. > > > > -Sandy > > > > > > On Fri, Sep 6, 2013 at 3:33 PM, hilfi alkaff <[email protected]> > > wrote: > > > > > I see. What I'm wondering about is; when an application master tries to > > > request a container from resource manager, which part of the code in > the > > > resource manager actually decide which node to fetch this container > from. > > > Is this step being done asynchronously (ie: Nodes keep checking if > there > > > are requests from the ResourceManager during the node update event?) > > > > > > > > > On Fri, Sep 6, 2013 at 1:22 AM, Sandy Ryza <[email protected]> > > > wrote: > > > > > > > Hi Hilfi, > > > > > > > > Nodes are constantly heartbeating to the ResourceManager. A node > > update > > > > event is triggered each time this happens. > > > > > > > > -Sandy > > > > > > > > > > > > On Fri, Sep 6, 2013 at 3:20 PM, hilfi alkaff <[email protected]> > > > > wrote: > > > > > > > > > Hi, > > > > > > > > > > I'm trying to trace the code flow on the scheduling done in YARN. I > > > would > > > > > like to know where the code that does which node to schedule for > the > > > > jobs. > > > > > > > > > > I found the handle() function in the resource manager's scheduler > > (eg: > > > > > CapacityScheduler.java) that handles node update event which then > > > > executes > > > > > the assignment of containers for that particular node, but I do not > > > > > understand how that node even get chosen. > > > > > > > > > > If anybody could tell me about a file, function or module name that > > > does > > > > > this, that would be extremely helpful. > > > > > > > > > > -- > > > > > ~Hilfi Alkaff~ > > > > > > > > > > > > > > > > > > > > > -- > > > ~Hilfi Alkaff~ > > > > > > > -- > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader > of this message is not the intended recipient, you are hereby notified that > any printing, copying, dissemination, distribution, disclosure or > forwarding of this communication is strictly prohibited. If you have > received this communication in error, please contact the sender immediately > and delete it from your system. Thank You. > -- ~Hilfi Alkaff~
