There's an auction/bid scheduler in the contrib library for Hadoop 2.20, I think you'd be better off working on getting that working with Hadoop 2.x, including the pre-emption feature, where someone bidding more can get lower-priced work killed.
https://github.com/apache/hadoop-common/tree/branch-0.21/mapreduce/src/contrib/dynamic-scheduler Trying to do fundamental changes to how the NM/RM system works is a major undertaking that I would not encourage you to attempt until you've spent a lot of time in the YARN codebase understanding how things glue together. At the very least, try writing your own YARN application with protobuf-based RPCs & get it working on a secure cluster, which I can assure you gets you into some of the issues. You seem to have got pretty far already, but there's still a lot left to go The scheduler plugin point is designed for third party schedulers, and you should focus on there: if there is information you need from the nodes, such as recent CPU, IO and RAM load, then those are enhancements that could be added to NM heartbeats -for the benefit of all schedulers. On 9 December 2013 20:43, santosh kumar <[email protected]> wrote: > Hi, > > I am trying to develop an auction based scheduling algorithm for YARN. I > have few questions about the same. > > Idea is to have an "Auction Manager" inside every "Node Manager" who > receive auction requests from "Resource Manager" for every resource request > from "Application Master". The cluster nodes are grouped such that each > rack has one seller and all other as participants. Based on few > parameters/metrics the seller selects a winning participant to serve the > resource request. > > Question 1: > -------------- > I added a new module for auctions in my local code repository and pretty > much covered most of the use cases. But I am stuck at understanding how > communication actually takes place between RM and NM or among NMs. I need > to include something on lines of send/recv commands. Like how the > map/reduce commands are given using some job.xml. > > Question 2: > -------------- > All my code additions are in: > hadoop-yarn-api; hadoop-yarn-common; hadoop-yarn-server-common; > hadoop-yarn-server-nodemanager; hadoop-yarn-server-resourcemanager > > Would I need other modules like hadoop-yarn-client etc? My guess is I don't > think so. > > Question 3: > -------------- > Did not really understand the Yarn-Protos. Any place where I can get some > decent documentation about this? > > > Any response/help is greatly appreciated. > > Thanks, > Santosh > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
