I'm not sure what you are trying to achieve here. Hadoop MapReduce works by *trying* to schedule tasks on nodes on which data is 'close', either node-local/rack-local.
We doesn't try to 'start'/'stop' nodes. If that is what you are trying to do, you need to look for something else. Arun On Dec 31, 2011, at 11:29 PM, mohak gupta wrote: > hi > > as part of my project I need to modify the data distribution layer in job > conf so as to achieve the following : > > 1) control which worker nodes should be started based on the input data > given to them. > > 2) keep other worker nodes in some kind of sleep state. > > 3) based on the output emitted by the worker nodes and the data distributed > allow other worker nodes to start . > > 4) Perform this in a looping structure till the output is achieved. > > basically I wish to control which worker nodes perform map and reduce > functions based on the data they have recieved. > > Could you please help me by suggesting if this could be achieved and also > what are the tradeoffs involved, Any help is really appreciated > > regards > Mohak Gupta
