hi as part of my project I need to modify the data distribution layer in job conf so as to achieve the following :
1) control which worker nodes should be started based on the input data given to them. 2) keep other worker nodes in some kind of sleep state. 3) based on the output emitted by the worker nodes and the data distributed allow other worker nodes to start . 4) Perform this in a looping structure till the output is achieved. basically I wish to control which worker nodes perform map and reduce functions based on the data they have recieved. Could you please help me by suggesting if this could be achieved and also what are the tradeoffs involved, Any help is really appreciated regards Mohak Gupta
