Another option might be to extend fair scheduler's task selector. The code is already set up to make this a configurable abstraction point. One word of caution, if you are using the cloudera distro, you will find that fair scheduler has been modified to not use those classes anymore. Something that wasn't documented very well. Still works with 20.2. Not sure about 21.
On Thursday, October 7, 2010, Jeff Zhang <[email protected]> wrote: > Currently, TaskTracker send heatbeat to JobTracker, and JobTracker > will send actions to TaskTracker according TaskTracker's status. I > think you should do some hacking about the heatbeat part of JobTracker > > public synchronized HeartbeatResponse heartbeat(TaskTrackerStatus status, > boolean restarted, > boolean initialContact, > boolean acceptNewTasks, > short responseId) > > > > On Fri, Oct 8, 2010 at 10:38 AM, Shen LI <[email protected]> wrote: >> Hi, Thanks you very much for your reply. I want to run my own algorithm for >> this part to see if we can achieve better outcome in specific scenario. So >> how can I modify it? >> Thanks a lot! >> Shen >> >> On Thu, Oct 7, 2010 at 6:33 PM, Jeff Zhang <[email protected]> wrote: >>> >>> I believe it is possible. But what is your purpose? I believe current >>> solution is good enough. >>> >>> >>> >>> >>> >>> On Fri, Oct 8, 2010 at 2:57 AM, Shen LI <[email protected]> wrote: >>> > Hi, >>> > How can I modify the task assignment strategy in hadoop which is used to >>> > assign tasks to different worker nodes? (Not the job scheduler) >>> > Big thanks, >>> > Shen >>> >>> >>> >>> -- >>> Best Regards >>> >>> Jeff Zhang >> >> > > > > -- > Best Regards > > Jeff Zhang >
