[
https://issues.apache.org/jira/browse/HADOOP-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12608449#action_12608449
]
Tom White commented on HADOOP-3412:
-----------------------------------
bq. My argument in favor of keeping that in the AbstractJobScheduler is that,
when I implemented other schedulers to test the API (just before v6), this part
enforcing limitations was common to all of them. The reason is that those
options keeps their meaning with all schedulers.
Thinking about this some more, perhaps a better way of achieving this would be
to make things more composable.
How about breaking apart the responsibilities, so there is a class for choosing
the next task to run - call it TaskScheduler (which has the functionality of
AbstractJobScheduler), and a class for prioritizing jobs: JobQueue (which is
basically FifoJobScheduler). This would permit mixing of different JobQueues
with different TaskSchedulers. This would not preclude more complex
TaskSchedulers that have their own specialized JobQueue when it's not possible
to sort jobs before choosing a task.
{code}
public abstract class JobQueue implements Iterable<JobInProgress> {
public abstract void add(JobInProgress job);
public abstract Iterator<JobInProgress> iterator();
public abstract void remove(JobInProgress job);
}
public abstract class TaskScheduler extends Configured {
private JobQueue jobQueue;
public TaskScheduler() {
// jobQueue specified by configuration
}
/**
* For subclasses that want to provide their own JobQueue
*/
protected TaskScheduler(JobQueue jobQueue) {
this.jobQueue = jobQueue;
}
public void setConf(Configuration conf) {
super.setConf(conf);
// initialize jobQueue here
}
public JobQueue getJobQueue() {
return jobQueue;
}
public abstract Task assignTask(String taskTrackerName) throws IOException;
public abstract boolean updateTaskTrackerStatus(String taskTrackerName,
TaskTrackerStatus newStatus);
// other methods...
}
{code}
Then I would have FifoJobQueue, and two TaskSchedulers: DefaultTaskScheduler
(which implements the current behaviour) and JobLimitedTaskScheduler both using
FifoJobQueue by default.
Thoughts?
A couple of other points:
* I'm not sure we need the getLockOnJobs method. As long as JobTracker is the
only client of JobScheduler then it can manage its access to JobScheduler using
external synchronization. Put another way, why it is not sufficient for
JobScheduler to be thread-safe when it comes to adding and removing jobs?
* Single line blocks need curly braces.
> Refactor the scheduler out of the JobTracker
> --------------------------------------------
>
> Key: HADOOP-3412
> URL: https://issues.apache.org/jira/browse/HADOOP-3412
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Brice Arnould
> Assignee: Brice Arnould
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: JobScheduler.patch, JobScheduler_v2.patch,
> JobScheduler_v3.patch, JobScheduler_v3b.patch, JobScheduler_v4.patch,
> JobScheduler_v5.patch, JobScheduler_v6.1.patch, JobScheduler_v6.2.patch,
> JobScheduler_v6.3.patch, JobScheduler_v6.4.patch, JobScheduler_v6.patch,
> RackAwareJobScheduler.java, SimpleResourceAwareJobScheduler.java
>
>
> First I would like warn you that my proposition is assumed to be very naive.
> I just hope that reading it won't make you lose time.
> h4. The aim
> It seems to me that improving Hadoop scheduling could be very profitable.
> But, it is hard to implement and compare schedulers, because the scheduling
> logic is mixed within the rest of the JobTracker.
> This bug is the first step of an attempt to improve the Hadoop scheduler. It
> re-implements the current scheduling algorithm in a separate class called
> JobScheduler. This new class is instantiated in the JobTracker.
> h4. Bug fixed as a side effects
> This patch probably cannot be submited as it is.
> A first difficulty is that it does not have exactly the same behaviour than
> the current JobTracker. More precisely, it doesn't re-implement things like
> code that seems to be never called or concurency problems.
> I wrote TOCONFIRM where my proposition differ from the current
> implementation, so you can find them easily.
> I know that fixing bugs silently is bad. So, independently of what you decide
> about this patch, I will open issues for bugs that you confirm.
> h4. Other side effects
> Another side effect of this patch is to add documentation about each step of
> the scheduling. I hope that it will help future improvement by lowering the
> level required to contribute to the scheduler.
> It also reduces the complexity and the granularity of the JobTracker (making
> it more parallel).
> h4. The future
> If you feel that this is a step the right direction, I will try to propose a
> JobSchedulerInterface that many JobSchedulers could implement and to propose
> alternatives to the current « FifoJobScheduler ». If some of you have ideas
> about that please tell ^^ I will also open issues for things marked as FIXME
> in the patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
