[
https://issues.apache.org/jira/browse/HADOOP-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613768#action_12613768
]
Owen O'Malley commented on HADOOP-3412:
---------------------------------------
Fair point about the JobInProgress being fine for the API, provided that the
scheduler is required to call initTasks on the JobInProgress when it should be
loaded.
On the other hand, we need
* an event when a TIP changes state, so that the scheduler can update its
data structures
* an api to the scheduler that breaks down the queues and order within the
queue, because the scheduler is the only place that has the order of jobs
within the queue.
So how does this look:
{code}
class JobInProgress {
...
String getQueueName();
Priority getPriority();
void initTasks();
}
class TaskScheduler {
void addJob(JobInProgress job) throws IOException;
void removeJob(JobInProgress job) throws IOException;
// the job has changed state
void updateJob(Progress job) throws IOException;
// the task (ie. map 0) has changed state
void updateTask(TaskInProgress tip) throws IOException;
// get a set of tasks for the given tracker
List<Task> assignTasks(TaskTrackerStatus taskTracker) throws IOException;
// get all of the queue names
List<String> getQueueNames();
// get an ordered list of the jobs in the given queue
List<JobDescription> getJobs(String queue) throws IOException;
}
{code}
I don't think we need updateTaskAttempt, because I can't see anything that a
potential scheduler would do with that fine of information.
I think that you are right that we want the JobTracker to keep track of the
running jobs and tasks, effectively owning the JobInProgress, TaskInProgress,
and Tasks and updating their state based on the task tracker reports.
In the medium term (ie. not this patch), this should only be for running and
pending jobs. Any finished jobs will need to be queried via JobHistory or a
similar interface.
> Refactor the scheduler out of the JobTracker
> --------------------------------------------
>
> Key: HADOOP-3412
> URL: https://issues.apache.org/jira/browse/HADOOP-3412
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Brice Arnould
> Assignee: Brice Arnould
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: JobScheduler-v10.patch, JobScheduler-v9.1.patch,
> JobScheduler-v9.2.patch, JobScheduler-v9.patch, JobScheduler.patch,
> JobScheduler_v2.patch, JobScheduler_v3.patch, JobScheduler_v3b.patch,
> JobScheduler_v4.patch, JobScheduler_v5.patch, JobScheduler_v6.1.patch,
> JobScheduler_v6.2.patch, JobScheduler_v6.3.patch, JobScheduler_v6.4.patch,
> JobScheduler_v6.patch, JobScheduler_v7.1.patch, JobScheduler_v7.patch,
> JobScheduler_v8.patch, RackAwareJobScheduler.java,
> SimpleResourceAwareJobScheduler.java
>
>
> First I would like warn you that my proposition is assumed to be very naive.
> I just hope that reading it won't make you lose time.
> h4. The aim
> It seems to me that improving Hadoop scheduling could be very profitable.
> But, it is hard to implement and compare schedulers, because the scheduling
> logic is mixed within the rest of the JobTracker.
> This bug is the first step of an attempt to improve the Hadoop scheduler. It
> re-implements the current scheduling algorithm in a separate class called
> JobScheduler. This new class is instantiated in the JobTracker.
> h4. Bug fixed as a side effects
> This patch probably cannot be submited as it is.
> A first difficulty is that it does not have exactly the same behaviour than
> the current JobTracker. More precisely, it doesn't re-implement things like
> code that seems to be never called or concurency problems.
> I wrote TOCONFIRM where my proposition differ from the current
> implementation, so you can find them easily.
> I know that fixing bugs silently is bad. So, independently of what you decide
> about this patch, I will open issues for bugs that you confirm.
> h4. Other side effects
> Another side effect of this patch is to add documentation about each step of
> the scheduling. I hope that it will help future improvement by lowering the
> level required to contribute to the scheduler.
> It also reduces the complexity and the granularity of the JobTracker (making
> it more parallel).
> h4. The future
> If you feel that this is a step the right direction, I will try to propose a
> JobSchedulerInterface that many JobSchedulers could implement and to propose
> alternatives to the current « FifoJobScheduler ». If some of you have ideas
> about that please tell ^^ I will also open issues for things marked as FIXME
> in the patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
