[
https://issues.apache.org/jira/browse/HADOOP-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609994#action_12609994
]
Brice Arnould commented on HADOOP-3412:
---------------------------------------
bq. How does this look?
This TaskScheduler is almost identical to what the JobScheduler was, so it will
work. I'm still a bit bothered by the fact this modification to the API is not
motivated by the example of some real code. But since you both seems to agree
it's probably the right thing to do ;-).
DefaultTaskScheduler would benefit of a protected accessor to the JobQueue to
ease inheritance, and TaskScheduler need the getLockOnJobs method. But appart
from that it should be OK.
bq. Also, retireOldJobs seems a bit out of place here and should really go back
in JobTracker. This should be easy since it just calls getJobsIterator.
Ooops :-° You're right. I initally intended to handle this as part of
[HADOOP-3609] but I should also have reverted the creation of retireOldJobs in
this branch.
{quote}
>> We don't need iterator() and getSortedJobs() - iterator() is sufficient.
> getSortedJobs() allows to bias the choice of the job by the characteristics
> of the TaskTracker, something that appeared to be useful when I played with
> the API. This new proposition however provides a default implementation for
> it.
But the iterator returned by iterator() can be any iterator - so we can make it
the same one returned by getSortedJobs(). In other words, we only need one way
of iterating over the JobQueue.
{quote}
Some implementation may not be backed by a single sorted container. Iterating
on their jobs (for example when retiring jobs) should be possible without
requiring them to create a single sorted container of all their jobs. Moreover,
as I said before, the removal of getSortedJobs(trackerName) implies to loose
the hability to bias the choice of the job by the characteristics of the
TaskTracker.
The attached {{SimpleResourceAwareJobScheduler.java}} illustrate both points.
Another example would be a JobQueue in which users would reserve a few nodes
for their exclusive usage (maybe the best way to ensure that Tasks running on
the same machine do not interfere, is that what you were talking about Matei ?).
On the other hand, providing a default getSortedJobs() method adds only three
lines and no complexity for those who don't need it.
bq. I also like the idea of putting the statistics and tasktracker status stuff
back into JobTracker, and making the TaskScheduler a listener only.
Another concern with that approch is the complexity of the JobTracker.
IMHO statistics are not used by the JobTracker (it just pass them away) and so
should be kept inside the {Job,Task}Scheduler. Since it's a static class that
have no additional cost.
bq. I haven't implemented my scheduler yet, but I will post when I have it.
Thanks ! I think that writing code is the best way to evaluate an API.
bq. Overall it will be similar to the proposal in HADOOP-3445.
I still have to understand this bug ^.^ I'll wait for your code.
> Refactor the scheduler out of the JobTracker
> --------------------------------------------
>
> Key: HADOOP-3412
> URL: https://issues.apache.org/jira/browse/HADOOP-3412
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Brice Arnould
> Assignee: Brice Arnould
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: JobScheduler.patch, JobScheduler_v2.patch,
> JobScheduler_v3.patch, JobScheduler_v3b.patch, JobScheduler_v4.patch,
> JobScheduler_v5.patch, JobScheduler_v6.1.patch, JobScheduler_v6.2.patch,
> JobScheduler_v6.3.patch, JobScheduler_v6.4.patch, JobScheduler_v6.patch,
> JobScheduler_v7.1.patch, JobScheduler_v7.patch, RackAwareJobScheduler.java,
> SimpleResourceAwareJobScheduler.java
>
>
> First I would like warn you that my proposition is assumed to be very naive.
> I just hope that reading it won't make you lose time.
> h4. The aim
> It seems to me that improving Hadoop scheduling could be very profitable.
> But, it is hard to implement and compare schedulers, because the scheduling
> logic is mixed within the rest of the JobTracker.
> This bug is the first step of an attempt to improve the Hadoop scheduler. It
> re-implements the current scheduling algorithm in a separate class called
> JobScheduler. This new class is instantiated in the JobTracker.
> h4. Bug fixed as a side effects
> This patch probably cannot be submited as it is.
> A first difficulty is that it does not have exactly the same behaviour than
> the current JobTracker. More precisely, it doesn't re-implement things like
> code that seems to be never called or concurency problems.
> I wrote TOCONFIRM where my proposition differ from the current
> implementation, so you can find them easily.
> I know that fixing bugs silently is bad. So, independently of what you decide
> about this patch, I will open issues for bugs that you confirm.
> h4. Other side effects
> Another side effect of this patch is to add documentation about each step of
> the scheduling. I hope that it will help future improvement by lowering the
> level required to contribute to the scheduler.
> It also reduces the complexity and the granularity of the JobTracker (making
> it more parallel).
> h4. The future
> If you feel that this is a step the right direction, I will try to propose a
> JobSchedulerInterface that many JobSchedulers could implement and to propose
> alternatives to the current « FifoJobScheduler ». If some of you have ideas
> about that please tell ^^ I will also open issues for things marked as FIXME
> in the patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
