[
https://issues.apache.org/jira/browse/HADOOP-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12599640#action_12599640
]
Matei Zaharia commented on HADOOP-3412:
---------------------------------------
I'd like to mention one other thing that it might be useful to think about:
Letting the pluggable scheduler control speculative execution. Currently, the
scheduler asks each job for a map or reduce task using obtainNewMapTask or
obtainNewReduceTask. These may return a speculative task based on some fixed
thresholds. It would make more sense to have the JobInProgress expose only a
list of task statuses, and let the scheduler decide which ones to speculate. In
effect, move obtainNewMapTask and obtainNewReduceTask into the scheduler (have
them be utility methods in AbstractJobScheduler). This would consolidate all
scheduling code into one class, making it easier to understand and modify. It's
fairly clear that there's no "one size fits all" solution for speculation, and
users often disable it, change the thresholds, etc, so it makes sense to make
the entire speculation algorithm pluggable.
This is not a critical problem because a JobScheduler implementation can choose
to not call obtainNewMapTask / obtainNewReduceTask and just look at the
JobInProgress and select tasks by itself. It's just something that would make
the code more understandable by consolidating scheduling decisions in one place.
We might also consider making the speculative execution scheduler a separate
pluggable object, so that multiple JobSchedulers can reuse it. Thus we might
create a TaskScheduler interface (which would just contain obtainNewMapTask()
and obtainNewReduceTask()), and a default implementation that uses the current
threshold-based algorithm. AbstractJobScheduler could then look up which
TaskScheduler to use in the config file. However, the most important thing is
to make sure that the JobScheduler has a narrow interface (addJob, removeJob
and assignTask), because then people can write this type of thing behind it.
> Refactor the scheduler out of the JobTracker
> --------------------------------------------
>
> Key: HADOOP-3412
> URL: https://issues.apache.org/jira/browse/HADOOP-3412
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Brice Arnould
> Assignee: Brice Arnould
> Priority: Minor
> Attachments: JobScheduler.patch, JobScheduler_v2.patch,
> JobScheduler_v3.patch, JobScheduler_v3b.patch, JobScheduler_v4.patch,
> JobScheduler_v5.patch, RackAwareJobScheduler.java
>
>
> First I would like warn you that my proposition is assumed to be very naive.
> I just hope that reading it won't make you lose time.
> h4. The aim
> It seems to me that improving Hadoop scheduling could be very profitable.
> But, it is hard to implement and compare schedulers, because the scheduling
> logic is mixed within the rest of the JobTracker.
> This bug is the first step of an attempt to improve the Hadoop scheduler. It
> re-implements the current scheduling algorithm in a separate class called
> JobScheduler. This new class is instantiated in the JobTracker.
> h4. Bug fixed as a side effects
> This patch probably cannot be submited as it is.
> A first difficulty is that it does not have exactly the same behaviour than
> the current JobTracker. More precisely, it doesn't re-implement things like
> code that seems to be never called or concurency problems.
> I wrote TOCONFIRM where my proposition differ from the current
> implementation, so you can find them easily.
> I know that fixing bugs silently is bad. So, independently of what you decide
> about this patch, I will open issues for bugs that you confirm.
> h4. Other side effects
> Another side effect of this patch is to add documentation about each step of
> the scheduling. I hope that it will help future improvement by lowering the
> level required to contribute to the scheduler.
> It also reduces the complexity and the granularity of the JobTracker (making
> it more parallel).
> h4. The future
> If you feel that this is a step the right direction, I will try to propose a
> JobSchedulerInterface that many JobSchedulers could implement and to propose
> alternatives to the current « FifoJobScheduler ». If some of you have ideas
> about that please tell ^^ I will also open issues for things marked as FIXME
> in the patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
