[
https://issues.apache.org/jira/browse/HADOOP-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612390#action_12612390
]
Matei Zaharia commented on HADOOP-3412:
---------------------------------------
HADOOP-3445 has been listed as "blocked" by this JIRA since at least May, but I
guess they've started implementing it against the old codebase now? In any
case, I think it would be great for Hadoop going forward if there was a clean
way to plug in separate schedulers. As Tom and I commented, all you really need
the scheduler to support is three methods - addJob, removeJob and assignTask -
plus visibility into the data structures in the JobTracker. If it were possible
to refactor the changes in HADOOP-3445 to work behind this interface that would
be great.
I should add that I'm also working on a Hadoop job scheduler at Facebook. The
goal of my scheduler is to implement fair sharing between jobs, so that short
jobs are not starved by long jobs, while allowing for the same kinds of
per-group quotas that HADOOP-3445 provides. It will be similar to the
Completely Fair Scheduler in Linux. This seems to be the right thing for
Facebook's use case, which is a single cluster running a mix of long jobs,
shorter ad-hoc "interactive" queries, and jobs that need some kind of
guaranteed capacity to finish in time. I'll post a JIRA with a detailed design
in the next few days. I've implemented my scheduler so that it works with
Brice's patch, although I've been developing against a slightly different patch
that I wrote for Hadoop 0.17 because we want to be able to use the scheduler in
0.17.
> Refactor the scheduler out of the JobTracker
> --------------------------------------------
>
> Key: HADOOP-3412
> URL: https://issues.apache.org/jira/browse/HADOOP-3412
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Brice Arnould
> Assignee: Brice Arnould
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: JobScheduler.patch, JobScheduler_v2.patch,
> JobScheduler_v3.patch, JobScheduler_v3b.patch, JobScheduler_v4.patch,
> JobScheduler_v5.patch, JobScheduler_v6.1.patch, JobScheduler_v6.2.patch,
> JobScheduler_v6.3.patch, JobScheduler_v6.4.patch, JobScheduler_v6.patch,
> JobScheduler_v7.1.patch, JobScheduler_v7.patch, JobScheduler_v8.patch,
> RackAwareJobScheduler.java, SimpleResourceAwareJobScheduler.java
>
>
> First I would like warn you that my proposition is assumed to be very naive.
> I just hope that reading it won't make you lose time.
> h4. The aim
> It seems to me that improving Hadoop scheduling could be very profitable.
> But, it is hard to implement and compare schedulers, because the scheduling
> logic is mixed within the rest of the JobTracker.
> This bug is the first step of an attempt to improve the Hadoop scheduler. It
> re-implements the current scheduling algorithm in a separate class called
> JobScheduler. This new class is instantiated in the JobTracker.
> h4. Bug fixed as a side effects
> This patch probably cannot be submited as it is.
> A first difficulty is that it does not have exactly the same behaviour than
> the current JobTracker. More precisely, it doesn't re-implement things like
> code that seems to be never called or concurency problems.
> I wrote TOCONFIRM where my proposition differ from the current
> implementation, so you can find them easily.
> I know that fixing bugs silently is bad. So, independently of what you decide
> about this patch, I will open issues for bugs that you confirm.
> h4. Other side effects
> Another side effect of this patch is to add documentation about each step of
> the scheduling. I hope that it will help future improvement by lowering the
> level required to contribute to the scheduler.
> It also reduces the complexity and the granularity of the JobTracker (making
> it more parallel).
> h4. The future
> If you feel that this is a step the right direction, I will try to propose a
> JobSchedulerInterface that many JobSchedulers could implement and to propose
> alternatives to the current « FifoJobScheduler ». If some of you have ideas
> about that please tell ^^ I will also open issues for things marked as FIXME
> in the patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
