[
https://issues.apache.org/jira/browse/HADOOP-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brice Arnould updated HADOOP-3412:
----------------------------------
Attachment: JobScheduler_v6.2.patch
*Tom White :*
{quote}1. You can change AbstractJobScheduler and FifoJobScheduler's
constructors to be no-arg constructors. The JobTracker isn't being used, and if
you make AbstractJobScheduler implement Configurable, the the Configuration
will be automatically called if you use ReflectionUtils#newInstance (this is
the standard Hadoop pattern).{quote}
I started using that pattern (up to v4), but I told myself that it could be
nice to give to the scheduler an access to the topology. It's true however that
the FifoJobScheduler do not use it, so I can remove it if you want.
{quote}2. This needs some unit tests. It should be possible to write a unit
test for FifoJobScheduler since it is relatively isolated.{quote}
A really good idea ! I found a bug writing the tests :-P (that bug occured when
two job of the same priority were added at the same time).
This test tries to circumvent the need of starting a whole cluster by adding a
protected constructor to JobInProgress, used by a class called
FakeJobInProgress. I really don't like the idea of adding a constructor to ease
the write of tests, but all other ideas I had seemed uglier. Suggestions are
welcome ^^
{quote}3. In AbstractJobScheduler#assignTask you move to the next step if the
following condition is true:
{code}(step == 0 || step == 2) && mapTasksNumber > maximumMapLoad
|| (step == 1 || step == 3) && reduceTasksNumber >
maximumReduceLoad{code}
Should these checks be >= (greater than or equal to) so the maximum load isn't
exceeded?{quote}
Ooops
Fixed ^^
{quote}4. Rather than having AbstractJobScheduler manage maxTasksPerJob, it
might be better to have a subclass of FifoJobScheduler that limits tasks per
job (TaskLimitedFifoJobScheduler). Or if this is possible through composition
rather than inheritance then even better. Then FifoJobScheduler would then just
preserve the current behaviour.{quote}My argument in favor of keeping that in
the AbstractJobScheduler is that, when I implemented other schedulers to test
the API (just before v6), this part enforcing limitations was common to all of
them. The reason is that those options keeps their meaning with all schedulers.
However if the point is that the current behaviour should still be the default,
I completely agree.
For sure I have nothing against making the modifications you propose in 1) and
4), but I wanted to be sure you had considered the reasons that made me choose
the current organisation.
> Refactor the scheduler out of the JobTracker
> --------------------------------------------
>
> Key: HADOOP-3412
> URL: https://issues.apache.org/jira/browse/HADOOP-3412
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Brice Arnould
> Assignee: Brice Arnould
> Priority: Minor
> Fix For: 0.19.0
>
> Attachments: JobScheduler.patch, JobScheduler_v2.patch,
> JobScheduler_v3.patch, JobScheduler_v3b.patch, JobScheduler_v4.patch,
> JobScheduler_v5.patch, JobScheduler_v6.1.patch, JobScheduler_v6.2.patch,
> JobScheduler_v6.patch, RackAwareJobScheduler.java
>
>
> First I would like warn you that my proposition is assumed to be very naive.
> I just hope that reading it won't make you lose time.
> h4. The aim
> It seems to me that improving Hadoop scheduling could be very profitable.
> But, it is hard to implement and compare schedulers, because the scheduling
> logic is mixed within the rest of the JobTracker.
> This bug is the first step of an attempt to improve the Hadoop scheduler. It
> re-implements the current scheduling algorithm in a separate class called
> JobScheduler. This new class is instantiated in the JobTracker.
> h4. Bug fixed as a side effects
> This patch probably cannot be submited as it is.
> A first difficulty is that it does not have exactly the same behaviour than
> the current JobTracker. More precisely, it doesn't re-implement things like
> code that seems to be never called or concurency problems.
> I wrote TOCONFIRM where my proposition differ from the current
> implementation, so you can find them easily.
> I know that fixing bugs silently is bad. So, independently of what you decide
> about this patch, I will open issues for bugs that you confirm.
> h4. Other side effects
> Another side effect of this patch is to add documentation about each step of
> the scheduling. I hope that it will help future improvement by lowering the
> level required to contribute to the scheduler.
> It also reduces the complexity and the granularity of the JobTracker (making
> it more parallel).
> h4. The future
> If you feel that this is a step the right direction, I will try to propose a
> JobSchedulerInterface that many JobSchedulers could implement and to propose
> alternatives to the current « FifoJobScheduler ». If some of you have ideas
> about that please tell ^^ I will also open issues for things marked as FIXME
> in the patch.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
