[
https://issues.apache.org/jira/browse/HADOOP-5199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vivek Ratan updated HADOOP-5199:
--------------------------------
Attachment: 5199.1.patch
I'm attaching a patch (5199.1.patch) that illustrates some of the proposals I
made in the previous comment. In this patch:
* {{HadoopTaskScheduler}} implements the Hadoop Scheduler. This class extends
{{TaskScheduler}}. Anyone who wants to write their own scheduler can continue
to extend {{TaskScheduler}}.
* I have copied taken the {{Pool}} and {{PoolManager}} classes from FS, and
copied them to src/mapred/o/a/h/mapred and renamed them to {{Pool2}} and
{{Pool2Manager}}. (If this proposal works, FS's {{Pool}} and {{Pool2Manager}}
classes will be deleted and {{Pool2}} and {{Pool2Manager}} will be renamed to
their original names. {{Pool2}} represents a pool - a collection - of jobs,
while {{Pool2Manager}} simply provides some methods to manage pools (and,
perhaps, can be removed).
* I needed to support the fact that we may have different ordering of jobs (as
described in the earlier comment). To do this, I have made the {{Pool2}} class
abstract. Developers are expected to extend this class to provide their own
ordering of jobs in a pool. I can think of two sub-classes of {{Pool2}}: one
which orders jobs based on priority and job submission time (which does what CS
and DS want), and one which orders jobs based on fairshare criteria. I have
implemented the former as the {{PriorityPool}} class, using the code from
{{JobQueueJobScheduler}}. The benefit of this approach is that anyone can
provide their own ordering of jobs and simply implement the ordering code.
Everything else is handled by the {{Pool2}} base class. When {{Pool2Manager}}
creates {{Pool2}} objects, it actually creates {{PriorityPool}} classes (or any
other sub-class of {{Pool2}}) as configured.
* When {{HadoopTaskScheduler}} starts up, it reads pool information from a pool
configuration file using the {{PoolConf}} class. This class expects
configuration to be in the same Hadoop format as other config files, rather
than in XML format as used by FS. Each pool is configured with a capacity
(which I refer to as 'guaranteed capacity' in the code). This should probably
be a percentage value and we should do the same sanity checks as we did in CS.
Pools will also be configured with user limits, which I haven't implemented
yet.
* I have not implemented user limits yet. We can presumably have user limits
based on tasks (each user can have no more than X% of the pool's capacity in
terms of running tasks, as CS does), or based on jobs (each user can have no
more than X% or Y jobs running at any given time) similar to what FS does.
* I have not implemented memory-based scheduling, but it has been implemented
in CS, and can be copied over with minimal fuss, I suspect.
I'd love to get feedback and thoughts from folks. This patch still needs work
and is meant to illustrate some of the proposals I had made earlier.
> A proposal to merge common functionality of various Schedulers
> --------------------------------------------------------------
>
> Key: HADOOP-5199
> URL: https://issues.apache.org/jira/browse/HADOOP-5199
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Vivek Ratan
> Attachments: 5199.1.patch
>
>
> There are at least 3 Schedulers in Hadoop today: Default, Capacity, and
> Fairshare. Over time, we're seeing a lot of functionality common to all
> three. Many bug fixes, improvements to existing functionality, and new
> functionality are applicable to all three schedulers. This trend seems to be
> getting stronger, as we notice similar problems, solutions, and ideas. This
> is a proposal to detect and consolidate such common functionality, while at
> the same time, allowing for differences in behavior and leaving the door open
> for other schedulers to be built, as per HADOOP-3412.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
