[
https://issues.apache.org/jira/browse/HADOOP-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557767#action_12557767
]
Doug Cutting commented on HADOOP-2573:
--------------------------------------
I think a static limit for all jobs would be useful and best to implement
first. After some experience with this, we would be better able to address its
shortcomings. Possible future extensions might be:
- dynamically altering the limit, e.g., limit=max(min.tasks.per.job,
numSlots/numJobsOutstanding)
-- ramping up the limit slowly, so that a users's sequential jobs don't have
all their slots immediately taken when one job completes
-- ramping down the limit slowly, so that tasks are given an opportunity to
finish normally before they are killed.
- incorporating job priority into the limit
> limit running tasks per job
> ---------------------------
>
> Key: HADOOP-2573
> URL: https://issues.apache.org/jira/browse/HADOOP-2573
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Doug Cutting
> Fix For: 0.17.0
>
>
> It should be possible to specify a limit to the number of tasks per job
> permitted to run simultaneously. If, for example, you have a cluster of 50
> nodes, with 100 map task slots and 100 reduce task slots, and the configured
> limit is 25 simultaneous tasks/job, then four or more jobs will be able to
> run at a time. This will permit short jobs to pass longer-running jobs.
> This also avoids some problems we've seen with HOD, where nodes are
> underutilized in their tail, and it should permit improved input locality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
