[
https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14340276#comment-14340276
]
Junping Du commented on MAPREDUCE-5583:
---------------------------------------
Thanks [~jlowe] for explanation and [~kasha] for confirmation for FS.
I agree that we should fix this in FIFOScheduler given the expected behavior
for Schedulers that we discussed above, and I will file a JIRA later for this
issue.
Patch looks good to me now except one NIT below:
{code}
+ ResourceRequest oldReqLimit = requestLimits.put(newReqLimit, newReqLimit);
+ if (oldReqLimit == null || oldReqLimit.getNumContainers() < limit) {
+ requestLimitsToUpdate.add(newReqLimit);
+ }
{code}
Looks like requestLimits will always have the same key and value here. So the
return value for requestLimits.put(newReqLimit, newReqLimit) can always be null
or the same as newReqLimit (event in numOfContainers), so checking for
"oldReqLimit.getNumContainers() < limit" sounds unnecessary to me.
> Ability to limit running map and reduce tasks
> ---------------------------------------------
>
> Key: MAPREDUCE-5583
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am, mrv2
> Affects Versions: 0.23.9, 2.1.1-beta
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: MAPREDUCE-5583-branch2.4.1.patch,
> MAPREDUCE-5583v1.patch, MAPREDUCE-5583v2.patch, MAPREDUCE-5583v3.patch,
> MAPREDUCE-5583v4.patch
>
>
> It would be nice if users could specify a limit to the number of map or
> reduce tasks that are running simultaneously. Occasionally users are
> performing operations in tasks that can lead to DDoS scenarios if too many
> tasks run simultaneously (e.g.: accessing a database, web service, etc.).
> Having the ability to throttle the number of tasks simultaneously running
> would provide users a way to mitigate issues with too many tasks on a large
> cluster attempting to access a serivce at any one time.
> This is similar to the functionality requested by MAPREDUCE-224 and
> implemented by HADOOP-3412 but was dropped in mrv2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
