[
https://issues.apache.org/jira/browse/HADOOP-4523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinod K V updated HADOOP-4523:
------------------------------
Attachment: HADOOP-4523-200811-06.txt
Messed up the approach. Here's another patch that gets it right. It
- monitors and kills any tasks that cross the individual tasks' limits they
have.
- kills the task with least progress(via tasktracker.findTaskToKill()) if even
after the first step the total memory usage across all tasks goes over the
total usage allowed.
- includes the tests
testTasksWithNoIndividualLimitsButTotalUsageWithinTTLimits,
testTasksWithinIndividualLimitsAndTotalUsageWithinTTLimits,
testTasksBeyondIndividualLimitsAndTotalUsageWithinTTLimits and
testTasksWithinIndividualLimitsButTotalUsageBeyondTTLimits.
> Enhance how memory-intensive user tasks are handled
> ---------------------------------------------------
>
> Key: HADOOP-4523
> URL: https://issues.apache.org/jira/browse/HADOOP-4523
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.19.0
> Reporter: Vivek Ratan
> Assignee: Vinod K V
> Attachments: HADOOP-4523-200811-05.txt, HADOOP-4523-200811-06.txt
>
>
> HADOOP-3581 monitors each Hadoop task to see if its memory usage (which
> includes usage of any tasks spawned by it and so on) is within a per-task
> limit. If the task's memory usage goes over its limit, the task is killed.
> This, by itself, is not enough to prevent badly behaving jobs from bringing
> down nodes. What is also needed is the ability to make sure that the sum
> total of VM usage of all Hadoop tasks does not exceed a certain limit.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
