[
https://issues.apache.org/jira/browse/HADOOP-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12613596#action_12613596
]
Hemanth Yamijala commented on HADOOP-3759:
------------------------------------------
Initial proposal:
- HADOOP-3581 proposes a maximum amount of virtual memory, say MAX_MEM, that
all tasks (and their descendants) on that tasktracker would use.
- By default, we can translate this to a per task memory limit which is =
MAX_MEM / number of slots
- To allow high memory jobs to run, we define a new configuration variable that
users can set to specify the maximum memory they expect their tasks to take.
- In each heartbeat, the tasktracker computes the amount of free memory (using
HADOOP-3581's fix) and reports that to the jobtracker. This is similar to the
approach followed in HADOOP-657 for disk space.
- The jobtracker schedules a new task on this tasktracker, only if it's job's
memory limit per task is less than the free memory.
- If a job's task takes more memory than what is specified (either default or
explicitly configured), per HADOOP-3581, it will be killed.
Comments ?
> Provide ability to run memory intensive jobs without affecting other running
> tasks on the nodes
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-3759
> URL: https://issues.apache.org/jira/browse/HADOOP-3759
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
>
> In HADOOP-3581, we are discussing how to prevent memory intensive tasks from
> affecting Hadoop daemons and other tasks running on a node. A related
> requirement is that users be provided an ability to run jobs which are memory
> intensive. The system must provide enough knobs to allow such jobs to be run
> while still maintaining the requirements of HADOOP-3581.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
