[
https://issues.apache.org/jira/browse/HADOOP-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12620129#action_12620129
]
Vivek Ratan commented on HADOOP-3759:
-------------------------------------
I'm in favor of the approach Hemanth has described above because it lets us
reason about memory constraints in terms of slots/containers. In his example,
each slot on the TT has a memory limit of 4GB. This also means that a task is
guaranteed up to 4GB, if it runs in that slot. So if a task needs 6GB, it ends
up using 2 slots. This helps schedulers, because they're really dealing with
slots. For example, if they're keeping track of how many slots are used by a
user or job, then assigning two slots to the task that wants 6GB works really
well.
Another good thing about this approach is that tasks are not affected by other
tasks, in terms of memory guarantees. Suppose task1 wanted 7.5 GB. This would
leave 0.5GB for task2, which is not right, as task2 shouldn't suffer from
task1's needs. It's better task2 not run in this case.
> Provide ability to run memory intensive jobs without affecting other running
> tasks on the nodes
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-3759
> URL: https://issues.apache.org/jira/browse/HADOOP-3759
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-3759.patch
>
>
> In HADOOP-3581, we are discussing how to prevent memory intensive tasks from
> affecting Hadoop daemons and other tasks running on a node. A related
> requirement is that users be provided an ability to run jobs which are memory
> intensive. The system must provide enough knobs to allow such jobs to be run
> while still maintaining the requirements of HADOOP-3581.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
