[
https://issues.apache.org/jira/browse/HADOOP-3759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12619474#action_12619474
]
Hemanth Yamijala commented on HADOOP-3759:
------------------------------------------
After some internal discussions, we are proposing the following behavior with
respect to default values of the configuration options MAX_MEM and
MAX_MEM_PER_TASK. In the description below, MAX_MEM is specified for a
tasktracker, and MAX_MEM_PER_TASK is specified in a job's configuration.
- We should provide an option to turn off the behavior described in the above
comment. We can do this by defaulting the configuration option values to
something invalid, like -1.
- This gives rise to the following 4 conditions, and the proposed behavior of
the system for each:
-- Both MAX_MEM and MAX_MEM_PER_TASK are set to invalid values: This will
result in the current Map/Reduce system behavior. Essentially, it will turn off
the fixes in this JIRA and in HADOOP-3581, thereby providing backwards
compatibility.
-- Both MAX_MEM and MAX_MEM_PER_TASK are specified as valid values: This will
result in the behavior described on this JIRA and also in HADOOP-3581. In
short, it will result in the tasks of the job being scheduled on TaskTrackers
with sufficient free memory, and tasks which exceed the memory limits will be
killed.
-- MAX_MEM is set to invalid value but MAX_MEM_PER_TASK is specified for a job:
MAX_MEM_PER_TASK is ignored. Since MAX_MEM is an administrative option, if an
administrator has disabled it , no memory specific scheduling can be turned on
by users.
-- MAX_MEM is specified, but MAX_MEM_PER_TASK is set to an invalid value: A
task of this job will run on this TT only if the free memory is at least equal
to MAX_MEM / number of slots on the TT. For e.g. consider a TT with 8 GB as
MAX_MEM and 2 slots. Consider two jobs: one with a 6 GB MAX_MEM_PER_TASK and
the other which has no valid value specified. If a task of the first job is
running on this TT, there is only 2 GB that can be guaranteed for any other
job. Since MAX_MEM / number of slots is 4 GB, no task of the second job would
run on this TT until the high memory task completes. In other words, we are
defining that if a job has not specified any MAX_MEM_PER_TASK, it is guaranteed
at least a MAX_MEM / number of slots amount of memory by the system. This seems
to be fairly deterministic compared to scheduling tasks of the second job and
binding their MAX_MEM_PER_TASK to some arbitrary amount of free memory.
As the above choice means that the RAM intensive jobs can prevent other jobs
from running, there must be some way to counter-balance this effect. One way
could be for schedulers like HADOOP-3445 to artifically consider the RAM
intensive job as using more slots than actual, and thereby subjecting them to
limits faster.
Comments ?
> Provide ability to run memory intensive jobs without affecting other running
> tasks on the nodes
> -----------------------------------------------------------------------------------------------
>
> Key: HADOOP-3759
> URL: https://issues.apache.org/jira/browse/HADOOP-3759
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Hemanth Yamijala
> Fix For: 0.19.0
>
> Attachments: HADOOP-3759.patch
>
>
> In HADOOP-3581, we are discussing how to prevent memory intensive tasks from
> affecting Hadoop daemons and other tasks running on a node. A related
> requirement is that users be provided an ability to run jobs which are memory
> intensive. The system must provide enough knobs to allow such jobs to be run
> while still maintaining the requirements of HADOOP-3581.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
