[
https://issues.apache.org/jira/browse/MAPREDUCE-1221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12779840#action_12779840
]
dhruba borthakur commented on MAPREDUCE-1221:
---------------------------------------------
> In any case, it currently works only on Linux
It is, of course, for Linux :-)
> The TaskMemoryManager indeed looks at the memory usage of all the processes
> in the task's process-tree irr
That's is right. But my point was that each python/php process that is forked
by the mapper actually uses about 500GB of virtual memory (at the minimum). And
for each python library that the process includes, it adds yet another chunk of
virtual memory. Most of this virtual memory is unused the python/php
interpreter; i.e. no physical pages allocated for the process. But since the
existing MR framework looks at the total virtual memory of the process subtree,
it starts killing tasks even though a large chunk of the physical RAM on on
the machine is unused.
The proposal to have the ability to kill tasks based on physical memory usage
should make system utilization much better, do you agree?
> Kill tasks on a node if the free physical memory on that machine falls below
> a configured threshold
> ---------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-1221
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1221
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: tasktracker
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
>
> The TaskTracker currently supports killing tasks if the virtual memory of a
> task exceeds a set of configured thresholds. I would like to extend this
> feature to enable killing tasks if the physical memory used by that task
> exceeds a certain threshold.
> On a certain operating system (guess?), if user space processes start using
> lots of memory, the machine hangs and dies quickly. This means that we would
> like to prevent map-reduce jobs from triggering this condition. From my
> understanding, the killing-based-on-virtual-memory-limits (HADOOP-5883) were
> designed to address this problem. This works well when most map-reduce jobs
> are Java jobs and have well-defined -Xmx parameters that specify the max
> virtual memory for each task. On the other hand, if each task forks off
> mappers/reducers written in other languages (python/php, etc), the total
> virtual memory usage of the process-subtree varies greatly. In these cases,
> it is better to use kill-tasks-using-physical-memory-limits.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
