[
https://issues.apache.org/jira/browse/HADOOP-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612901#action_12612901
]
Vinod Kumar Vavilapalli commented on HADOOP-3581:
-------------------------------------------------
Sorry about not stating this before my last comment, but in this JIRA I am
restricting myself to writing utilities that track process trees, watch their
memory usage, destroy them when overflowing etc. We definitely need the
functionality of these classes, and we can use them where ever we wish to, once
HADOOP-3675 moves. Will create a new JIRA for this, if that is the correct step
forward.
Regarding where we want to perform the resource management - tracking disk
space, memory usage etc., we can either implement it in the wrapper or in the
TaskTracker itself.
- Implementation in wrapper implies tracking per task and thus we will not
have a global picture of resource usage at TaskTracker level. Further, it is
set-once-and-run kind of mechanism - before launching tasks itself, we will
have to declare the limits within which tasks can run. If we wish to make these
limits dynamic, we will need an extra communication pipe between the wrapper
and TaskTracker.
- Implementing resource tracking in TaskTracker helps us avoid all this.
Moreover, it gives the global idea of usage over all tasks and can thus be
flexible.
Brice, your idea of doing it in the wrapper script relates to the first point
above and thus has some disadvantages. It would be good, if we could just
restrict the wrapper to do the most basic need of isoated
work-space/chroot/jail-like mechanism and do task's resource management in TT.
Comments?
> Prevent memory intensive user tasks from taking down nodes
> ----------------------------------------------------------
>
> Key: HADOOP-3581
> URL: https://issues.apache.org/jira/browse/HADOOP-3581
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Attachments: patch_3581_0.1.txt
>
>
> Sometimes user Map/Reduce applications can get extremely memory intensive,
> maybe due to some inadvertent bugs in the user code, or the amount of data
> processed. When this happens, the user tasks start to interfere with the
> proper execution of other processes on the node, including other Hadoop
> daemons like the DataNode and TaskTracker. Thus, the node would become
> unusable for any Hadoop tasks. There should be a way to prevent such tasks
> from bringing down the node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
