[
https://issues.apache.org/jira/browse/HADOOP-3581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12612873#action_12612873
]
Brice Arnould commented on HADOOP-3581:
---------------------------------------
Hi !
I think that there is a more general problem, that is task insulation, because
a bugged process could many other things than just overloading the memory.
The userBasedInsulator.sh that I proposed in [HADOOP-3675] could solve this
issue (and a few others) in an easier way. We don't need [HADOOP-3675] to be
complete to introduce that approch however.
The idea is to provide a "wrapper" charged to enforce local policies. This
wrapper can be written as a shell script to work on must Unix (and maybe
cygwin), and requires much less change to the core of Hadoop, that is :
{code}
+ ArrayList<String> vargs = new ArrayList<String>(8);
+ // Check for the eventual wrapper script
+ final String wrapper = conf.get("mapred.child.wrapper");
+ if (wrapper != null)
+ vargs.add(wrapper);
- Vector<String> vargs = new Vector<String>(8);
{code}
> Prevent memory intensive user tasks from taking down nodes
> ----------------------------------------------------------
>
> Key: HADOOP-3581
> URL: https://issues.apache.org/jira/browse/HADOOP-3581
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Reporter: Hemanth Yamijala
> Assignee: Vinod Kumar Vavilapalli
> Attachments: patch_3581_0.1.txt
>
>
> Sometimes user Map/Reduce applications can get extremely memory intensive,
> maybe due to some inadvertent bugs in the user code, or the amount of data
> processed. When this happens, the user tasks start to interfere with the
> proper execution of other processes on the node, including other Hadoop
> daemons like the DataNode and TaskTracker. Thus, the node would become
> unusable for any Hadoop tasks. There should be a way to prevent such tasks
> from bringing down the node.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
