[
https://issues.apache.org/jira/browse/HADOOP-3675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12619489#action_12619489
]
Steve Loughran commented on HADOOP-3675:
----------------------------------------
In the ant context In-VM execution of things like Javac and junit was always
good for speed, but bad for long term memory consumption, both of the normal
heap and of PermGenHeapSpace
* you can't unload a class until all references are removed
* you can't unload a classloader until all classes in it are removed
* its very hard to be sure that classes/classloaders are fully unloaded.
You also need to be sure that you dont pass down any of your own classes to the
code, just those in the JDK, which forces you to add extra code and tests,
especially for the com.sun stuff
Here is some of the code, which is intended to give a hint as to how hard it
is, rather than provide a starting point:
http://svn.apache.org/viewvc/ant/core/trunk/src/main/org/apache/tools/ant/util/JavaEnvUtils.java?view=markup
http://svn.apache.org/viewvc/ant/core/trunk/src/main/org/apache/tools/ant/AntClassLoader.java?view=markup
Unless you are planning on killing the tasktracker process on a regular basis
and restarting it, a more reliable alternative might be for the task tracker to
start and communicate with a secondary process that the tasktracker can keep
alive, but which lets you start even more work. This is effectively what
smartfrog does, using RMI to communicate between processes. Again, classloaders
are painful to work with, but you can kill the children more easily. It has
proven to work better over long-lived deployments -but took a lot more
engineering effort.
> Provide more flexibility in the way tasks are run
> -------------------------------------------------
>
> Key: HADOOP-3675
> URL: https://issues.apache.org/jira/browse/HADOOP-3675
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Brice Arnould
> Assignee: Brice Arnould
> Priority: Minor
> Attachments: TaskWrapper_v0.patch, userBasedInsulator.sh
>
>
> *The aim*
> With [HADOOP-3421] speaking about sharing a cluster among more than one
> organization (so potentially with non-cooperative users), and posts on the ML
> speaking about virtualization and the ability to re-use the TaskTracker's VM
> to run new tasks, it could be useful for admins to choose the way TaskRunners
> run their children.
> More specifically, it could be useful to provide a way to imprison a Task in
> its working directory, or in a virtual machine.
> In some cases, reusing the VM might be useful, since it seems that this
> feature is really wanted ([HADOOP-249]).
> *Concretely*
> What I propose is a new class, called called SeperateVMTaskWrapper which
> contains the current logic for running tasks in another JVM. This class
> extends another, called TaskWrapper, which could be inherited to provide new
> ways of running tasks.
> As part of this issue I would also like to provide two other TaskWrappers :
> the first would run the tasks as Thread of the TaskRunner's VM (if it is
> possible without too much changes), the second would use a fixed pool of
> local unix accounts to insulate tasks from each others (so potentially
> non-cooperating users will be hable to share a cluster, as described in
> [HADOOP-3421]).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
