[
https://issues.apache.org/jira/browse/HADOOP-3675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12619492#action_12619492
]
Devaraj Das commented on HADOOP-3675:
-------------------------------------
Hemanth, yes, the memory _should_ be freed up but in the cases where a task is
using direct buffers and so on, which makes GC hard, one of the downstream
tasks would suffer. But that's not the most typical use case. I hate to make
the jvm reuse a per job user configurable option but if I do so, then users can
turn this feature off if they see their job behaving erratically. Would that
work?
Steve, actually, the task is run as a separate process. Its just that the same
JVM would be reused for running more tasks of the same job (so classloader
issue is a non-issue here). So the TT is not affected by this really.
> Provide more flexibility in the way tasks are run
> -------------------------------------------------
>
> Key: HADOOP-3675
> URL: https://issues.apache.org/jira/browse/HADOOP-3675
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Brice Arnould
> Assignee: Brice Arnould
> Priority: Minor
> Attachments: TaskWrapper_v0.patch, userBasedInsulator.sh
>
>
> *The aim*
> With [HADOOP-3421] speaking about sharing a cluster among more than one
> organization (so potentially with non-cooperative users), and posts on the ML
> speaking about virtualization and the ability to re-use the TaskTracker's VM
> to run new tasks, it could be useful for admins to choose the way TaskRunners
> run their children.
> More specifically, it could be useful to provide a way to imprison a Task in
> its working directory, or in a virtual machine.
> In some cases, reusing the VM might be useful, since it seems that this
> feature is really wanted ([HADOOP-249]).
> *Concretely*
> What I propose is a new class, called called SeperateVMTaskWrapper which
> contains the current logic for running tasks in another JVM. This class
> extends another, called TaskWrapper, which could be inherited to provide new
> ways of running tasks.
> As part of this issue I would also like to provide two other TaskWrappers :
> the first would run the tasks as Thread of the TaskRunner's VM (if it is
> possible without too much changes), the second would use a fixed pool of
> local unix accounts to insulate tasks from each others (so potentially
> non-cooperating users will be hable to share a cluster, as described in
> [HADOOP-3421]).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
