[
https://issues.apache.org/jira/browse/HADOOP-3675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12619448#action_12619448
]
Devaraj Das commented on HADOOP-3675:
-------------------------------------
I am looking at the JVM reuse issue particularly. I think that can be done
independently of this jira. My idea is to still have the umbilical protocol for
task executions. The difference with the current approach is that that JVM
would not exit at the end of the current task execution but the
TaskTracker.Child.main would have a loop inside. The crux of the loop would
look something like:
{code}
while (true) {
Task t = umbilical.getTask();
if (t != null) {
t.run(umbilical);
}
}
{code}
If JVM reuse is enabled, the tasktracker would have 1 JVM per job. The
tasktracker would kill the JVM process when the corresponding job finishes. If
it is already running too many JVMs, it would kill one based on LRU.
Also, the JVM would kill itself whenever it encounters an exception (a running
task on encountering an exception would result in the shutdown of the JVM).
That would take care of problems such as a long running JVM running out of
memory. The only thing is a task attempt might get penalized for no fault of
the task (in the case where the JVM is leaking memory, for example) since that
attempt is bound to fail. So this is something to watch out for but I am not
too worried about it at this point of time.
Overall, the above would probably benefit jobs with many short running tasks
(like for maps which are short lived).
I am thinking of submitting the patch with the above for HADOOP-249.
Thoughts?
> Provide more flexibility in the way tasks are run
> -------------------------------------------------
>
> Key: HADOOP-3675
> URL: https://issues.apache.org/jira/browse/HADOOP-3675
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Brice Arnould
> Assignee: Brice Arnould
> Priority: Minor
> Attachments: TaskWrapper_v0.patch, userBasedInsulator.sh
>
>
> *The aim*
> With [HADOOP-3421] speaking about sharing a cluster among more than one
> organization (so potentially with non-cooperative users), and posts on the ML
> speaking about virtualization and the ability to re-use the TaskTracker's VM
> to run new tasks, it could be useful for admins to choose the way TaskRunners
> run their children.
> More specifically, it could be useful to provide a way to imprison a Task in
> its working directory, or in a virtual machine.
> In some cases, reusing the VM might be useful, since it seems that this
> feature is really wanted ([HADOOP-249]).
> *Concretely*
> What I propose is a new class, called called SeperateVMTaskWrapper which
> contains the current logic for running tasks in another JVM. This class
> extends another, called TaskWrapper, which could be inherited to provide new
> ways of running tasks.
> As part of this issue I would also like to provide two other TaskWrappers :
> the first would run the tasks as Thread of the TaskRunner's VM (if it is
> possible without too much changes), the second would use a fixed pool of
> local unix accounts to insulate tasks from each others (so potentially
> non-cooperating users will be hable to share a cluster, as described in
> [HADOOP-3421]).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
