BoYang commented on YARN-2518:

In my rough testing, it did not fail the node manager process. In my Container 
Executor implementation (launchContainer method), I register a new application 
master, send a message to another long running process, and unregister the 
application master. I can see the application finished successfully.

Of course, that was my very draft initial testing. We could fine-tune the code 
to make it work better. But technically it seems doable now. Thus I am curious 
whether the YARN community could take this feature and provide official support.

> Support in-process container executor
> -------------------------------------
>                 Key: YARN-2518
>                 URL: https://issues.apache.org/jira/browse/YARN-2518
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: nodemanager
>    Affects Versions: 2.5.0
>         Environment: Linux, Windows
>            Reporter: BoYang
>            Priority: Minor
>              Labels: container, dispatch, in-process, job, node
> Node Manage always creates a new process for a new application. We have hit a 
> scenario where we want the node manager to execute the application inside its 
> own process, so we get fast response time. It would be nice if Node Manager 
> or YARN can provide native support for that.
> In general, the scenario is that we have a long running process which can 
> accept requests and process the requests inside its own process. Since YARN 
> is good at scheduling jobs, we want to use YARN to dispatch jobs (e.g. 
> requests in JSON) to the long running process. In that case, we do not want 
> YARN container to spin up a new process for each request. Instead, we want 
> YARN container to send the request to the long running process for further 
> processing.

This message was sent by Atlassian JIRA

Reply via email to