[
https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854528#comment-15854528
]
Haibo Chen commented on YARN-1593:
----------------------------------
[~vvasudev] Want to check to see if you have some time to address some of the
comments above?
> support out-of-proc AuxiliaryServices
> -------------------------------------
>
> Key: YARN-1593
> URL: https://issues.apache.org/jira/browse/YARN-1593
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, rolling upgrade
> Reporter: Ming Ma
> Assignee: Varun Vasudev
> Attachments: SystemContainersandSystemServices.pdf
>
>
> AuxiliaryServices such as ShuffleHandler currently run in the same process as
> NM. There are some benefits to host them in dedicated processes.
> 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the
> ShuffleHandler restart. If ShuffleHandler runs as a separate process,
> ShuffleHandler can continue to run during NM restart. NM can reconnect the
> the running ShuffleHandler after restart.
> 2. Resource management. It is possible another type of AuxiliaryServices will
> be implemented. AuxiliaryServices are considered YARN application specific
> and could consume lots of resources. Running AuxiliaryServices in separate
> processes allow easier resource management. NM could potentially stop a
> specific AuxiliaryServices process from running if it consumes resource way
> above its allocation.
> Here are some high level ideas:
> 1. NM provides a hosting process for each AuxiliaryService. Existing
> AuxiliaryService API doesn't change.
> 2. The hosting process provides RPC server for AuxiliaryService proxy object
> inside NM to connect to.
> 3. When we rolling restart NM, the existing AuxiliaryService processes will
> continue to run. NM could reconnect to the running AuxiliaryService processes
> upon restart.
> 4. Policy and resource management of AuxiliaryServices. So far we don't have
> immediate need for this. AuxiliaryService could run inside a container and
> its resource utilization could be taken into account by RM and RM could
> consider a specific type of applications overutilize cluster resource.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
