[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16010805#comment-16010805 ] Haibo Chen commented on YARN-1593: -- It seems system services are now addressed by YARN-6601 separately. Do you have plans in the short term to update the design doc and maybe work on system container [~vvasudev]? > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Varun Vasudev > Attachments: SystemContainersandSystemServices.pdf > > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15980954#comment-15980954 ] Rohith Sharma K S commented on YARN-1593: - Thanks [~vvasudev] for detailed doc and thanks to other folks for design discussions. The doc talks about both system container and system services. IIUC, these 2 use cases and scopes are different. System_services are admin configuration and managed via native service layer along with AppMaster. And system_containers are special containers requested by individual NodeManager to replace auxiliary services. These are independent containers which do not need app-master. Could we separate system_containers and system_services into separate JIRA? P.S : For flow-collector launching we are looking for system services so that any arbitrary client should be able to publish data into ATSv2. > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Varun Vasudev > Attachments: SystemContainersandSystemServices.pdf > > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854528#comment-15854528 ] Haibo Chen commented on YARN-1593: -- [~vvasudev] Want to check to see if you have some time to address some of the comments above? > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Varun Vasudev > Attachments: SystemContainersandSystemServices.pdf > > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15710178#comment-15710178 ] Haibo Chen commented on YARN-1593: -- Thanks for starting the work on this, Varun Vasudev! I’d like to understand the proposal better. A few comments/questions on the proposal. Please correct me as necessary. It seems like system containers are overloaded in the design doc. From a NM’s perspective, my understanding is that system containers are special container runtime (relative to the container types we have today in NM) provided by NM to be used by system services to run their components/instances. In other cases, system containers represent components/instances of system services on the worker nodes. In the former case, we may only need to be concerned with issues such as classpath and container executors. For ShuffleHandler for instance, it is an alternative of the in-process runtime it gets from NM today. The latter, is where we discuss whether RM or NM does the heavy-lifting of managing system containers. As you mention, no one option suits all use cases. Option 1 suits some, while option 3 suits others. I wonder if this is because we are conflating two different types of containers in the proposal - (1) framework-specific services like MR shuffle, and (2) application-specific services. Framework services are to be run on all nodes that support the framework (e.g. MR). Since these run on every node, node-level configs (option 3) would work best. Application-services (e.g. ATS AM-companion-collector), on the other hand, are application specific and need to run on a subset of cluster nodes; option 1 readily applies to these. Is this categorization accurate? And, do you see merit in differentiating between these two? bq. Allow shuffle to run on the NodeManagers without requiring it to be setup as an AuxiliaryService Not sure if I understand this correctly, IHO, we could let the user continue with their current configuration for AuxiliaryService, but just run them in containers with AuxiliaryService proxy like Junping said in the jira description. bq. Handling container status for system-containers - we will need to add logic to not act upon the container status of a system-container. Can you please elaborate more on this? Shouldn’t NM try to relaunch system containers? Does this mean that RM will take the responsibility of handling system container failures? bq. I think discovery is going to be one major piece that needs to be addressed from the beginning Agree with Sangjin that discovery problem needs to be addressed right at the beginning. For option 3, I think we can add a queryable registry in AuxiliaryServices when NM launches a proxied AuxiliaryService assuming that NM will launch the AuxiliaryServices in the right order and each AuxiliaryService knows its dependent services. bq. the NodeManager will block container requests until all the system-containers are running With global scheduling and resource affinity, NM does not necessarily need to block container launching. NM can launch system containers asynchronously and report to resource manager upon launch success, and RM can only schedule containers on those nodes if the services that the containers depend on have been launched on those nodes. But that’s way in the future I guess bq. We can’t solve the dependency management and affinity/anti-affinity requirements. (One of cons in option 3) Not quite sure how option 1 solves the affinity requirement. Can you elaborate a little more on this? To solve the dependency management issue, one thing that occurred to me, but I have not thought about in much details, is, we could have RM manages all system services together and construct a DAG of system services that need to be launched on each NM. Alternatively, RM can just decide what services need to be launched on which nodes with their dependency clearly defined, and then NM can construct the DAG themselves and launches them in topological order. This however, does put some burden on RM. > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Varun Vasudev > Attachments: SystemContainersandSystemServices.pdf > > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668916#comment-15668916 ] Konstantinos Karanasos commented on YARN-1593: -- Thanks for starting this! As [~asuresh] and [~hrsharma] pointed out, this is very related to the container pooling we have been thinking of, so it's great to see there is more work to this direction. Here are some first thoughts: - There seems to be a common need to have containers not belonging to an AM. I like your analysis about the pros and cons of the three approaches. Ideally, and if possible, it would be good to agree on an approach that is not hybrid, i.e., to not have some containers going through option (1) and some others through option (3), but rather have a unified approach. In container pooling we have thought of having a component in the RM that manages how many "system" containers will running at each node, but we are willing to adopt another approach if it is more suitable. - Looking both at your document and the comments above, it seems that no approach can properly tackle the dependencies problem. Probably we should solve this in the scheduler: just like there will be support for (anti-)affinity constraints, we can add support for dependencies in the scheduler, e.g., to not schedule that container to a node before a shuffle container is running on that node. - Although I like your proposal of using a new ExecutionType for the system containers, I am not sure it is always desirable to couple system containers with the highest priority ExecutionType. For instance, there can be system containers that are not as important and can be preempted to make space if needed. Also, apart from the execution priority, I am not sure if the ExecutionType should determine whether a container should be automatically relaunched. If we end up having a component managing those containers, maybe it is its role to determine if they get restarted upon failure (irrespective of their ExecutionType). > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Varun Vasudev > Attachments: SystemContainersandSystemServices.pdf > > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668649#comment-15668649 ] Hitesh Sharma commented on YARN-1593: - Thanks [~asuresh] for pointing to [YARN-5501]. Agree with you folks that there is some overlap and we will be happy to converge and discuss the best way to leverage the efforts here. [~vvasudev], with regards to pooled container the behavior is to allow NM to serve container requests even if the pre-initialized container is not ready. For container pooling this behavior makes sense as we eventually want to advertise pre-initialized container as a resource and have the AM ask for it. Regarding the 2nd point, current implementation starts a fixed number of pre-initialized container on each node (what to start, resources to localize, and other details are currently passed via config files). Eventually we intend the RM to pick up some nodes where the pre-initialized container should be started. This is something we are starting to work upon. > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Varun Vasudev > Attachments: SystemContainersandSystemServices.pdf > > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[
https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15668499#comment-15668499
]
Sangjin Lee commented on YARN-1593:
---
Thanks for starting the proposal!
I took a quick look at it, and here are some of my initial thoughts (maybe more
later).
One use case that is not mentioned is the timeline service v.2 collector
(writer). We can think of it in two possible approaches: (1) another system
container/service that needs to be launched on every node before NM can serve
containers, or (2) system container that can be started on demand when an app
is started (one container per app). I think (1) fits nicely with the system
container you're envisioning. (2) is much more dynamic than any of the
approaches discussed in the doc. FYI.
I think discovery is going to be one major piece that needs to be addressed
from the beginning. Even in the most basic use cases (e.g. MR shuffle handler,
timeline reader, etc.), the discoverability of the containers and their
endpoints is hugely important. It would be great if it is addressed in the
first design.
I also agree that localization is going to be a problem, and I think it's going
to be an issue no matter which option you take. If the system container needs
to run as long as the node is up, it's hard to avoid the issue of localization
unless you pre-deliver the bits as part of setting up the nodes.
In terms of the approaches, I lean slightly towards (3). It feels awkward to
treat it as "just another app" as they have different semantics from any other
app. If we're elevating the notion of the system containers to first class, we
might as well be explicit while still trying to reuse a lot of the pieces for
implementation. That's my 2 cents.
One question: what do we do with the resource utilization of these system
containers? Should they be reported just like any container (I'm thinking of
{{ContainersMonitorImpl}}, {{NMTimelinePublisher}} and so on)? Or should they
be considered outside the monitoring scope, like a YARN daemon today? Have you
thought about that?
> support out-of-proc AuxiliaryServices
> -
>
> Key: YARN-1593
> URL: https://issues.apache.org/jira/browse/YARN-1593
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, rolling upgrade
>Reporter: Ming Ma
>Assignee: Varun Vasudev
> Attachments: SystemContainersandSystemServices.pdf
>
>
> AuxiliaryServices such as ShuffleHandler currently run in the same process as
> NM. There are some benefits to host them in dedicated processes.
> 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the
> ShuffleHandler restart. If ShuffleHandler runs as a separate process,
> ShuffleHandler can continue to run during NM restart. NM can reconnect the
> the running ShuffleHandler after restart.
> 2. Resource management. It is possible another type of AuxiliaryServices will
> be implemented. AuxiliaryServices are considered YARN application specific
> and could consume lots of resources. Running AuxiliaryServices in separate
> processes allow easier resource management. NM could potentially stop a
> specific AuxiliaryServices process from running if it consumes resource way
> above its allocation.
> Here are some high level ideas:
> 1. NM provides a hosting process for each AuxiliaryService. Existing
> AuxiliaryService API doesn't change.
> 2. The hosting process provides RPC server for AuxiliaryService proxy object
> inside NM to connect to.
> 3. When we rolling restart NM, the existing AuxiliaryService processes will
> continue to run. NM could reconnect to the running AuxiliaryService processes
> upon restart.
> 4. Policy and resource management of AuxiliaryServices. So far we don't have
> immediate need for this. AuxiliaryService could run inside a container and
> its resource utilization could be taken into account by RM and RM could
> consider a specific type of applications overutilize cluster resource.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[
https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667638#comment-15667638
]
Varun Vasudev commented on YARN-1593:
-
[~asuresh] -
{quote}
Thanks for driving this Varun Vasudev
At first glance, this looks similar in spirit to YARN-5501, and maybe even
supersedes it. It would be advantageous to model pooled containers as a system
container.
Further to the point raised by Hitesh Shah about formalizing how we affinitize
an application's container to a Node on a which a dependent system container is
run, we were also investigating a scenario where an application might also need
a countable number of system containers on a Node. An initial thought was to
probably expose the container as a Generalized resource (YARN-3926). For eg,
assume spark Executors can be started as Pre-started containers on select
nodes. Assume a node A has 2 pre-started spark executors, and Node B has 4. A
spark app might have 3 ContainerRequests that requires <4 VCores, 2 GB, 2
spark-executors>, in which case the ResourceManager will ensure that 1 such
container is allocated on Node A and 2 on Node B.
Thoughts ?
{quote}
I think there's quite a bit of overlap. Couple of questions about pooled
containers -
1) If they fail to come up should the NM continue to accept container requests
so should it stop accepting container requests?
2) Are they meant to run on a subset of nodes or on all nodes? Is this
controlled by an admin?
Like I mentioned to Hitesh above - the affinity stuff is something we think is
the long term solution, but we also realize that a solution which is
essentially "launch this container on every node" will help bridge the gap for
now. Hence, the inclusion of both in the design doc.
> support out-of-proc AuxiliaryServices
> -
>
> Key: YARN-1593
> URL: https://issues.apache.org/jira/browse/YARN-1593
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, rolling upgrade
>Reporter: Ming Ma
>Assignee: Varun Vasudev
> Attachments: SystemContainersandSystemServices.pdf
>
>
> AuxiliaryServices such as ShuffleHandler currently run in the same process as
> NM. There are some benefits to host them in dedicated processes.
> 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the
> ShuffleHandler restart. If ShuffleHandler runs as a separate process,
> ShuffleHandler can continue to run during NM restart. NM can reconnect the
> the running ShuffleHandler after restart.
> 2. Resource management. It is possible another type of AuxiliaryServices will
> be implemented. AuxiliaryServices are considered YARN application specific
> and could consume lots of resources. Running AuxiliaryServices in separate
> processes allow easier resource management. NM could potentially stop a
> specific AuxiliaryServices process from running if it consumes resource way
> above its allocation.
> Here are some high level ideas:
> 1. NM provides a hosting process for each AuxiliaryService. Existing
> AuxiliaryService API doesn't change.
> 2. The hosting process provides RPC server for AuxiliaryService proxy object
> inside NM to connect to.
> 3. When we rolling restart NM, the existing AuxiliaryService processes will
> continue to run. NM could reconnect to the running AuxiliaryService processes
> upon restart.
> 4. Policy and resource management of AuxiliaryServices. So far we don't have
> immediate need for this. AuxiliaryService could run inside a container and
> its resource utilization could be taken into account by RM and RM could
> consider a specific type of applications overutilize cluster resource.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[
https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667624#comment-15667624
]
Varun Vasudev commented on YARN-1593:
-
[~hitesh] -
{quote}
My concern is around the feedback loop in terms of failure handling by the apps
when the system container dies at any of the following points:
system container dies before an allocated container is launched on that node
it dies while a container is running
it dies after a container has completed
Would applications that define affinity to these system services now be getting
updates (notifications) when system service containers go down or come back up?
{quote}
All of these are questions that we have to solve for the general services
scenarios and I suspect that they might take some time to get right. Our
solution till we have a well rounded story for these questions is to use the
second method I mentioned above where we launch the Tez shuffle service on
every node. That way Tez doesn't need to change any behaviour for now. Once we
have the services scheduling and notification pieces sorted out we can start
moving to the affinity model.
{quote}
In addition to the feedback loop, is there any behavior change as a result of
this? i.e. if the system container is not alive, will the app container still
get launched given that its dependent service is down ( for shuffle, this might
be ok if the system container eventually comes up but there might be other
services that provide more synchronous functionality such as a caching layer?
{quote}
This depends on whether it's a system service or a system container (the
difference is that the first one has an AM running whereas the second is more
like auxiliary services running as a container). In case of system containers -
the NM will stop accepting container requests until the system container is
back up. In case of the system service, the NM will continue to accept
container requests.
> support out-of-proc AuxiliaryServices
> -
>
> Key: YARN-1593
> URL: https://issues.apache.org/jira/browse/YARN-1593
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager, rolling upgrade
>Reporter: Ming Ma
>Assignee: Varun Vasudev
> Attachments: SystemContainersandSystemServices.pdf
>
>
> AuxiliaryServices such as ShuffleHandler currently run in the same process as
> NM. There are some benefits to host them in dedicated processes.
> 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the
> ShuffleHandler restart. If ShuffleHandler runs as a separate process,
> ShuffleHandler can continue to run during NM restart. NM can reconnect the
> the running ShuffleHandler after restart.
> 2. Resource management. It is possible another type of AuxiliaryServices will
> be implemented. AuxiliaryServices are considered YARN application specific
> and could consume lots of resources. Running AuxiliaryServices in separate
> processes allow easier resource management. NM could potentially stop a
> specific AuxiliaryServices process from running if it consumes resource way
> above its allocation.
> Here are some high level ideas:
> 1. NM provides a hosting process for each AuxiliaryService. Existing
> AuxiliaryService API doesn't change.
> 2. The hosting process provides RPC server for AuxiliaryService proxy object
> inside NM to connect to.
> 3. When we rolling restart NM, the existing AuxiliaryService processes will
> continue to run. NM could reconnect to the running AuxiliaryService processes
> upon restart.
> 4. Policy and resource management of AuxiliaryServices. So far we don't have
> immediate need for this. AuxiliaryService could run inside a container and
> its resource utilization could be taken into account by RM and RM could
> consider a specific type of applications overutilize cluster resource.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665094#comment-15665094 ] Hitesh Shah commented on YARN-1593: --- Thanks [~vvasudev]. It does so partially. My concern is around the feedback loop in terms of failure handling by the apps when the system container dies at any of the following points: - system container dies before an allocated container is launched on that node - it dies while a container is running - it dies after a container has completed Would applications that define affinity to these system services now be getting updates (notifications) when system service containers go down or come back up? In addition to the feedback loop, is there any behavior change as a result of this? i.e. if the system container is not alive, will the app container still get launched given that its dependent service is down ( for shuffle, this might be ok if the system container eventually comes up but there might be other services that provide more synchronous functionality such as a caching layer? > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Varun Vasudev > Attachments: SystemContainersandSystemServices.pdf > > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15664849#comment-15664849 ] Arun Suresh commented on YARN-1593: --- Thanks for driving this [~vvasudev] At first glance, this looks similar in spirit to YARN-5501, and maybe even supersedes it. It would be advantageous to model pooled containers as a system container. Further to the point raised by [~hitesh] about formalizing how we affinitize an application's container to a Node on a which a dependent system container is run, we were also investigating a scenario where an application might also need a countable number of system containers on a Node. An initial thought was to probably expose the container as a Generalized resource (YARN-3926). For eg, assume spark Executors can be started as Pre-started containers on select nodes. Assume a node A has 2 pre-started spark executors, and Node B has 4. A spark app might have 3 ContainerRequests that requires <4 VCores, 2 GB, 2 spark-executors>, in which case the ResourceManager will ensure that 1 such container is allocated on Node A and 2 on Node B. Thoughts ? > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Varun Vasudev > Attachments: SystemContainersandSystemServices.pdf > > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663162#comment-15663162 ] Varun Vasudev commented on YARN-1593: - Good question [~hitesh] - this is one piece where we're looking for feedback on the approach. bq. The doc does not seem to cover how user applications can define dependencies on these system services. For example, how to ensure that an MR/Tez/xyz container that requires the shuffle service does not get launched on a node where the system service is not running. This has 2 aspects - firstly how to ensure container allocations happen on correct nodes where these services are running and secondly, the service might be down when the container actually gets launched and therefore how the behavior will change as a result ( does the container eventually fail, does the NM itself stop the launch of the container and send an error back, etc). There are two modes to system containers and services - and I suspect we need a hybrid mode. The first mode is to launch them as YARN services(e.g. Tez shuffle service). Tez would then add an affinity requirement between the containers it launches and the Tez shuffle service containers. This would require changes on both the application and YARN level. The second mode is to launch Tez shuffle on all nodes (like we do with auxiliary services today) as "system" containers which are managed by YARN. The NMs will not accept container requests until the system containers are up and running. In this mode - Tez requires no change at all - since the Tez shuffle is running on every container. Does that answer your question? > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Varun Vasudev > Attachments: SystemContainersandSystemServices.pdf > > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15651723#comment-15651723 ] Hitesh Shah commented on YARN-1593: --- [~vvasudev] One question on the design doc. The doc does not seem to cover how user applications can define dependencies on these system services. For example, how to ensure that an MR/Tez/xyz container that requires the shuffle service does not get launched on a node where the system service is not running. This has 2 aspects - firstly how to ensure container allocations happen on correct nodes where these services are running and secondly, the service might be down when the container actually gets launched and therefore how the behavior will change as a result ( does the container eventually fail, does the NM itself stop the launch of the container and send an error back, etc). > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Varun Vasudev > Attachments: SystemContainersandSystemServices.pdf > > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627911#comment-15627911 ] Varun Vasudev commented on YARN-1593: - [~haibochen] - my apologies - I was out for a few days. I'll have a document ready soon. > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Junping Du > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612519#comment-15612519 ] Haibo Chen commented on YARN-1593: -- [~djp] Any update? > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Junping Du > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15574396#comment-15574396 ] Junping Du commented on YARN-1593: -- Thanks [~haibochen] for checking me on this. Actually, [~vvasudev] and I are working on design for this issue. We will put up a design doc for review in next one day or 2. It would be great if you can help to review and comments afterwards. > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Junping Du > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15573534#comment-15573534 ] Haibo Chen commented on YARN-1593: -- Hi, [~djp]. Are you actively working on this? As part of the ATS v2 effort, I have been recently looking at this issue. If you have not started working on this, mind if I take it over? > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Junping Du > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1593) support out-of-proc AuxiliaryServices
[ https://issues.apache.org/jira/browse/YARN-1593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15102821#comment-15102821 ] Sangjin Lee commented on YARN-1593: --- Just a quick observation that making auxiliary services out of process still likely requires some kind of classloading isolation. It will still run hadoop code, and the hadoop classpath will carry over into the out-of-process aux service. > support out-of-proc AuxiliaryServices > - > > Key: YARN-1593 > URL: https://issues.apache.org/jira/browse/YARN-1593 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager, rolling upgrade >Reporter: Ming Ma >Assignee: Junping Du > > AuxiliaryServices such as ShuffleHandler currently run in the same process as > NM. There are some benefits to host them in dedicated processes. > 1. NM rolling restart. If we want to upgrade YARN , NM restart will force the > ShuffleHandler restart. If ShuffleHandler runs as a separate process, > ShuffleHandler can continue to run during NM restart. NM can reconnect the > the running ShuffleHandler after restart. > 2. Resource management. It is possible another type of AuxiliaryServices will > be implemented. AuxiliaryServices are considered YARN application specific > and could consume lots of resources. Running AuxiliaryServices in separate > processes allow easier resource management. NM could potentially stop a > specific AuxiliaryServices process from running if it consumes resource way > above its allocation. > Here are some high level ideas: > 1. NM provides a hosting process for each AuxiliaryService. Existing > AuxiliaryService API doesn't change. > 2. The hosting process provides RPC server for AuxiliaryService proxy object > inside NM to connect to. > 3. When we rolling restart NM, the existing AuxiliaryService processes will > continue to run. NM could reconnect to the running AuxiliaryService processes > upon restart. > 4. Policy and resource management of AuxiliaryServices. So far we don't have > immediate need for this. AuxiliaryService could run inside a container and > its resource utilization could be taken into account by RM and RM could > consider a specific type of applications overutilize cluster resource. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
