[
https://issues.apache.org/jira/browse/YARN-9023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16699649#comment-16699649
]
Eric Yang commented on YARN-9023:
---------------------------------
[~oshevchenko] Thank you for the proposal. The proposal is an extension to
make auxiliary service more generic for YARN daemons. However, the original
design of YARN auxiliary service suffered from a couple short comings:
# The injected service can not share the same jetty server. If the aux service
like to add more REST API endpoint, this needs to route to a separate port.
The end result is to add duplicated efforts to secure newly opened port.
# Auxiliary service must upgrade at the same time when YARN framework upgrades.
This creates risk for upgrading YARN framework that might break auxiliary
service.
# New functionality can only be added with restart of YARN framework.
# Resource used by auxiliary service is not limited by resource management due
to parasitic nature of the design. This can leads to resource leaks by
auxiliary service and worst yet to bring down YARN framework.
# YARN user credential is too powerful, mapreduce shuffle service should not
run with YARN user credential to prevent third party to trigger code injection
exploits via mapreduce.
Those lessons learned from node manager auxiliary service, I think it would be
better to redesign the auxiliary service from scratch. It would be better to
launch the auxiliary service as yarn containers. This will ensure there is a
way to add on new services without interruption to YARN framework, and reduce
security holes that can be exposed to YARN daemons. ATS v2 deployed HBase
auxiliary service is an example of the direction that has been taken to solve
auxiliary service problem. This path allows the community to harden the core
technology and move bell and whistle features into containers. Thoughts?
> Pluggable YARN Auxiliary Services
> ---------------------------------
>
> Key: YARN-9023
> URL: https://issues.apache.org/jira/browse/YARN-9023
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Oleksandr Shevchenko
> Assignee: Oleksandr Shevchenko
> Priority: Major
> Attachments: Pluggable YARN Auxiliary Services Doc v1.pdf
>
>
> The propose is to create customizable auxiliary services infrastructure for
> all YARN services to provide abilities to extends or change functionality for
> all YARN demons.
> Currently, YARN has the ability to add auxiliary services only for
> NodeManager.
> There are some cases when we need to add auxiliary services for some other
> YARN demons. For example, in YARN-2267 proposed to add Auxiliary Service
> support in RM for implementation monitor plugins or some alarm services. We
> faced with one more case, we need to create/setup some storage for YARN demon
> data (for example for RM state store or AM intermediate data) before demon
> started. Also, will be good if we can use some auxiliary service in different
> YARN demons.
> The end users can use YARN Auxiliary Services (YAS) to customize or extend
> the functionality of all YARN demons. We can provide an ability to create aux
> service both for one and for all demons. This will make easy creating of aux
> service for shuffle or just to up some service within another YARN daemon. We
> will get a unified and more extensible architecture for all aux services
> which may need to create in the future.
> Please look at the attached design doc for the details. [^Pluggable YARN
> Auxiliary Services Doc v1.pdf]
> Also, you can add your comments to this doc:
> https://docs.google.com/document/d/1Pxs8eeWZwnlwryPj7gyN8Ej1O-FRBSX_C0ispV9ORT8/edit?usp=sharing
> Thanks a lot for any comments and attention to this feature!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
