[
https://issues.apache.org/jira/browse/YARN-7399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16261167#comment-16261167
]
Billie Rinaldi commented on YARN-7399:
--------------------------------------
[~eyang], thanks for providing the image, it makes the goals very clear.
What would the ServiceTemplate look like? I assume the recommended templates
are for displaying in the UI. How are recommended templates populated (that is,
do we need a method for adding service templates for the recommended list)?
Will all users have the same recommended templates?
For the update methods, will we provide a way to merge the
Service/ServiceTemplate into an existing service, or will it just overwrite the
existing one? If it is overwriting, perhaps put would be a better name for the
methods. Would it make sense to support versioning of Service or
ServiceTemplate objects? We don't do that now, so it could be left for future
discussion/implementation.
Do the get/update/remove methods refer to one service instance? In this case
they should have a user parameter as well?
> Yarn services metadata storage improvement
> ------------------------------------------
>
> Key: YARN-7399
> URL: https://issues.apache.org/jira/browse/YARN-7399
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn-native-services
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: YARN-7399.001.patch, YARN-7399.png
>
>
> In Slider, metadata is stored in user's home directory. Slider command line
> interface interacts with HDFS directly to list deployed applications and
> invoke YARN API or HDFS API to provide information to user. This design works
> for a single user manage his/her own applications. When this design has been
> ported to Yarn services, it becomes apparent that this design is difficult to
> list all deployed applications on Hadoop cluster for administrator to manage
> applications. Resource Manager needs to crawl through every user's home
> directory to compile metadata about deployed applications. This can trigger
> high load on namenode to list hundreds or thousands of list directory calls
> owned by different users. Hence, it might be best to centralize the metadata
> storage to Solr or HBase to reduce number of IO calls to namenode for manage
> applications.
> In Slider, one application is composed of metainfo, specifications in json,
> and payload of zip file that contains application code and deployment code.
> Both meta information, and zip file payload are stored in the same
> application directory in HDFS. This works well for distributed applications
> without central application manager that oversee all application.
> In the next generation of application management, we like to centralize
> metainfo and specifications in json to a centralized storage managed by YARN
> user, and keep the payload zip file in user's home directory or in docker
> registry. This arrangement can provide a faster lookup for metainfo when we
> list all deployed applications and services on YARN dashboard.
> When we centralize metainfo to YARN user, we also need to build ACL to
> enforce who can manage applications, and make update. The current proposal is:
> yarn.admin.acl - list of groups that can submit/reconfigure/pause/kill all
> applications
> normal users - submit/reconfigure/pause/kill his/her own applications
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
