[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221142#comment-16221142
]
Eric Yang commented on YARN-7217:
---------------------------------
I opened YARN-7399 as storage improvement for YARN service metadata.
> Improve API service usability for updating service spec and state
> -----------------------------------------------------------------
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch,
> YARN-7217.yarn-native-services.002.patch,
> YARN-7217.yarn-native-services.003.patch,
> YARN-7217.yarn-native-services.004.patch,
> YARN-7217.yarn-native-services.005.patch
>
>
> API service for deploy, and manage YARN services have several limitations.
> {{updateService}} API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers. (This was removed in YARN-7323).
> The overloading is buggy depending on how the configuration should be applied.
> h4. Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> h4. Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> h4. Scenario 3
> There is no API to list all deployed applications by the same user.
> h4. Scenario 4
> Desired state (spec) and current state are represented by the same Service
> object. There is no easy way to identify "state" is desired state to reach
> or, the current state of the service. It would be nice to have ability to
> retrieve both desired state, and current state with separated entry points.
> By implementing /spec and /state, it can resolve this problem.
> h4. Scenario 5
> List all services deploy by the same user can trigger a directory listing
> operation on namenode if hdfs is used as storage for metadata. When hundred
> of users use Service UI to view or deploy applications, this will trigger
> denial of services attack on namenode. The sparse small metadata files also
> reduce efficiency of Namenode memory usage. Hence, a cache layer for storing
> service metadata can reduce namenode stress.
> h3. Proposed change
> ApiService can separate the PUT method into two PUT methods for configuration
> changes vs operation changes. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/spec
> Request Data:
> {
> "name": "amp",
> "components": [
> {
> "name": "mysql",
> "number_of_containers": 2,
> "artifact": {
> "id": "centos/mysql-57-centos7:latest",
> "type": "DOCKER"
> },
> "run_privileged_container": false,
> "launch_command": "",
> "resource": {
> "cpus": 1,
> "memory": "2048"
> },
> "configuration": {
> "env": {
> "MYSQL_USER":"${USER}",
> "MYSQL_PASSWORD":"password"
> }
> }
> }
> ],
> "quicklinks": {
> "Apache Document Root":
> "http://httpd.${SERVICE_NAME}.${USER}.${DOMAIN}:8080/";,
> "PHP MyAdmin": "http://phpmyadmin.${SERVICE_NAME}.${USER}.${DOMAIN}:8080/";
> }
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "amp",
> "components": [
> {
> "name": "mysql",
> "state": "STOPPED"
> }
> ]
> }
> {code}
> SOLR can be used to cache Yarnfile to improve lookup performance and reduce
> stress of namenode small file problems and high frequency lookup. SOLR is
> chosen for caching metadata because its indexing feature can be used to build
> full text search for application catalog as well.
> For service that requires configuration changes to increase or decrease node
> count. The calling sequence is:
> {code}
> # GET /ws/v1/services/{service_name}/spec
> # Change number_of_containers to desired number.
> # PUT /ws/v1/services/{service_name}/spec to update the spec.
> # PUT /ws/v1/services/{service_name}/state to stop existing service.
> # PUT /ws/v1/services/{service_name}/state to start service.
> {code}
> For components that can increase node count without rewrite configuration:
> {code}
> # GET /ws/v1/services/{service_name}/spec
> # Change number_of_containers to desired number.
> # PUT /ws/v1/services/{service_name}/spec to update the spec.
> # PUT /ws/v1/services/{service_name}/component/{component_name} to change
> node count.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
