[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7217:
----------------------------
Description:
API service for deploy, and manage YARN services have several limitations.
{{updateService}} API provides multiple functions:
# Stopping a service.
# Start a service.
# Increase or decrease number of containers. (This was removed in YARN-7323).
The overloading is buggy depending on how the configuration should be applied.
h4. Scenario 1
A user retrieves Service object from getService call, and the Service object
contains state: STARTED. The user would like to increase number of containers
for the deployed service. The JSON has been updated to increase container
count. The PUT method does not actually increase container count.
h4. Scenario 2
A user retrieves Service object from getService call, and the Service object
contains state: STOPPED. The user would like to make a environment
configuration change. The configuration does not get updated after PUT method.
This is possible to address by rearranging the logic of START/STOP after
configuration update. However, there are other potential combinations that can
break PUT method. For example, user like to make configuration changes, but
not yet restart the service until a later time.
h4. Scenario 3
There is no API to list all deployed applications by the same user.
h4. Scenario 4
Desired state (spec) and current state are represented by the same Service
object. There is no easy way to identify "state" is desired state to reach or,
the current state of the service. It would be nice to have ability to retrieve
both desired state, and current state with separated entry points. By
implementing /spec and /state, it can resolve this problem.
h4. Scenario 5
List all services deploy by the same user can trigger a directory listing
operation on namenode if hdfs is used as storage for metadata. When hundred of
users use Service UI to view or deploy applications, this will trigger denial
of services attack on namenode. The sparse small metadata files also reduce
efficiency of Namenode memory usage. Hence, a cache layer for storing service
metadata can reduce namenode stress.
h3. Proposed change
ApiService can separate the PUT method into two PUT methods for configuration
changes vs operation changes. New API could look like:
{code}
@PUT
/ws/v1/services/[service_name]/spec
Request Data:
{
"name": "amp",
"components": [
{
"name": "mysql",
"number_of_containers": 2,
"artifact": {
"id": "centos/mysql-57-centos7:latest",
"type": "DOCKER"
},
"run_privileged_container": false,
"launch_command": "",
"resource": {
"cpus": 1,
"memory": "2048"
},
"configuration": {
"env": {
"MYSQL_USER":"${USER}",
"MYSQL_PASSWORD":"password"
}
}
}
],
"quicklinks": {
"Apache Document Root":
"http://httpd.${SERVICE_NAME}.${USER}.${DOMAIN}:8080/";,
"PHP MyAdmin": "http://phpmyadmin.${SERVICE_NAME}.${USER}.${DOMAIN}:8080/";
}
}
{code}
{code}
@PUT
/ws/v1/services/[service_name]/state
Request data:
{
"name": "amp",
"components": [
{
"name": "mysql",
"state": "STOPPED"
}
]
}
{code}
SOLR can be used to cache Yarnfile to improve lookup performance and reduce
stress of namenode small file problems and high frequency lookup. SOLR is
chosen for caching metadata because its indexing feature can be used to build
full text search for application catalog as well.
For service that requires configuration changes to increase or decrease node
count. The calling sequence is:
{code}
# GET /ws/v1/services/{service_name}/spec
# Change number_of_containers to desired number.
# PUT /ws/v1/services/{service_name}/spec to update the spec.
# PUT /ws/v1/services/{service_name}/state to stop existing service.
# PUT /ws/v1/services/{service_name}/state to start service.
{code}
For components that can increase node count without rewrite configuration:
{code}
# GET /ws/v1/services/{service_name}/spec
# Change number_of_containers to desired number.
# PUT /ws/v1/services/{service_name}/spec to update the spec.
# PUT /ws/v1/services/{service_name}/component/{component_name} to change node
count.
{code}
was:
API service for deploy, and manage YARN services have several limitations.
{{updateService}} API provides multiple functions:
# Stopping a service.
# Start a service.
# Increase or decrease number of containers. (This was removed in YARN-7323).
The overloading is buggy depending on how the configuration should be applied.
h4. Scenario 1
A user retrieves Service object from getService call, and the Service object
contains state: STARTED. The user would like to increase number of containers
for the deployed service. The JSON has been updated to increase container
count. The PUT method does not actually increase container count.
h4. Scenario 2
A user retrieves Service object from getService call, and the Service object
contains state: STOPPED. The user would like to make a environment
configuration change. The configuration does not get updated after PUT method.
This is possible to address by rearranging the logic of START/STOP after
configuration update. However, there are other potential combinations that can
break PUT method. For example, user like to make configuration changes, but
not yet restart the service until a later time.
h4. Scenario 3
There is no API to list all deployed applications by the same user.
h4. Scenario 4
Desired state (spec) and current state are represented by the same Service
object. There is no easy way to identify "state" is desired state to reach or,
the current state of the service. It would be nice to have ability to retrieve
both desired state, and current state with separated entry points. By
implementing /spec and /state, it can resolve this problem.
h4. Scenario 5
List all services deploy by the same user can trigger a directory listing
operation on namenode if hdfs is used as storage for metadata. When hundred of
users use Service UI to view or deploy applications, this will trigger denial
of services attack on namenode. The sparse small metadata files also reduce
efficiency of Namenode memory usage. Hence, a cache layer for storing service
metadata can reduce namenode stress.
h3. Proposed change
ApiService can separate the PUT method into two PUT methods for configuration
changes vs operation changes. New API could look like:
{code}
@PUT
/ws/v1/services/[service_name]/spec
Request Data:
{
"name": "amp",
"components": [
{
"name": "mysql",
"number_of_containers": 2,
"artifact": {
"id": "centos/mysql-57-centos7:latest",
"type": "DOCKER"
},
"run_privileged_container": false,
"launch_command": "",
"resource": {
"cpus": 1,
"memory": "2048"
},
"configuration": {
"env": {
"MYSQL_USER":"${USER}",
"MYSQL_PASSWORD":"password"
}
}
}
],
"quicklinks": {
"Apache Document Root":
"http://httpd.${SERVICE_NAME}.${USER}.${DOMAIN}:8080/";,
"PHP MyAdmin": "http://phpmyadmin.${SERVICE_NAME}.${USER}.${DOMAIN}:8080/";
}
}
{code}
{code}
@PUT
/ws/v1/services/[service_name]/state
Request data:
{
"name": "amp",
"components": [
{
"name": "mysql",
"state": "STOPPED"
}
]
}
{code}
SOLR can be used to cache Yarnfile to improve lookup performance and reduce
stress of namenode small file problems and high frequency lookup. SOLR is
chosen for caching metadata because its indexing feature can be used to build
full text search for application catalog as well.
For service that requires configuration to increase or decrease node count.
The calling sequence is:
{code}
# GET /ws/v1/services/{service_name}/spec
# Change number_of_containers to desired number.
# PUT /ws/v1/services/{service_name}/spec to update the spec.
# PUT /ws/v1/services/{service_name}/state to stop existing service.
# PUT /ws/v1/services/{service_name}/state to start service.
{code}
For components that can increase node count without rewrite configuration:
{code}
# GET /ws/v1/services/{service_name}/spec
# Change number_of_containers to desired number.
# PUT /ws/v1/services/{service_name}/spec to update the spec.
# PUT /ws/v1/services/{service_name}/component/{component_name} to change node
count.
{code}
> Improve API service usability for updating service spec and state
> -----------------------------------------------------------------
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch,
> YARN-7217.yarn-native-services.002.patch,
> YARN-7217.yarn-native-services.003.patch,
> YARN-7217.yarn-native-services.004.patch
>
>
> API service for deploy, and manage YARN services have several limitations.
> {{updateService}} API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers. (This was removed in YARN-7323).
> The overloading is buggy depending on how the configuration should be applied.
> h4. Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> h4. Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> h4. Scenario 3
> There is no API to list all deployed applications by the same user.
> h4. Scenario 4
> Desired state (spec) and current state are represented by the same Service
> object. There is no easy way to identify "state" is desired state to reach
> or, the current state of the service. It would be nice to have ability to
> retrieve both desired state, and current state with separated entry points.
> By implementing /spec and /state, it can resolve this problem.
> h4. Scenario 5
> List all services deploy by the same user can trigger a directory listing
> operation on namenode if hdfs is used as storage for metadata. When hundred
> of users use Service UI to view or deploy applications, this will trigger
> denial of services attack on namenode. The sparse small metadata files also
> reduce efficiency of Namenode memory usage. Hence, a cache layer for storing
> service metadata can reduce namenode stress.
> h3. Proposed change
> ApiService can separate the PUT method into two PUT methods for configuration
> changes vs operation changes. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/spec
> Request Data:
> {
> "name": "amp",
> "components": [
> {
> "name": "mysql",
> "number_of_containers": 2,
> "artifact": {
> "id": "centos/mysql-57-centos7:latest",
> "type": "DOCKER"
> },
> "run_privileged_container": false,
> "launch_command": "",
> "resource": {
> "cpus": 1,
> "memory": "2048"
> },
> "configuration": {
> "env": {
> "MYSQL_USER":"${USER}",
> "MYSQL_PASSWORD":"password"
> }
> }
> }
> ],
> "quicklinks": {
> "Apache Document Root":
> "http://httpd.${SERVICE_NAME}.${USER}.${DOMAIN}:8080/";,
> "PHP MyAdmin": "http://phpmyadmin.${SERVICE_NAME}.${USER}.${DOMAIN}:8080/";
> }
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "amp",
> "components": [
> {
> "name": "mysql",
> "state": "STOPPED"
> }
> ]
> }
> {code}
> SOLR can be used to cache Yarnfile to improve lookup performance and reduce
> stress of namenode small file problems and high frequency lookup. SOLR is
> chosen for caching metadata because its indexing feature can be used to build
> full text search for application catalog as well.
> For service that requires configuration changes to increase or decrease node
> count. The calling sequence is:
> {code}
> # GET /ws/v1/services/{service_name}/spec
> # Change number_of_containers to desired number.
> # PUT /ws/v1/services/{service_name}/spec to update the spec.
> # PUT /ws/v1/services/{service_name}/state to stop existing service.
> # PUT /ws/v1/services/{service_name}/state to start service.
> {code}
> For components that can increase node count without rewrite configuration:
> {code}
> # GET /ws/v1/services/{service_name}/spec
> # Change number_of_containers to desired number.
> # PUT /ws/v1/services/{service_name}/spec to update the spec.
> # PUT /ws/v1/services/{service_name}/component/{component_name} to change
> node count.
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
