[
https://issues.apache.org/jira/browse/YARN-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Yang updated YARN-7217:
----------------------------
Description:
API service for deploy, and manage YARN services have several limitations.
The update service API provides multiple functions:
# Stopping a service.
# Start a service.
# Increase or decrease number of containers.
The overloading is buggy depending on how the configuration should be applied.
Scenario 1
A user retrieves Service object from getService call, and the Service object
contains state: STARTED. The user would like to increase number of containers
for the deployed service. The JSON has been updated to increase container
count. The PUT method does not actually increase container count.
Scenario 2
A user retrieves Service object from getService call, and the Service object
contains state: STOPPED. The user would like to make a environment
configuration change. The configuration does not get updated after PUT method.
This is possible to address by rearranging the logic of START/STOP after
configuration update. However, there are other potential combinations that can
break PUT method. For example, user like to make configuration changes, but
not yet restart the service until a later time.
The alternative is to separate the PUT method into PUT method for configuration
vs status. This increase the number of action that can be performed. New API
could look like:
{code}
@PUT
/ws/v1/services/[service_name]/spec
Request Data:
{
"name":"[service_name]",
"number_of_containers": 5
}
{code}
{code}
@PUT
/ws/v1/services/[service_name]/state
Request data:
{
"name": "[service_name]",
"state": "STOPPED|STARTED"
}
{code}
Scenario 3
There is no API to list all deployed applications by the same user.
Scenario 4
Desired state (spec) and current state are represented by the same Service
object. There is no easy way to identify "state" is desired state to reach or,
the current state of the service. It would be nice to have ability to retrieve
both desired state, and current state with separated entry points. By
implementing /spec and /state, it can resolve this problem.
Scenario 5
List all services deploy by the same user can trigger a directory listing
operation on namenode if hdfs is used as storage for metadata. When hundred of
users use Service UI to view or deploy applications, this will trigger denial
of services attack on namenode. The sparse small metadata files also reduce
efficiency of Namenode memory usage. Hence, a cache layer for storing service
metadata would be nice.
was:
API service for deploy, and manage YARN services have several limitations.
The update service API provides multiple functions:
# Stopping a service.
# Start a service.
# Increase or decrease number of containers.
The overloading is buggy depending on how the configuration should be applied.
Scenario 1
A user retrieves Service object from getService call, and the Service object
contains state: STARTED. The user would like to increase number of containers
for the deployed service. The JSON has been updated to increase container
count. The PUT method does not actually increase container count.
Scenario 2
A user retrieves Service object from getService call, and the Service object
contains state: STOPPED. The user would like to make a environment
configuration change. The configuration does not get updated after PUT method.
This is possible to address by rearranging the logic of START/STOP after
configuration update. However, there are other potential combinations that can
break PUT method. For example, user like to make configuration changes, but
not yet restart the service until a later time.
The alternative is to separate the PUT method into PUT method for configuration
vs status. This increase the number of action that can be performed. New API
could look like:
{code}
@PUT
/ws/v1/services/[service_name]/spec
Request Data:
{
"name":"[service_name]",
"number_of_containers": 5
}
{code}
{code}
@PUT
/ws/v1/services/[service_name]/state
Request data:
{
"name": "[service_name]",
"state": "STOPPED|STARTED"
}
{code}
Scenario 3
There is no API to list all deployed applications by the same user.
Scenario 4
Desired state (spec) and current state are represented by the same Service
object. There is no easy way to identify "state" is desired state to reach or,
the current state of the service. It would be nice to have ability to retrieve
both desired state, and current state with separated entry points. By
implementing /spec and /state, it can resolve this problem.
Scenario 5
List all services deploy by the same user will trigger a directory listing
operation on namenode. When hundred of users use Service UI to view or deploy
applications, this will trigger denial of services attack on namenode.
Recommendation is to have a cache layer for storing service metadata.
> Improve API service usability for updating service spec and state
> -----------------------------------------------------------------
>
> Key: YARN-7217
> URL: https://issues.apache.org/jira/browse/YARN-7217
> Project: Hadoop YARN
> Issue Type: Task
> Components: api, applications
> Reporter: Eric Yang
> Assignee: Eric Yang
> Attachments: YARN-7217.yarn-native-services.001.patch,
> YARN-7217.yarn-native-services.002.patch,
> YARN-7217.yarn-native-services.003.patch,
> YARN-7217.yarn-native-services.004.patch
>
>
> API service for deploy, and manage YARN services have several limitations.
> The update service API provides multiple functions:
> # Stopping a service.
> # Start a service.
> # Increase or decrease number of containers.
> The overloading is buggy depending on how the configuration should be applied.
> Scenario 1
> A user retrieves Service object from getService call, and the Service object
> contains state: STARTED. The user would like to increase number of
> containers for the deployed service. The JSON has been updated to increase
> container count. The PUT method does not actually increase container count.
> Scenario 2
> A user retrieves Service object from getService call, and the Service object
> contains state: STOPPED. The user would like to make a environment
> configuration change. The configuration does not get updated after PUT
> method.
> This is possible to address by rearranging the logic of START/STOP after
> configuration update. However, there are other potential combinations that
> can break PUT method. For example, user like to make configuration changes,
> but not yet restart the service until a later time.
> The alternative is to separate the PUT method into PUT method for
> configuration vs status. This increase the number of action that can be
> performed. New API could look like:
> {code}
> @PUT
> /ws/v1/services/[service_name]/spec
> Request Data:
> {
> "name":"[service_name]",
> "number_of_containers": 5
> }
> {code}
> {code}
> @PUT
> /ws/v1/services/[service_name]/state
> Request data:
> {
> "name": "[service_name]",
> "state": "STOPPED|STARTED"
> }
> {code}
> Scenario 3
> There is no API to list all deployed applications by the same user.
> Scenario 4
> Desired state (spec) and current state are represented by the same Service
> object. There is no easy way to identify "state" is desired state to reach
> or, the current state of the service. It would be nice to have ability to
> retrieve both desired state, and current state with separated entry points.
> By implementing /spec and /state, it can resolve this problem.
> Scenario 5
> List all services deploy by the same user can trigger a directory listing
> operation on namenode if hdfs is used as storage for metadata. When hundred
> of users use Service UI to view or deploy applications, this will trigger
> denial of services attack on namenode. The sparse small metadata files also
> reduce efficiency of Namenode memory usage. Hence, a cache layer for storing
> service metadata would be nice.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
