Repository: hadoop
Updated Branches:
refs/heads/trunk a668f8e6c -> 7affa3053
YARN-9131. Updated document usage for dynamic auxiliary service.
Contributed by Billie Rinaldi
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/7affa305
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/7affa305
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/7affa305
Branch: refs/heads/trunk
Commit: 7affa3053c9660ea8aee2e3bfe748bbefbae16ea
Parents: a668f8e
Author: Eric Yang <[email protected]>
Authored: Thu Dec 20 18:45:49 2018 -0500
Committer: Eric Yang <[email protected]>
Committed: Thu Dec 20 18:45:49 2018 -0500
----------------------------------------------------------------------
.../PluggableShuffleAndPluggableSort.md | 71 +++++++++++++++
.../src/site/markdown/NodeManager.md | 46 +++++++++-
.../src/site/markdown/NodeManagerRest.md | 93 ++++++++++++++++++++
.../src/site/markdown/TimelineServiceV2.md | 16 ++++
4 files changed, 225 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7affa305/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/PluggableShuffleAndPluggableSort.md
----------------------------------------------------------------------
diff --git
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/PluggableShuffleAndPluggableSort.md
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/PluggableShuffleAndPluggableSort.md
index c868644..6b1caf4 100644
---
a/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/PluggableShuffleAndPluggableSort.md
+++
b/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/site/markdown/PluggableShuffleAndPluggableSort.md
@@ -63,6 +63,9 @@ The collector class configuration may specify a
comma-separated list of collecto
### NodeManager Configuration properties, `yarn-site.xml` in all nodes:
+There are two ways to configure auxiliary services, through a manifest file or
through the Configuration (the old way). If a manifest file is used, auxiliary
service configurations are not read from the Configuration.
+If using a manifest file, the file name should be set in *yarn-site.xml* under
the property `yarn.nodemanager.aux-services.manifest`. The NMs will check this
file for new modifications at an interval specified by
`yarn.nodemanager.aux-services.manifest.reload-ms`. Otherwise, set the
following properties to configure aux services through the Configuration.
+
| **Property** | **Default Value** | **Explanation** |
|:---- |:---- |:---- |
| `yarn.nodemanager.aux-services` | `...,mapreduce_shuffle` | The auxiliary
service name |
@@ -72,6 +75,39 @@ The collector class configuration may specify a
comma-separated list of collecto
#### Example of loading jar file from HDFS:
+Using manifest:
+```
+{
+ "services": [
+ {
+ "name": "mapreduce_shuffle",
+ "version": "1",
+ "configuration": {
+ "properties": {
+ "class.name": "org.apache.hadoop.mapred.ShuffleHandler"
+ }
+ }
+ },
+ {
+ "name": "AuxServiceFromHDFS",
+ "version": "1",
+ "configuration": {
+ "properties": {
+ "class.name": "org.apache.auxtest.AuxServiceFromHDFS2"
+ },
+ "files": [
+ {
+ "src_file": "hdfs:///aux/test/aux-service-hdfs.jar",
+ "type": "STATIC"
+ }
+ ]
+ }
+ }
+ ]
+}
+```
+
+Or using Configuration:
```xml
<configuration>
<property>
@@ -93,6 +129,39 @@ The collector class configuration may specify a
comma-separated list of collecto
#### Example of loading jar file from local file system:
+Using manifest:
+```
+{
+ "services": [
+ {
+ "name": "mapreduce_shuffle",
+ "version": "1",
+ "configuration": {
+ "properties": {
+ "class.name": "org.apache.hadoop.mapred.ShuffleHandler"
+ }
+ }
+ },
+ {
+ "name": "AuxServiceFromHDFS",
+ "version": "1",
+ "configuration": {
+ "properties": {
+ "class.name": "org.apache.auxtest.AuxServiceFromHDFS2"
+ },
+ "files": [
+ {
+ "src_file": "file:///aux/test/aux-service-hdfs.jar",
+ "type": "STATIC"
+ }
+ ]
+ }
+ }
+ ]
+}
+```
+
+Or using Configuration:
```xml
<configuration>
<property>
@@ -117,3 +186,5 @@ The collector class configuration may specify a
comma-separated list of collecto
`yarn.nodemanager.aux-services` property, for example `mapred.shufflex`.
Then the property defining the corresponding class must be
`yarn.nodemanager.aux-services.mapreduce_shufflex.class`.
+Alternatively, if an aux services manifest file is used, the service
+should be added to the service list.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7affa305/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManager.md
----------------------------------------------------------------------
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManager.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManager.md
index 18aeff4..d211198 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManager.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManager.md
@@ -95,6 +95,7 @@ Step 5. Auxiliary services.
* A simple example for the above is the auxiliary service 'ShuffleHandler'
for MapReduce (MR). ShuffleHandler respects the above two requirements already,
so users/admins don't have to do anything for it to support NM restart: (1) The
configuration property **mapreduce.shuffle.port** controls which port the
ShuffleHandler on a NodeManager host binds to, and it defaults to a
non-ephemeral port. (2) The ShuffleHandler service also already supports
recovery of previous state after NM restarts.
+ * There are two ways to configure auxiliary services, through a manifest
file or through the Configuration. If a manifest file is used, auxiliary
service configurations are not read from the Configuration. If no manifest file
is specified, auxiliary services will be loaded via the prior method of using
Configuration properties. One advantage of using the manifest file is that the
NMs will detect changes in auxiliary services specified in the manifest file
(as determined by the service name and version) and will remove or reload
running auxiliary services as needed. To support reloading, AuxiliaryService
implementations must perform any cleanup that is needed during the service stop
phase for the NM to be able to create a new instance of the auxiliary service.
Auxiliary Service Classpath Isolation
-------------------------------------
@@ -102,8 +103,51 @@ Auxiliary Service Classpath Isolation
### Introduction
To launch auxiliary services on a NodeManager, users have to add their jar to
NodeManager's classpath directly, thus put them on the system classloader. But
if multiple versions of the plugin are present on the classpath, there is no
control over which version actually gets loaded. Or if there are any conflicts
between the dependencies introduced by the auxiliary services and the
NodeManager itself, they can break the NodeManager, the auxiliary services, or
both. To solve this issue, we can instantiate auxiliary services using a
classloader that is different from the system classloader.
+### Manifest
+This section describes the auxiliary service manifest for aux-service
classpath isolation.
+
+The manifest file should be set in *yarn-site.xml* under the property
`yarn.nodemanager.aux-services.manifest`. The NMs will check this file for new
modifications at an interval specified by
`yarn.nodemanager.aux-services.manifest.reload-ms`.
+
+An example manifest that configures classpath isolation for a CustomAuxService
follows. One or more files may be specified to make up the classpath of a
service, with jar or archive formats being supported.
+```
+{
+ "services": [
+ {
+ "name": "mapreduce_shuffle",
+ "version": "2",
+ "configuration": {
+ "properties": {
+ "class.name": "org.apache.hadoop.mapred.ShuffleHandler",
+ "mapreduce.shuffle.transfer.buffer.size": "102400",
+ "mapreduce.shuffle.port": "13562"
+ }
+ }
+ },
+ {
+ "name": "CustomAuxService",
+ "version": "1",
+ "configuration": {
+ "properties": {
+ "class.name": "org.aux.CustomAuxService"
+ },
+ "files": [
+ {
+ "src_file": "${remote-dir}/CustomAuxService.jar",
+ "type": "STATIC"
+ },
+ {
+ "src_file": "${remote-dir}/CustomAuxService.tgz",
+ "type": "ARCHIVE"
+ }
+ ]
+ }
+ }
+ ]
+}
+```
+
### Configuration
-This section describes the configuration variables for aux-service classpath
isolation.
+This section describes the configuration variables for aux-service classpath
isolation. Aux services will only be loaded from the configuration if a
manifest file is not specified.
The following settings need to be set in *yarn-site.xml*.
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7affa305/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerRest.md
----------------------------------------------------------------------
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerRest.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerRest.md
index 2a20ca9..03bf4f2 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerRest.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/NodeManagerRest.md
@@ -585,3 +585,96 @@ Response Body:
<containerLogFiles>syslog</containerLogFiles>
</container>
```
+
+Auxiliary Services API
+----------------------
+
+With the auxiliary services API, you can obtain a collection of resources,
each of which represents an auxiliary service. When you run a GET operation on
this resource, you obtain a collection of auxiliary service information objects.
+
+### URI
+
+ * http://nm-http-address:port/ws/v1/node/auxiliaryservices
+
+### HTTP Operations Supported
+
+ * GET
+
+### Query Parameters Supported
+
+ None
+
+### Elements of the *auxiliaryservices* object
+
+When you make a request for the list of auxiliary services, the information
will be returned as collection of service information objects.
+
+| Properties | Data Type | Description |
+|:---- |:---- |:---- |
+| services | array of service information objects(JSON)/zero or more service
information objects (XML) | A collection of service information objects |
+
+### Response Examples
+
+**JSON response**
+
+HTTP Request:
+
+ GET http://nm-http-address:port/ws/v1/node/auxiliaryservices
+
+Response Header:
+
+ HTTP/1.1 200 OK
+ Content-Type: application/json
+ Transfer-Encoding: chunked
+ Server: Jetty(6.1.26)
+
+Response Body:
+
+```json
+{
+ "services": {
+ "service": [
+ {
+ "name": "timeline_collector",
+ "startTime": "2018-12-18 21:24:27",
+ "version": "1"
+ },
+ {
+ "name": "mapreduce_shuffle",
+ "startTime": "2018-12-18 21:24:27",
+ "version": "2"
+ }
+ ]
+ }
+}
+```
+
+**XML response**
+
+HTTP Request:
+
+ GET http://nm-http-address:port/ws/v1/node/auxiliaryservices
+ Accept: application/xml
+
+Response Header:
+
+ HTTP/1.1 200 OK
+ Content-Type: application/xml
+ Content-Length: 299
+ Server: Jetty(6.1.26)
+
+Response Body:
+
+```xml
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<services>
+ <service>
+ <name>timeline_collector</name>
+ <version>1</version>
+ <startTime>2018-12-18 21:00:00</startTime>
+ </service>
+ <service>
+ <name>mapreduce_shuffle</name>
+ <version>2</version>
+ <startTime>2018-12-18 21:00:00</startTime>
+ </service>
+</services>
+```
http://git-wip-us.apache.org/repos/asf/hadoop/blob/7affa305/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
----------------------------------------------------------------------
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
index 86faf6c..96a92fc 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md
@@ -305,6 +305,22 @@ Following are the basic configurations to start Timeline
service v.2:
</property>
```
+If using an aux services manifest instead of setting aux services through the
Configuration, ensure that the manifest services array includes the
timeline\_collector service as follows:
+```
+{
+ "services": [
+ {
+ "name": "timeline_collector",
+ "configuration": {
+ "properties": {
+ "class.name":
"org.apache.hadoop.yarn.server.timelineservice.collector.PerNodeTimelineCollectorsAuxService"
+ }
+ }
+ }
+ ]
+}
+```
+
In addition, you may want to set the YARN cluster name to a reasonably unique
value in case you
are using multiple clusters to store data in the same Apache HBase storage:
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
