http://git-wip-us.apache.org/repos/asf/hadoop/blob/2e44b75f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRestart.md
----------------------------------------------------------------------
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRestart.md
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRestart.md
new file mode 100644
index 0000000..e516afb
--- /dev/null
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ResourceManagerRestart.md
@@ -0,0 +1,181 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+ResourceManger Restart
+======================
+
+* [Overview](#Overview)
+* [Feature](#Feature)
+* [Configurations](#Configurations)
+    * [Enable RM Restart](#Enable_RM_Restart)
+    * [Configure the state-store for persisting the RM 
state](#Configure_the_state-store_for_persisting_the_RM_state)
+    * [How to choose the state-store 
implementation](#How_to_choose_the_state-store_implementation)
+    * [Configurations for Hadoop FileSystem based state-store 
implementation](#Configurations_for_Hadoop_FileSystem_based_state-store_implementation)
+    * [Configurations for ZooKeeper based state-store 
implementation](#Configurations_for_ZooKeeper_based_state-store_implementation)
+    * [Configurations for LevelDB based state-store 
implementation](#Configurations_for_LevelDB_based_state-store_implementation)
+    * [Configurations for work-preserving RM 
recovery](#Configurations_for_work-preserving_RM_recovery)
+* [Notes](#Notes)
+* [Sample Configurations](#Sample_Configurations)
+
+Overview
+--------
+
+ResourceManager is the central authority that manages resources and schedules 
applications running atop of YARN. Hence, it is potentially a single point of 
failure in a Apache YARN cluster.
+`
+This document gives an overview of ResourceManager Restart, a feature that 
enhances ResourceManager to keep functioning across restarts and also makes 
ResourceManager down-time invisible to end-users.
+
+ResourceManager Restart feature is divided into two phases: 
+
+* **ResourceManager Restart Phase 1 (Non-work-preserving RM restart)**: 
Enhance RM to persist application/attempt state and other credentials 
information in a pluggable state-store. RM will reload this information from 
state-store upon restart and re-kick the previously running applications. Users 
are not required to re-submit the applications.
+
+* **ResourceManager Restart Phase 2 (Work-preserving RM restart)**: Focus on 
re-constructing the running state of ResourceManager by combining the container 
statuses from NodeManagers and container requests from ApplicationMasters upon 
restart. The key difference from phase 1 is that previously running 
applications will not be killed after RM restarts, and so applications won't 
lose its work because of RM outage.
+
+Feature
+-------
+
+* **Phase 1: Non-work-preserving RM restart** 
+
+     As of Hadoop 2.4.0 release, only ResourceManager Restart Phase 1 is 
implemented which is described below.
+
+     The overall concept is that RM will persist the application metadata 
(i.e. ApplicationSubmissionContext) in a pluggable state-store when client 
submits an application and also saves the final status of the application such 
as the completion state (failed, killed, finished) and diagnostics when the 
application completes. Besides, RM also saves the credentials like security 
keys, tokens to work in a secure  environment. Any time RM shuts down, as long 
as the required information (i.e.application metadata and the alongside 
credentials if running in a secure environment) is available in the 
state-store, when RM restarts, it can pick up the application metadata from the 
state-store and re-submit the application. RM won't re-submit the applications 
if they were already completed (i.e. failed, killed, finished) before RM went 
down.
+
+     NodeManagers and clients during the down-time of RM will keep polling RM 
until RM comes up. When RM becomes alive, it will send a re-sync command to all 
the NodeManagers and ApplicationMasters it was talking to via heartbeats. As of 
Hadoop 2.4.0 release, the behaviors for NodeManagers and ApplicationMasters to 
handle this command are: NMs will kill all its managed containers and 
re-register with RM. From the RM's perspective, these re-registered 
NodeManagers are similar to the newly joining NMs. AMs(e.g. MapReduce AM) are 
expected to shutdown when they receive the re-sync command. After RM restarts 
and loads all the application metadata, credentials from state-store and 
populates them into memory, it will create a new attempt (i.e. 
ApplicationMaster) for each application that was not yet completed and re-kick 
that application as usual. As described before, the previously running 
applications' work is lost in this manner since they are essentially killed by 
RM via the re-sync co
 mmand on restart.
+
+* **Phase 2: Work-preserving RM restart** 
+
+     As of Hadoop 2.6.0, we further enhanced RM restart feature to address the 
problem to not kill any applications running on YARN cluster if RM restarts.
+
+     Beyond all the groundwork that has been done in Phase 1 to ensure the 
persistency of application state and reload that state on recovery, Phase 2 
primarily focuses on re-constructing the entire running state of YARN cluster, 
the majority of which is the state of the central scheduler inside RM which 
keeps track of all containers' life-cycle, applications' headroom and resource 
requests, queues' resource usage etc. In this way, RM doesn't need to kill the 
AM and re-run the application from scratch as it is done in Phase 1. 
Applications can simply re-sync back with RM and resume from where it were left 
off.
+
+     RM recovers its runing state by taking advantage of the container 
statuses sent from all NMs. NM will not kill the containers when it re-syncs 
with the restarted RM. It continues managing the containers and send the 
container statuses across to RM when it re-registers. RM reconstructs the 
container instances and the associated applications' scheduling status by 
absorbing these containers' information. In the meantime, AM needs to re-send 
the outstanding resource requests to RM because RM may lose the unfulfilled 
requests when it shuts down. Application writers using AMRMClient library to 
communicate with RM do not need to worry about the part of AM re-sending 
resource requests to RM on re-sync, as it is automatically taken care by the 
library itself.
+
+Configurations
+--------------
+
+This section describes the configurations involved to enable RM Restart 
feature.
+
+### Enable RM Restart
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.recovery.enabled` | `true` |
+
+### Configure the state-store for persisting the RM state
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.store.class` | The class name of the state-store to be 
used for saving application/attempt state and the credentials. The available 
state-store implementations are 
`org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore`, a 
ZooKeeper based state-store implementation and 
`org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore`,
 a Hadoop FileSystem based state-store implementation like HDFS and local FS. 
`org.apache.hadoop.yarn.server.resourcemanager.recovery.LeveldbRMStateStore`, a 
LevelDB based state-store implementation. The default value is set to 
`org.apache.hadoop.yarn.server.resourcemanager.recovery.FileSystemRMStateStore`.
 |
+
+### How to choose the state-store implementation
+
+   * **ZooKeeper based state-store**: User is free to pick up any storage to 
set up RM restart, but must use ZooKeeper based state-store to support RM HA. 
The reason is that only ZooKeeper based state-store supports fencing mechanism 
to avoid a split-brain situation where multiple RMs assume they are active and 
can edit the state-store at the same time.
+
+   * **FileSystem based state-store**: HDFS and local FS based state-store are 
supported. Fencing mechanism is not supported.
+
+   * **LevelDB based state-store**: LevelDB based state-store is considered 
more light weight than HDFS and ZooKeeper based state-store. LevelDB supports 
better atomic operations, fewer I/O ops per state update,
+    and far fewer total files on the filesystem. Fencing mechanism is not 
supported.
+
+### Configurations for Hadoop FileSystem based state-store implementation
+
+   Support both HDFS and local FS based state-store implementation. The type 
of file system to be used is determined by the scheme of URI. e.g. 
`hdfs://localhost:9000/rmstore` uses HDFS as the storage and 
`file:///tmp/yarn/rmstore` uses local FS as the storage. If no scheme(`hdfs://` 
or `file://`) is specified in the URI, the type of storage to be used is 
determined by `fs.defaultFS` defined in `core-site.xml`.
+
+* Configure the URI where the RM state will be saved in the Hadoop FileSystem 
state-store.
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.fs.state-store.uri` | URI pointing to the location of 
the FileSystem path where RM state will be stored (e.g. 
hdfs://localhost:9000/rmstore). Default value is 
`${hadoop.tmp.dir}/yarn/system/rmstore`. If FileSystem name is not provided, 
`fs.default.name` specified in **conf/core-site.xml* will be used. |
+
+* Configure the retry policy state-store client uses to connect with the 
Hadoop FileSystem.
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.fs.state-store.retry-policy-spec` | Hadoop FileSystem 
client retry policy specification. Hadoop FileSystem client retry is always 
enabled. Specified in pairs of sleep-time and number-of-retries i.e. (t0, n0), 
(t1, n1), ..., the first n0 retries sleep t0 milliseconds on average, the 
following n1 retries sleep t1 milliseconds on average, and so on. Default value 
is (2000, 500) |
+
+### Configurations for ZooKeeper based state-store implementation
+  
+* Configure the ZooKeeper server address and the root path where the RM state 
is stored.
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.zk-address` | Comma separated list of Host:Port pairs. 
Each corresponds to a ZooKeeper server (e.g. 
"127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002") to be used by the RM for 
storing RM state. |
+| `yarn.resourcemanager.zk-state-store.parent-path` | The full path of the 
root znode where RM state will be stored. Default value is /rmstore. |
+
+* Configure the retry policy state-store client uses to connect with the 
ZooKeeper server.
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.zk-num-retries` | Number of times RM tries to connect 
to ZooKeeper server if the connection is lost. Default value is 500. |
+| `yarn.resourcemanager.zk-retry-interval-ms` | The interval in milliseconds 
between retries when connecting to a ZooKeeper server. Default value is 2 
seconds. |
+| `yarn.resourcemanager.zk-timeout-ms` | ZooKeeper session timeout in 
milliseconds. This configuration is used by the ZooKeeper server to determine 
when the session expires. Session expiration happens when the server does not 
hear from the client (i.e. no heartbeat) within the session timeout period 
specified by this configuration. Default value is 10 seconds |
+
+* Configure the ACLs to be used for setting permissions on ZooKeeper znodes.
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.zk-acl` | ACLs to be used for setting permissions on 
ZooKeeper znodes. Default value is `world:anyone:rwcda` |
+
+### Configurations for LevelDB based state-store implementation
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.leveldb-state-store.path` | Local path where the RM 
state will be stored. Default value is `${hadoop.tmp.dir}/yarn/system/rmstore` |
+
+### Configurations for work-preserving RM recovery
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.work-preserving-recovery.scheduling-wait-ms` | Set the 
amount of time RM waits before allocating new containers on RM work-preserving 
recovery. Such wait period gives RM a chance to settle down resyncing with NMs 
in the cluster on recovery, before assigning new containers to applications.|
+
+Notes
+-----
+
+ContainerId string format is changed if RM restarts with work-preserving 
recovery enabled. It used to be such format:
+
+    Container_{clusterTimestamp}_{appId}_{attemptId}_{containerId}, e.g. 
Container_1410901177871_0001_01_000005.
+
+It is now changed to:
+
+    Container_e{epoch}_{clusterTimestamp}_{appId}_{attemptId}_{containerId}, 
e.g. Container_e17_1410901177871_0001_01_000005.
+ 
+Here, the additional epoch number is a monotonically increasing integer which 
starts from 0 and is increased by 1 each time RM restarts. If epoch number is 
0, it is omitted and the containerId string format stays the same as before.
+
+Sample Configurations
+---------------------
+
+Below is a minimum set of configurations for enabling RM work-preserving 
restart using ZooKeeper based state store.
+
+
+     <property>
+       <description>Enable RM to recover state after starting. If true, then 
+       yarn.resourcemanager.store.class must be specified</description>
+       <name>yarn.resourcemanager.recovery.enabled</name>
+       <value>true</value>
+     </property>
+   
+     <property>
+       <description>The class to use as the persistent store.</description>
+       <name>yarn.resourcemanager.store.class</name>
+       
<value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
+     </property>
+
+     <property>
+       <description>Comma separated list of Host:Port pairs. Each corresponds 
to a ZooKeeper server
+       (e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002") to be used by the 
RM for storing RM state.
+       This must be supplied when using 
org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
+       as the value for yarn.resourcemanager.store.class</description>
+       <name>yarn.resourcemanager.zk-address</name>
+       <value>127.0.0.1:2181</value>
+     </property>
+
+

http://git-wip-us.apache.org/repos/asf/hadoop/blob/2e44b75f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/SecureContainer.md
----------------------------------------------------------------------
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/SecureContainer.md
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/SecureContainer.md
new file mode 100644
index 0000000..f32e460
--- /dev/null
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/SecureContainer.md
@@ -0,0 +1,135 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+YARN Secure Containers
+======================
+
+* [Overview](#Overview)
+
+Overview
+--------
+
+YARN containers in a secure cluster use the operating system facilities to 
offer execution isolation for containers. Secure containers execute under the 
credentials of the job user. The operating system enforces access restriction 
for the container. The container must run as the use that submitted the 
application.
+
+Secure Containers work only in the context of secured YARN clusters.
+
+###Container isolation requirements
+
+  The container executor must access the local files and directories needed by 
the container such as jars, configuration files, log files, shared objects etc. 
Although it is launched by the NodeManager, the container should not have 
access to the NodeManager private files and configuration. Container running 
applications submitted by different users should be isolated and unable to 
access each other files and directories. Similar requirements apply to other 
system non-file securable objects like named pipes, critical sections, LPC 
queues, shared memory etc.
+
+###Linux Secure Container Executor
+
+  On Linux environment the secure container executor is the 
`LinuxContainerExecutor`. It uses an external program called the 
**container-executor**\> to launch the container. This program has the `setuid` 
access right flag set which allows it to launch the container with the 
permissions of the YARN application user.
+
+###Configuration
+
+  The configured directories for `yarn.nodemanager.local-dirs` and 
`yarn.nodemanager.log-dirs` must be owned by the configured NodeManager user 
(`yarn`) and group (`hadoop`). The permission set on these directories must be 
`drwxr-xr-x`.
+
+  The `container-executor` program must be owned by `root` and have the 
permission set `---sr-s---`.
+
+  To configure the `NodeManager` to use the `LinuxContainerExecutor` set the 
following in the **conf/yarn-site.xml**:
+
+```xml
+<property>
+  <name>yarn.nodemanager.container-executor.class</name>
+  
<value>org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor</value>
+</property>
+
+<property>
+  <name>yarn.nodemanager.linux-container-executor.group</name>
+  <value>hadoop</value>
+</property>
+```
+
+  Additionally the LCE requires the `container-executor.cfg` file, which is 
read by the `container-executor` program.
+
+```
+yarn.nodemanager.linux-container-executor.group=#configured value of 
yarn.nodemanager.linux-container-executor.group
+banned.users=#comma separated list of users who can not run applications
+allowed.system.users=#comma separated list of allowed system users
+min.user.id=1000#Prevent other super-users
+```
+
+###Windows Secure Container Executor (WSCE)
+
+  The Windows environment secure container executor is the 
`WindowsSecureContainerExecutor`. It uses the Windows S4U infrastructure to 
launch the container as the YARN application user. The WSCE requires the 
presense of the `hadoopwinutilsvc` service. This services is hosted by 
`%HADOOP_HOME%\bin\winutils.exe` started with the `service` command line 
argument. This service offers some privileged operations that require 
LocalSystem authority so that the NM is not required to run the entire JVM and 
all the NM code in an elevated context. The NM interacts with the 
`hadoopwintulsvc` service by means of Local RPC (LRPC) via calls JNI to the RCP 
client hosted in `hadoop.dll`.
+
+###Configuration
+
+  To configure the `NodeManager` to use the `WindowsSecureContainerExecutor` 
set the following in the **conf/yarn-site.xml**:
+
+```xml
+        <property>
+          <name>yarn.nodemanager.container-executor.class</name>
+          
<value>org.apache.hadoop.yarn.server.nodemanager.WindowsSecureContainerExecutor</value>
+        </property>
+
+        <property>
+          <name>yarn.nodemanager.windows-secure-container-executor.group</name>
+          <value>yarn</value>
+        </property>
+```
+   
+  The hadoopwinutilsvc uses `%HADOOP_HOME%\etc\hadoop\wsce_site.xml` to 
configure access to the privileged operations.
+
+```xml
+<property>
+ 
<name>yarn.nodemanager.windows-secure-container-executor.impersonate.allowed</name>
+  <value>HadoopUsers</value>
+</property>
+
+<property>
+  
<name>yarn.nodemanager.windows-secure-container-executor.impersonate.denied</name>
+  <value>HadoopServices,Administrators</value>
+</property>
+
+<property>
+  <name>yarn.nodemanager.windows-secure-container-executor.allowed</name>
+  <value>nodemanager</value>
+</property>
+
+<property>
+  <name>yarn.nodemanager.windows-secure-container-executor.local-dirs</name>
+  <value>nm-local-dir, nm-log-dirs</value>
+</property>
+
+<property>
+  <name>yarn.nodemanager.windows-secure-container-executor.job-name</name>
+  <value>nodemanager-job-name</value>
+</property>  
+```
+
+  `yarn.nodemanager.windows-secure-container-executor.allowed` should contain 
the name of the service account running the nodemanager. This user will be 
allowed to access the hadoopwintuilsvc functions.
+
+  `yarn.nodemanager.windows-secure-container-executor.impersonate.allowed` 
should contain the users that are allowed to create containers in the cluster. 
These users will be allowed to be impersonated by hadoopwinutilsvc.
+
+  `yarn.nodemanager.windows-secure-container-executor.impersonate.denied` 
should contain users that are explictly forbiden from creating containers. 
hadoopwinutilsvc will refuse to impersonate these users.
+
+  `yarn.nodemanager.windows-secure-container-executor.local-dirs` should 
contain the nodemanager local dirs. hadoopwinutilsvc will allow only file 
operations under these directories. This should contain the same values as 
`$yarn.nodemanager.local-dirs, $yarn.nodemanager.log-dirs` but note that 
hadoopwinutilsvc XML configuration processing does not do substitutions so the 
value must be the final value. All paths must be absolute and no environment 
variable substitution will be performed. The paths are compared 
LOCAL\_INVARIANT case insensitive string comparison, the file path validated 
must start with one of the paths listed in local-dirs configuration. Use comma 
as path separator:`,`
+
+  `yarn.nodemanager.windows-secure-container-executor.job-name` should contain 
an Windows NT job name that all containers should be added to. This 
configuration is optional. If not set, the container is not added to a global 
NodeManager job. Normally this should be set to the job that the NM is assigned 
to, so that killing the NM kills also all containers. Hadoopwinutilsvc will not 
attempt to create this job, the job must exists when the container is launched. 
If the value is set and the job does not exists, container launch will fail 
with error 2 `The system cannot find the file specified`. Note that this global 
NM job is not related to the container job, which always gets created for each 
container and is named after the container ID. This setting controls a global 
job that spans all containers and the parent NM, and as such it requires nested 
jobs. Nested jobs are available only post Windows 8 and Windows Server 2012.
+
+####Useful Links
+
+  * [Exploring S4U Kerberos Extensions in Windows Server 
2003](http://msdn.microsoft.com/en-us/magazine/cc188757.aspx)
+
+  * [Nested 
Jobs](http://msdn.microsoft.com/en-us/library/windows/desktop/hh448388.aspx)
+
+  * [Winutils needs ability to create task as domain 
user](https://issues.apache.org/jira/browse/YARN-1063)
+
+  * [Implement secure Windows Container 
Executor](https://issues.apache.org/jira/browse/YARN-1972)
+
+  * [Remove the need to run NodeManager as privileged account for Windows 
Secure Container Executor](https://issues.apache.org/jira/browse/YARN-2198)
+
+

http://git-wip-us.apache.org/repos/asf/hadoop/blob/2e44b75f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md
----------------------------------------------------------------------
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md
new file mode 100644
index 0000000..4889936
--- /dev/null
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServer.md
@@ -0,0 +1,231 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+YARN Timeline Server
+====================
+
+* [Overview](#Overview)
+* [Current Status](#Current_Status)
+* [Basic Configuration](#Basic_Configuration)
+* [Advanced Configuration](#Advanced_Configuration)
+* [Generic-data related Configuration](#Generic-data_related_Configuration)
+* [Per-framework-date related 
Configuration](#Per-framework-date_related_Configuration)
+* [Running Timeline server](#Running_Timeline_server)
+* [Accessing generic-data via 
command-line](#Accessing_generic-data_via_command-line)
+* [Publishing of per-framework data by 
applications](#Publishing_of_per-framework_data_by_applications)
+
+Overview
+--------
+
+Storage and retrieval of applications' current as well as historic information 
in a generic fashion is solved in YARN through the Timeline Server (previously 
also called Generic Application History Server). This serves two 
responsibilities:
+
+* Generic information about completed applications
+    
+    Generic information includes application level data like queue-name, user 
information etc in the ApplicationSubmissionContext, list of 
application-attempts that ran for an application, information about each 
application-attempt, list of containers run under each application-attempt, and 
information about each container. Generic data is stored by ResourceManager to 
a history-store (default implementation on a file-system) and used by the 
web-UI to display information about completed applications.
+
+* Per-framework information of running and completed applications
+    
+    Per-framework information is completely specific to an application or 
framework. For example, Hadoop MapReduce framework can include pieces of 
information like number of map tasks, reduce tasks, counters etc. Application 
developers can publish the specific information to the Timeline server via 
TimelineClient from within a client, the ApplicationMaster and/or the 
application's containers. This information is then queryable via REST APIs for 
rendering by application/framework specific UIs.
+
+Current Status
+--------------
+
+Timeline sever is a work in progress. The basic storage and retrieval of 
information, both generic and framework specific, are in place. Timeline server 
doesn't work in secure mode yet. The generic information and the per-framework 
information are today collected and presented separately and thus are not 
integrated well together. Finally, the per-framework information is only 
available via RESTful APIs, using JSON type content - ability to install 
framework specific UIs in YARN isn't supported yet.
+
+Basic Configuration
+-------------------
+
+Users need to configure the Timeline server before starting it. The simplest 
configuration you should add in `yarn-site.xml` is to set the hostname of the 
Timeline server.
+
+```xml
+<property>
+  <description>The hostname of the Timeline service web 
application.</description>
+  <name>yarn.timeline-service.hostname</name>
+  <value>0.0.0.0</value>
+</property>
+```
+
+Advanced Configuration
+----------------------
+
+In addition to the hostname, admins can also configure whether the service is 
enabled or not, the ports of the RPC and the web interfaces, and the number of 
RPC handler threads.
+
+```xml
+<property>
+  <description>Address for the Timeline server to start the RPC 
server.</description>
+  <name>yarn.timeline-service.address</name>
+  <value>${yarn.timeline-service.hostname}:10200</value>
+</property>
+
+<property>
+  <description>The http address of the Timeline service web 
application.</description>
+  <name>yarn.timeline-service.webapp.address</name>
+  <value>${yarn.timeline-service.hostname}:8188</value>
+</property>
+
+<property>
+  <description>The https address of the Timeline service web 
application.</description>
+  <name>yarn.timeline-service.webapp.https.address</name>
+  <value>${yarn.timeline-service.hostname}:8190</value>
+</property>
+
+<property>
+  <description>Handler thread count to serve the client RPC 
requests.</description>
+  <name>yarn.timeline-service.handler-thread-count</name>
+  <value>10</value>
+</property>
+
+<property>
+  <description>Enables cross-origin support (CORS) for web services where
+  cross-origin web response headers are needed. For example, javascript making
+  a web services request to the timeline server.</description>
+  <name>yarn.timeline-service.http-cross-origin.enabled</name>
+  <value>false</value>
+</property>
+
+<property>
+  <description>Comma separated list of origins that are allowed for web
+  services needing cross-origin (CORS) support. Wildcards (*) and patterns
+  allowed</description>
+  <name>yarn.timeline-service.http-cross-origin.allowed-origins</name>
+  <value>*</value>
+</property>
+
+<property>
+  <description>Comma separated list of methods that are allowed for web
+  services needing cross-origin (CORS) support.</description>
+  <name>yarn.timeline-service.http-cross-origin.allowed-methods</name>
+  <value>GET,POST,HEAD</value>
+</property>
+
+<property>
+  <description>Comma separated list of headers that are allowed for web
+  services needing cross-origin (CORS) support.</description>
+  <name>yarn.timeline-service.http-cross-origin.allowed-headers</name>
+  <value>X-Requested-With,Content-Type,Accept,Origin</value>
+</property>
+
+<property>
+  <description>The number of seconds a pre-flighted request can be cached
+  for web services needing cross-origin (CORS) support.</description>
+  <name>yarn.timeline-service.http-cross-origin.max-age</name>
+  <value>1800</value>
+</property>
+```
+
+Generic-data related Configuration
+----------------------------------
+
+Users can specify whether the generic data collection is enabled or not, and 
also choose the storage-implementation class for the generic data. There are 
more configurations related to generic data collection, and users can refer to 
`yarn-default.xml` for all of them.
+
+```xml
+<property>
+  <description>Indicate to ResourceManager as well as clients whether
+  history-service is enabled or not. If enabled, ResourceManager starts
+  recording historical data that Timelien service can consume. Similarly,
+  clients can redirect to the history service when applications
+  finish if this is enabled.</description>
+  <name>yarn.timeline-service.generic-application-history.enabled</name>
+  <value>false</value>
+</property>
+
+<property>
+  <description>Store class name for history store, defaulting to file system
+  store</description>
+  <name>yarn.timeline-service.generic-application-history.store-class</name>
+  
<value>org.apache.hadoop.yarn.server.applicationhistoryservice.FileSystemApplicationHistoryStore</value>
+</property>
+```
+
+Per-framework-date related Configuration
+----------------------------------------
+
+Users can specify whether per-framework data service is enabled or not, choose 
the store implementation for the per-framework data, and tune the retention of 
the per-framework data. There are more configurations related to per-framework 
data service, and users can refer to `yarn-default.xml` for all of them.
+
+```xml
+<property>
+  <description>Indicate to clients whether Timeline service is enabled or not.
+  If enabled, the TimelineClient library used by end-users will post entities
+  and events to the Timeline server.</description>
+  <name>yarn.timeline-service.enabled</name>
+  <value>true</value>
+</property>
+
+<property>
+  <description>Store class name for timeline store.</description>
+  <name>yarn.timeline-service.store-class</name>
+  <value>org.apache.hadoop.yarn.server.timeline.LeveldbTimelineStore</value>
+</property>
+
+<property>
+  <description>Enable age off of timeline store data.</description>
+  <name>yarn.timeline-service.ttl-enable</name>
+  <value>true</value>
+</property>
+
+<property>
+  <description>Time to live for timeline store data in 
milliseconds.</description>
+  <name>yarn.timeline-service.ttl-ms</name>
+  <value>604800000</value>
+</property>
+```
+
+Running Timeline server
+-----------------------
+
+Assuming all the aforementioned configurations are set properly, admins can 
start the Timeline server/history service with the following command:
+
+      $ yarn timelineserver
+
+Or users can start the Timeline server / history service as a daemon:
+
+      $ yarn --daemon start timelineserver
+
+Accessing generic-data via command-line
+---------------------------------------
+
+Users can access applications' generic historic data via the command line as 
below. Note that the same commands are usable to obtain the corresponding 
information about running applications.
+
+```
+      $ yarn application -status <Application ID>
+      $ yarn applicationattempt -list <Application ID>
+      $ yarn applicationattempt -status <Application Attempt ID>
+      $ yarn container -list <Application Attempt ID>
+      $ yarn container -status <Container ID>
+```
+
+Publishing of per-framework data by applications
+------------------------------------------------
+
+Developers can define what information they want to record for their 
applications by composing `TimelineEntity` and `TimelineEvent` objects, and put 
the entities and events to the Timeline server via `TimelineClient`. Following 
is an example:
+
+```java
+// Create and start the Timeline client
+TimelineClient client = TimelineClient.createTimelineClient();
+client.init(conf);
+client.start();
+
+TimelineEntity entity = null;
+// Compose the entity
+try {
+  TimelinePutResponse response = client.putEntities(entity);
+} catch (IOException e) {
+  // Handle the exception
+} catch (YarnException e) {
+  // Handle the exception
+}
+
+// Stop the Timeline client
+client.stop();
+```

http://git-wip-us.apache.org/repos/asf/hadoop/blob/2e44b75f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WebApplicationProxy.md
----------------------------------------------------------------------
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WebApplicationProxy.md
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WebApplicationProxy.md
new file mode 100644
index 0000000..8d6187d
--- /dev/null
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WebApplicationProxy.md
@@ -0,0 +1,24 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+Web Application Proxy
+=====================
+
+The Web Application Proxy is part of YARN. By default it will run as part of 
the Resource Manager(RM), but can be configured to run in stand alone mode. The 
reason for the proxy is to reduce the possibility of web based attacks through 
YARN.
+
+In YARN the Application Master(AM) has the responsibility to provide a web UI 
and to send that link to the RM. This opens up a number of potential issues. 
The RM runs as a trusted user, and people visiting that web address will treat 
it, and links it provides to them as trusted, when in reality the AM is running 
as a non-trusted user, and the links it gives to the RM could point to anything 
malicious or otherwise. The Web Application Proxy mitigates this risk by 
warning users that do not own the given application that they are connecting to 
an untrusted site.
+
+In addition to this the proxy also tries to reduce the impact that a malicious 
AM could have on a user. It primarily does this by stripping out cookies from 
the user, and replacing them with a single cookie providing the user name of 
the logged in user. This is because most web based authentication systems will 
identify a user based off of a cookie. By providing this cookie to an untrusted 
application it opens up the potential for an exploit. If the cookie is designed 
properly that potential should be fairly minimal, but this is just to reduce 
that potential attack vector. The current proxy implementation does nothing to 
prevent the AM from providing links to malicious external sites, nor does it do 
anything to prevent malicious javascript code from running as well. In fact 
javascript can be used to get the cookies, so stripping the cookies from the 
request has minimal benefit at this time.
+
+In the future we hope to address the attack vectors described above and make 
attaching to an AM's web UI safer.

http://git-wip-us.apache.org/repos/asf/hadoop/blob/2e44b75f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WebServicesIntro.md
----------------------------------------------------------------------
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WebServicesIntro.md
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WebServicesIntro.md
new file mode 100644
index 0000000..0e89a50
--- /dev/null
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/WebServicesIntro.md
@@ -0,0 +1,569 @@
+<!---
+  Licensed under the Apache License, Version 2.0 (the "License");
+  you may not use this file except in compliance with the License.
+  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing, software
+  distributed under the License is distributed on an "AS IS" BASIS,
+  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  See the License for the specific language governing permissions and
+  limitations under the License. See accompanying LICENSE file.
+-->
+
+Hadoop YARN - Introduction to the web services REST API's
+==========================================================
+
+* [Overview](#Overview)
+* [URI's](#URIs)
+* [HTTP Requests](#HTTP_Requests)
+    * [Summary of HTTP operations](#Summary_of_HTTP_operations)
+    * [Security](#Security)
+    * [Headers Supported](#Headers_Supported)
+* [HTTP Responses](#HTTP_Responses)
+    * [Compression](#Compression)
+    * [Response Formats](#Response_Formats)
+    * [Response Errors](#Response_Errors)
+    * [Response Examples](#Response_Examples)
+* [Sample Usage](#Sample_Usage)
+
+Overview
+--------
+
+The Hadoop YARN web service REST APIs are a set of URI resources that give 
access to the cluster, nodes, applications, and application historical 
information. The URI resources are grouped into APIs based on the type of 
information returned. Some URI resources return collections while others return 
singletons.
+
+URI's
+-----
+
+The URIs for the REST-based Web services have the following syntax:
+
+      http://{http address of service}/ws/{version}/{resourcepath}
+
+The elements in this syntax are as follows:
+
+      {http address of service} - The http address of the service to get 
information about. 
+                                  Currently supported are the ResourceManager, 
NodeManager, 
+                                  MapReduce application master, and history 
server.
+      {version} - The version of the APIs. In this release, the version is v1.
+      {resourcepath} - A path that defines a singleton resource or a 
collection of resources. 
+
+HTTP Requests
+-------------
+
+To invoke a REST API, your application calls an HTTP operation on the URI 
associated with a resource.
+
+### Summary of HTTP operations
+
+Currently only GET is supported. It retrieves information about the resource 
specified.
+
+### Security
+
+The web service REST API's go through the same security as the web UI. If your 
cluster adminstrators have filters enabled you must authenticate via the 
mechanism they specified.
+
+### Headers Supported
+
+      * Accept 
+      * Accept-Encoding
+
+Currently the only fields used in the header is `Accept` and 
`Accept-Encoding`. `Accept` currently supports XML and JSON for the response 
type you accept. `Accept-Encoding` currently supports only gzip format and will 
return gzip compressed output if this is specified, otherwise output is 
uncompressed. All other header fields are ignored.
+
+HTTP Responses
+--------------
+
+The next few sections describe some of the syntax and other details of the 
HTTP Responses of the web service REST APIs.
+
+### Compression
+
+This release supports gzip compression if you specify gzip in the 
Accept-Encoding header of the HTTP request (Accept-Encoding: gzip).
+
+### Response Formats
+
+This release of the web service REST APIs supports responses in JSON and XML 
formats. JSON is the default. To set the response format, you can specify the 
format in the Accept header of the HTTP request.
+
+As specified in HTTP Response Codes, the response body can contain the data 
that represents the resource or an error message. In the case of success, the 
response body is in the selected format, either JSON or XML. In the case of 
error, the resonse body is in either JSON or XML based on the format requested. 
The Content-Type header of the response contains the format requested. If the 
application requests an unsupported format, the response status code is 500. 
Note that the order of the fields within response body is not specified and 
might change. Also, additional fields might be added to a response body. 
Therefore, your applications should use parsing routines that can extract data 
from a response body in any order.
+
+### Response Errors
+
+After calling an HTTP request, an application should check the response status 
code to verify success or detect an error. If the response status code 
indicates an error, the response body contains an error message. The first 
field is the exception type, currently only RemoteException is returned. The 
following table lists the items within the RemoteException error message:
+
+|      Item | Data Type |          Description |
+|:---- |:---- |:---- |
+|   exception |   String |         Exception type |
+| javaClassName |   String |  Java class name of exception |
+|    message |   String | Detailed message of exception |
+
+### Response Examples
+
+#### JSON response with single resource
+
+HTTP Request: GET 
http://rmhost.domain:8088/ws/v1/cluster/app/application\_1324057493980\_0001
+
+Response Status Line: HTTP/1.1 200 OK
+
+Response Header:
+
+      HTTP/1.1 200 OK
+      Content-Type: application/json
+      Transfer-Encoding: chunked
+      Server: Jetty(6.1.26)
+
+Response Body:
+
+```json
+{
+  app":
+  {
+    "id":"application_1324057493980_0001",
+    "user":"user1",
+    "name":"",
+    "queue":"default",
+    "state":"ACCEPTED",
+    "finalStatus":"UNDEFINED",
+    "progress":0,
+    "trackingUI":"UNASSIGNED",
+    "diagnostics":"",
+    "clusterId":1324057493980,
+    "startedTime":1324057495921,
+    "finishedTime":0,
+    "elapsedTime":2063,
+    
"amContainerLogs":"http:\/\/amNM:2\/node\/containerlogs\/container_1324057493980_0001_01_000001",
+    "amHostHttpAddress":"amNM:2"
+  }
+}
+```
+
+#### JSON response with Error response
+
+Here we request information about an application that doesn't exist yet.
+
+HTTP Request: GET 
http://rmhost.domain:8088/ws/v1/cluster/app/application\_1324057493980\_9999
+
+Response Status Line: HTTP/1.1 404 Not Found
+
+Response Header:
+
+      HTTP/1.1 404 Not Found
+      Content-Type: application/json
+      Transfer-Encoding: chunked
+      Server: Jetty(6.1.26)
+
+Response Body:
+
+```json
+{
+   "RemoteException" : {
+      "javaClassName" : "org.apache.hadoop.yarn.webapp.NotFoundException",
+      "exception" : "NotFoundException",
+      "message" : "java.lang.Exception: app with id: 
application_1324057493980_9999 not found"
+   }
+}
+```
+
+Sample Usage
+-------------
+
+You can use any number of ways/languages to use the web services REST API's. 
This example uses the curl command line interface to do the REST GET calls.
+
+In this example, a user submits a MapReduce application to the ResourceManager 
using a command like:
+
+      hadoop jar hadoop-mapreduce-test.jar sleep -Dmapred.job.queue.name=a1 -m 
1 -r 1 -rt 1200000 -mt 20
+
+The client prints information about the job submitted along with the 
application id, similar to:
+
+    12/01/18 04:25:15 INFO mapred.ResourceMgrDelegate: Submitted application 
application_1326821518301_0010 to ResourceManager at 
host.domain.com/10.10.10.10:8032
+    12/01/18 04:25:15 INFO mapreduce.Job: Running job: job_1326821518301_0010
+    12/01/18 04:25:21 INFO mapred.ClientServiceDelegate: The url to track the 
job: host.domain.com:8088/proxy/application_1326821518301_0010/
+    12/01/18 04:25:22 INFO mapreduce.Job: Job job_1326821518301_0010 running 
in uber mode : false
+    12/01/18 04:25:22 INFO mapreduce.Job:  map 0% reduce 0%
+
+The user then wishes to track the application. The users starts by getting the 
information about the application from the ResourceManager. Use the 
--comopressed option to request output compressed. curl handles uncompressing 
on client side.
+
+    curl --compressed -H "Accept: application/json" -X GET 
"http://host.domain.com:8088/ws/v1/cluster/apps/application_1326821518301_0010"; 
+
+Output:
+
+```json
+{
+   "app" : {
+      "finishedTime" : 0,
+      "amContainerLogs" : 
"http://host.domain.com:8042/node/containerlogs/container_1326821518301_0010_01_000001";,
+      "trackingUI" : "ApplicationMaster",
+      "state" : "RUNNING",
+      "user" : "user1",
+      "id" : "application_1326821518301_0010",
+      "clusterId" : 1326821518301,
+      "finalStatus" : "UNDEFINED",
+      "amHostHttpAddress" : "host.domain.com:8042",
+      "progress" : 82.44703,
+      "name" : "Sleep job",
+      "startedTime" : 1326860715335,
+      "elapsedTime" : 31814,
+      "diagnostics" : "",
+      "trackingUrl" : 
"http://host.domain.com:8088/proxy/application_1326821518301_0010/";,
+      "queue" : "a1"
+   }
+}
+```
+
+The user then wishes to get more details about the running application and 
goes directly to the MapReduce application master for this application. The 
ResourceManager lists the trackingUrl that can be used for this application: 
http://host.domain.com:8088/proxy/application\_1326821518301\_0010. This could 
either go to the web browser or use the web service REST API's. The user uses 
the web services REST API's to get the list of jobs this MapReduce application 
master is running:
+
+     curl --compressed -H "Accept: application/json" -X GET 
"http://host.domain.com:8088/proxy/application_1326821518301_0010/ws/v1/mapreduce/jobs";
+
+Output:
+
+```json
+{
+   "jobs" : {
+      "job" : [
+         {
+            "runningReduceAttempts" : 1,
+            "reduceProgress" : 72.104515,
+            "failedReduceAttempts" : 0,
+            "newMapAttempts" : 0,
+            "mapsRunning" : 0,
+            "state" : "RUNNING",
+            "successfulReduceAttempts" : 0,
+            "reducesRunning" : 1,
+            "acls" : [
+               {
+                  "value" : " ",
+                  "name" : "mapreduce.job.acl-modify-job"
+               },
+               {
+                  "value" : " ",
+                  "name" : "mapreduce.job.acl-view-job"
+               }
+            ],
+            "reducesPending" : 0,
+            "user" : "user1",
+            "reducesTotal" : 1,
+            "mapsCompleted" : 1,
+            "startTime" : 1326860720902,
+            "id" : "job_1326821518301_10_10",
+            "successfulMapAttempts" : 1,
+            "runningMapAttempts" : 0,
+            "newReduceAttempts" : 0,
+            "name" : "Sleep job",
+            "mapsPending" : 0,
+            "elapsedTime" : 64432,
+            "reducesCompleted" : 0,
+            "mapProgress" : 100,
+            "diagnostics" : "",
+            "failedMapAttempts" : 0,
+            "killedReduceAttempts" : 0,
+            "mapsTotal" : 1,
+            "uberized" : false,
+            "killedMapAttempts" : 0,
+            "finishTime" : 0
+         }
+      ]
+   }
+}
+```
+
+The user then wishes to get the task details about the job with job id 
job\_1326821518301\_10\_10 that was listed above.
+
+     curl --compressed -H "Accept: application/json" -X GET 
"http://host.domain.com:8088/proxy/application_1326821518301_0010/ws/v1/mapreduce/jobs/job_1326821518301_10_10/tasks";
 
+
+Output:
+
+```json
+{
+   "tasks" : {
+      "task" : [
+         {
+            "progress" : 100,
+            "elapsedTime" : 5059,
+            "state" : "SUCCEEDED",
+            "startTime" : 1326860725014,
+            "id" : "task_1326821518301_10_10_m_0",
+            "type" : "MAP",
+            "successfulAttempt" : "attempt_1326821518301_10_10_m_0_0",
+            "finishTime" : 1326860730073
+         },
+         {
+            "progress" : 72.104515,
+            "elapsedTime" : 0,
+            "state" : "RUNNING",
+            "startTime" : 1326860732984,
+            "id" : "task_1326821518301_10_10_r_0",
+            "type" : "REDUCE",
+            "successfulAttempt" : "",
+            "finishTime" : 0
+         }
+      ]
+   }
+}
+```
+
+The map task has finished but the reduce task is still running. The users 
wishes to get the task attempt information for the reduce task 
task\_1326821518301\_10\_10\_r\_0, note that the Accept header isn't really 
required here since JSON is the default output format:
+
+      curl --compressed -X GET 
"http://host.domain.com:8088/proxy/application_1326821518301_0010/ws/v1/mapreduce/jobs/job_1326821518301_10_10/tasks/task_1326821518301_10_10_r_0/attempts";
+
+Output:
+
+```json
+{
+   "taskAttempts" : {
+      "taskAttempt" : [
+         {
+            "elapsedMergeTime" : 158,
+            "shuffleFinishTime" : 1326860735378,
+            "assignedContainerId" : "container_1326821518301_0010_01_000003",
+            "progress" : 72.104515,
+            "elapsedTime" : 0,
+            "state" : "RUNNING",
+            "elapsedShuffleTime" : 2394,
+            "mergeFinishTime" : 1326860735536,
+            "rack" : "/10.10.10.0",
+            "elapsedReduceTime" : 0,
+            "nodeHttpAddress" : "host.domain.com:8042",
+            "type" : "REDUCE",
+            "startTime" : 1326860732984,
+            "id" : "attempt_1326821518301_10_10_r_0_0",
+            "finishTime" : 0
+         }
+      ]
+   }
+}
+```
+
+The reduce attempt is still running and the user wishes to see the current 
counter values for that attempt:
+
+     curl --compressed -H "Accept: application/json"  -X GET 
"http://host.domain.com:8088/proxy/application_1326821518301_0010/ws/v1/mapreduce/jobs/job_1326821518301_10_10/tasks/task_1326821518301_10_10_r_0/attempts/attempt_1326821518301_10_10_r_0_0/counters";
 
+
+Output:
+
+```json
+{
+   "JobTaskAttemptCounters" : {
+      "taskAttemptCounterGroup" : [
+         {
+            "counterGroupName" : 
"org.apache.hadoop.mapreduce.FileSystemCounter",
+            "counter" : [
+               {
+                  "value" : 4216,
+                  "name" : "FILE_BYTES_READ"
+               }, 
+               {
+                  "value" : 77151,
+                  "name" : "FILE_BYTES_WRITTEN"
+               }, 
+               {
+                  "value" : 0,
+                  "name" : "FILE_READ_OPS"
+               },
+               {
+                  "value" : 0,
+                  "name" : "FILE_LARGE_READ_OPS"
+               },
+               {
+                  "value" : 0,
+                  "name" : "FILE_WRITE_OPS"
+               },
+               {
+                  "value" : 0,
+                  "name" : "HDFS_BYTES_READ"
+               },
+               {
+                  "value" : 0,
+                  "name" : "HDFS_BYTES_WRITTEN"
+               },
+               {
+                  "value" : 0,
+                  "name" : "HDFS_READ_OPS"
+               },
+               {
+                  "value" : 0,
+                  "name" : "HDFS_LARGE_READ_OPS"
+               },
+               {
+                  "value" : 0,
+                  "name" : "HDFS_WRITE_OPS"
+               }
+            ]  
+         }, 
+         {
+            "counterGroupName" : "org.apache.hadoop.mapreduce.TaskCounter",
+            "counter" : [
+               {
+                  "value" : 0,
+                  "name" : "COMBINE_INPUT_RECORDS"
+               }, 
+               {
+                  "value" : 0,
+                  "name" : "COMBINE_OUTPUT_RECORDS"
+               }, 
+               {  
+                  "value" : 1767,
+                  "name" : "REDUCE_INPUT_GROUPS"
+               },
+               {  
+                  "value" : 25104,
+                  "name" : "REDUCE_SHUFFLE_BYTES"
+               },
+               {
+                  "value" : 1767,
+                  "name" : "REDUCE_INPUT_RECORDS"
+               },
+               {
+                  "value" : 0,
+                  "name" : "REDUCE_OUTPUT_RECORDS"
+               },
+               {
+                  "value" : 0,
+                  "name" : "SPILLED_RECORDS"
+               },
+               {
+                  "value" : 1,
+                  "name" : "SHUFFLED_MAPS"
+               },
+               {
+                  "value" : 0,
+                  "name" : "FAILED_SHUFFLE"
+               },
+               {
+                  "value" : 1,
+                  "name" : "MERGED_MAP_OUTPUTS"
+               },
+               {
+                  "value" : 50,
+                  "name" : "GC_TIME_MILLIS"
+               },
+               {
+                  "value" : 1580,
+                  "name" : "CPU_MILLISECONDS"
+               },
+               {
+                  "value" : 141320192,
+                  "name" : "PHYSICAL_MEMORY_BYTES"
+               },
+              {
+                  "value" : 1118552064,
+                  "name" : "VIRTUAL_MEMORY_BYTES"
+               }, 
+               {  
+                  "value" : 73728000,
+                  "name" : "COMMITTED_HEAP_BYTES"
+               }
+            ]
+         },
+         {  
+            "counterGroupName" : "Shuffle Errors",
+            "counter" : [
+               {  
+                  "value" : 0,
+                  "name" : "BAD_ID"
+               },
+               {  
+                  "value" : 0,
+                  "name" : "CONNECTION"
+               },
+               {  
+                  "value" : 0,
+                  "name" : "IO_ERROR"
+               },
+               {  
+                  "value" : 0,
+                  "name" : "WRONG_LENGTH"
+               },
+               {  
+                  "value" : 0,
+                  "name" : "WRONG_MAP"
+               },
+               {  
+                  "value" : 0,
+                  "name" : "WRONG_REDUCE"
+               }
+            ]
+         },
+         {  
+            "counterGroupName" : 
"org.apache.hadoop.mapreduce.lib.output.FileOutputFormatCounter",
+            "counter" : [
+              {  
+                  "value" : 0,
+                  "name" : "BYTES_WRITTEN"
+               }
+            ]
+         }
+      ],
+      "id" : "attempt_1326821518301_10_10_r_0_0"
+   }
+}
+```
+
+The job finishes and the user wishes to get the final job information from the 
history server for this job.
+
+      curl --compressed -X GET 
"http://host.domain.com:19888/ws/v1/history/mapreduce/jobs/job_1326821518301_10_10";
 
+
+Output:
+
+```json
+{
+   "job" : {
+      "avgReduceTime" : 1250784,
+      "failedReduceAttempts" : 0,
+      "state" : "SUCCEEDED",
+      "successfulReduceAttempts" : 1,
+      "acls" : [
+         {
+            "value" : " ",
+            "name" : "mapreduce.job.acl-modify-job"
+         },
+         {
+            "value" : " ",
+            "name" : "mapreduce.job.acl-view-job"
+         }
+      ],
+      "user" : "user1",
+      "reducesTotal" : 1,
+      "mapsCompleted" : 1,
+      "startTime" : 1326860720902,
+      "id" : "job_1326821518301_10_10",
+      "avgMapTime" : 5059,
+      "successfulMapAttempts" : 1,
+      "name" : "Sleep job",
+      "avgShuffleTime" : 2394,
+      "reducesCompleted" : 1,
+      "diagnostics" : "",
+      "failedMapAttempts" : 0,
+      "avgMergeTime" : 2552,
+      "killedReduceAttempts" : 0,
+      "mapsTotal" : 1,
+      "queue" : "a1",
+      "uberized" : false,
+      "killedMapAttempts" : 0,
+      "finishTime" : 1326861986164
+   }
+}
+```
+
+The user also gets the final applications information from the ResourceManager.
+
+      curl --compressed -H "Accept: application/json" -X GET 
"http://host.domain.com:8088/ws/v1/cluster/apps/application_1326821518301_0010"; 
+
+Output:
+
+```json
+{
+   "app" : {
+      "finishedTime" : 1326861991282,
+      "amContainerLogs" : 
"http://host.domain.com:8042/node/containerlogs/container_1326821518301_0010_01_000001";,
+      "trackingUI" : "History",
+      "state" : "FINISHED",
+      "user" : "user1",
+      "id" : "application_1326821518301_0010",
+      "clusterId" : 1326821518301,
+      "finalStatus" : "SUCCEEDED",
+      "amHostHttpAddress" : "host.domain.com:8042",
+      "progress" : 100,
+      "name" : "Sleep job",
+      "startedTime" : 1326860715335,
+      "elapsedTime" : 1275947,
+      "diagnostics" : "",
+      "trackingUrl" : 
"http://host.domain.com:8088/proxy/application_1326821518301_0010/jobhistory/job/job_1326821518301_10_10";,
+      "queue" : "a1"
+   }
+}
+```
\ No newline at end of file

Reply via email to