Repository: hadoop
Updated Branches:
refs/heads/branch-2 12ccdd654 -> dac2021f8
YARN-4826. Document configuration of ReservationSystem for CapacityScheduler.
(Subru Krishnan via wangda)
(cherry picked from commit fcb3fcd4c674d58e5f40b17eff7048e8ae45d151)
Conflicts:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/dac2021f
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/dac2021f
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/dac2021f
Branch: refs/heads/branch-2
Commit: dac2021f870af674ab1a64e937c4f2a388dcc692
Parents: 12ccdd6
Author: Wangda Tan <[email protected]>
Authored: Fri Apr 8 16:21:04 2016 -0700
Committer: Wangda Tan <[email protected]>
Committed: Fri Apr 8 16:28:14 2016 -0700
----------------------------------------------------------------------
.../src/site/markdown/CapacityScheduler.md | 40 ++++++++++++++++++++
1 file changed, 40 insertions(+)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/hadoop/blob/dac2021f/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
----------------------------------------------------------------------
diff --git
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
index 8b845c2..007842a 100644
---
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
+++
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
@@ -24,6 +24,7 @@ Hadoop: Capacity Scheduler
* [Queue Properties](#Queue_Properties)
* [Setup for application priority](#Setup_for_application_priority.)
* [Capacity Scheduler container
preemption](#Capacity_Scheduler_container_preemption)
+ * [Configuring `ReservationSystem` with
`CapacityScheduler`](#Configuring_ReservationSystem_with_CapacityScheduler)
* [Other Properties](#Other_Properties)
* [Reviewing the configuration of the
CapacityScheduler](#Reviewing_the_configuration_of_the_CapacityScheduler)
* [Changing Queue Configuration](#Changing_Queue_Configuration)
@@ -229,6 +230,45 @@ The following configuration parameters can be configured
in yarn-site.xml to con
|:---- |:---- |
| `yarn.scheduler.capacity.<queue-path>.disable_preemption` | This
configuration can be set to `true` to selectively disable preemption of
application containers submitted to a given queue. This property applies only
when system wide preemption is enabled by configuring
`yarn.resourcemanager.scheduler.monitor.enable` to *true* and
`yarn.resourcemanager.scheduler.monitor.policies` to
*ProportionalCapacityPreemptionPolicy*. If this property is not set for a
queue, then the property value is inherited from the queue's parent. Default
value is false.
+###Reservation Properties
+
+ * Reservation Administration & Permissions
+
+ The `CapacityScheduler` supports the following parameters to control the
creation, deletion, update, and listing of reservations. Note that any user can
update, delete, or list their own reservations. If reservation ACLs are enabled
but not defined, everyone will have access. In the examples below, \<queue\> is
the queue name. For example, to set the reservation ACL to administer
reservations on the default queue, use the property
`yarn.scheduler.capacity.root.default.acl_administer_reservations`
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.scheduler.capacity.root.<queue>.acl_administer_reservations` | The ACL
which controls who can *administer* reservations to the given queue. If the
given user/group has necessary ACLs on the given queue or they can submit,
delete, update and list all reservations. ACLs for this property *are not*
inherited from the parent queue if not specified. |
+| `yarn.scheduler.capacity.root.<queue>.acl_list_reservations` | The ACL which
controls who can *list* reservations to the given queue. If the given
user/group has necessary ACLs on the given queue they can list all
applications. ACLs for this property *are not* inherited from the parent queue
if not specified. |
+| `yarn.scheduler.capacity.root.<queue>.acl_submit_reservations` | The ACL
which controls who can *submit* reservations to the given queue. If the given
user/group has necessary ACLs on the given queue they can submit reservations.
ACLs for this property *are not* inherited from the parent queue if not
specified. |
+
+### Configuring `ReservationSystem` with `CapacityScheduler`
+
+ The `CapacityScheduler` supports the **ReservationSystem** which allows users
to reserve resources ahead of time. The application can request the reserved
resources at runtime by specifying the `reservationId` during submission. The
following configuration parameters can be configured in yarn-site.xml for
`ReservationSystem`.
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.reservation-system.enable` | *Mandatory* parameter: to
enable the `ReservationSystem` in the **ResourceManager**. Boolean value
expected. The default value is *false*, i.e. `ReservationSystem` is not enabled
by default. |
+| `yarn.resourcemanager.reservation-system.class` | *Optional* parameter: the
class name of the `ReservationSystem`. The default value is picked based on the
configured Scheduler, i.e. if `CapacityScheduler` is configured, then it is
`CapacityReservationSystem`. |
+| `yarn.resourcemanager.reservation-system.plan.follower` | *Optional*
parameter: the class name of the `PlanFollower` that runs on a timer, and
synchronizes the `CapacityScheduler` with the `Plan` and viceversa. The default
value is picked based on the configured Scheduler, i.e. if `CapacityScheduler`
is configured, then it is `CapacitySchedulerPlanFollower`. |
+| `yarn.resourcemanager.reservation-system.planfollower.time-step` |
*Optional* parameter: the frequency in milliseconds of the `PlanFollower`
timer. Long value expected. The default value is *1000*. |
+
+
+The `ReservationSystem` is integrated with the `CapacityScheduler` queue
hierachy and can be configured for any **LeafQueue** currently. The
`CapacityScheduler` supports the following parameters to tune the
`ReservationSystem`:
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.scheduler.capacity.<queue-path>.reservable` | *Mandatory* parameter:
indicates to the `ReservationSystem` that the queue's resources is available
for users to reserve. Boolean value expected. The default value is *false*,
i.e. reservations are not enabled in *LeafQueues* by default. |
+| `yarn.scheduler.capacity.<queue-path>.reservation-agent` | *Optional*
parameter: the class name that will be used to determine the implementation of
the `ReservationAgent` which will attempt to place the user's reservation
request in the `Plan`. The default value is
*org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy*.
|
+| `yarn.scheduler.capacity.<queue-path>.reservation-move-on-expiry` |
*Optional* parameter to specify to the `ReservationSystem` whether the
applications should be moved or killed to the parent reservable queue
(configured above) when the associated reservation expires. Boolean value
expected. The default value is *true* indicating that the application will be
moved to the reservable queue. |
+| `yarn.scheduler.capacity.<queue-path>.show-reservations-as-queues` |
*Optional* parameter to show or hide the reservation queues in the Scheduler
UI. Boolean value expected. The default value is *false*, i.e. reservation
queues will be hidden. |
+| `yarn.scheduler.capacity.<queue-path>.reservation-policy` | *Optional*
parameter: the class name that will be used to determine the implementation of
the `SharingPolicy` which will validate if the new reservation doesn't violate
any invariants.. The default value is
*org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacityOverTimePolicy*.
|
+| `yarn.scheduler.capacity.<queue-path>.reservation-window` | *Optional*
parameter representing the time in milliseconds for which the `SharingPolicy`
will validate if the constraints in the Plan are satisfied. Long value
expected. The default value is one day. |
+| `yarn.scheduler.capacity.<queue-path>.instantaneous-max-capacity` |
*Optional* parameter: maximum capacity at any time in percentage (%) as a float
that the `SharingPolicy` allows a single user to reserve. The default value is
1, i.e. 100%. |
+| `yarn.scheduler.capacity.<queue-path>.average-capacity` | *Optional*
parameter: the average allowed capacity which will aggregated over the
*ReservationWindow* in percentage (%) as a float that the `SharingPolicy`
allows a single user to reserve. The default value is 1, i.e. 100%. |
+| `yarn.scheduler.capacity.<queue-path>.reservation-planner` | *Optional*
parameter: the class name that will be used to determine the implementation of
the *Planner* which will be invoked if the `Plan` capacity fall below (due to
scheduled maintenance or node failuers) the user reserved resources. The
default value is
*org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.SimpleCapacityReplanner*
which scans the `Plan` and greedily removes reservations in reversed order of
acceptance (LIFO) till the reserved resources are within the `Plan` capacity |
+| `yarn.scheduler.capacity.<queue-path>.reservation-enforcement-window` |
*Optional* parameter representing the time in milliseconds for which the
`Planner` will validate if the constraints in the Plan are satisfied. Long
value expected. The default value is one hour. |
+
###Other Properties
* Resource Calculator
