YARN-4827. Document configuration of ReservationSystem for FairScheduler. 
(Yufei Gu via Subru).

(cherry picked from commit 3fae675383489129b3ca3c66683a1215d0c6edf0)


Project: http://git-wip-us.apache.org/repos/asf/hadoop/repo
Commit: http://git-wip-us.apache.org/repos/asf/hadoop/commit/74aee867
Tree: http://git-wip-us.apache.org/repos/asf/hadoop/tree/74aee867
Diff: http://git-wip-us.apache.org/repos/asf/hadoop/diff/74aee867

Branch: refs/heads/branch-2
Commit: 74aee8673f047414a38fc4f5b0570a90465550ec
Parents: 774a575
Author: Subru Krishnan <su...@apache.org>
Authored: Wed Oct 25 15:07:50 2017 -0700
Committer: Subru Krishnan <su...@apache.org>
Committed: Wed Oct 25 15:32:03 2017 -0700

----------------------------------------------------------------------
 .../reservation/CapacityOverTimePolicy.java     |  3 ---
 .../src/site/markdown/FairScheduler.md          | 24 ++++++++++++++++++++
 .../src/site/markdown/ReservationSystem.md      |  2 +-
 3 files changed, 25 insertions(+), 4 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hadoop/blob/74aee867/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java
----------------------------------------------------------------------
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java
index acd5774..1f3f9bc 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/reservation/CapacityOverTimePolicy.java
@@ -56,9 +56,6 @@ public class CapacityOverTimePolicy extends 
NoOverCommitPolicy {
   private float maxInst;
   private float maxAvg;
 
-  // For now this is CapacityScheduler specific, but given a hierarchy in the
-  // configuration structure of the schedulers (e.g., SchedulerConfiguration)
-  // it should be easy to remove this limitation
   @Override
   public void init(String reservationQueuePath,
       ReservationSchedulerConfiguration conf) {

http://git-wip-us.apache.org/repos/asf/hadoop/blob/74aee867/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
----------------------------------------------------------------------
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
index 941bfa2..e5cb31b 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md
@@ -107,6 +107,8 @@ The allocation file must be in XML format. The format 
contains five types of ele
 
     * **allowPreemptionFrom**: determines whether the scheduler is allowed to 
preempt resources from the queue. The default is true. If a queue has this 
property set to false, this property will apply recursively to all child queues.
 
+    * **reservation**: indicates to the `ReservationSystem` that the queue's 
resources is available for users to reserve. This only applies for leaf queues. 
A leaf queue is not reservable if this property isn't configured.
+
 * **User elements**: which represent settings governing the behavior of 
individual users. They can contain a single property: maxRunningApps, a limit 
on the number of running apps for a particular user.
 
 * **A userMaxAppsDefault element**: which sets the default running app limit 
for any users whose limit is not otherwise specified.
@@ -125,6 +127,12 @@ The allocation file must be in XML format. The format 
contains five types of ele
 
 * **A defaultQueueSchedulingPolicy element**: which sets the default 
scheduling policy for queues; overridden by the schedulingPolicy element in 
each queue if specified. Defaults to "fair".
 
+* **A reservation-agent element**: which sets the class name of the 
implementation of the `ReservationAgent`, which attempts to place the user's 
reservation request in the `Plan`. The default value is 
`org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.AlignedPlannerWithGreedy`.
+
+* **A reservation-policy element**: which sets the class name of the 
implementation of the `SharingPolicy`, which validates if the new reservation 
doesn't violate any invariants. The default value is 
`org.apache.hadoop.yarn.server.resourcemanager.reservation.CapacityOverTimePolicy`.
+
+* **A reservation-planner element**: which sets the class name of the 
implementation of the `Planner`, which is invoked if the `Plan` capacity fall 
below (due to scheduled maintenance or node failuers) the user reserved 
resources. The default value is 
`org.apache.hadoop.yarn.server.resourcemanager.reservation.planning.SimpleCapacityReplanner`,
 which scans the `Plan` and greedily removes reservations in reversed order of 
acceptance (LIFO) till the reserved resources are within the `Plan` capacity.
+
 * **A queuePlacementPolicy element**: which contains a list of rule elements 
that tell the scheduler how to place incoming apps into queues. Rules are 
applied in the order that they are listed. Rules may take arguments. All rules 
accept the "create" argument, which indicates whether the rule can create a new 
queue. "Create" defaults to true; if set to false and the rule would place the 
app in a queue that is not configured in the allocations file, we continue on 
to the next rule. The last rule must be one that can never issue a continue. 
Valid rules are:
 
     * **specified**: the app is placed into the queue it requested. If the app 
requested no queue, i.e. it specified "default", we continue. If the app 
requested a queue name starting or ending with period, i.e. names like ".q1" or 
"q1." will be rejected.
@@ -157,6 +165,9 @@ The allocation file must be in XML format. The format 
contains five types of ele
       <aclSubmitApps>charlie</aclSubmitApps>
       <minResources>5000 mb,0vcores</minResources>
     </queue>
+    <queue name="sample_reservable_queue">
+      <reservation></reservation>
+    </queue>
   </queue>
 
   <queueMaxAMShareDefault>0.5</queueMaxAMShareDefault>
@@ -199,6 +210,19 @@ The root queue's ACLs are "\*" by default which, because 
ACLs are passed down, m
 
 Reservation Access Control Lists (ACLs) allow administrators to control who 
may take reservation actions on particular queues. They are configured with the 
aclAdministerReservations, aclListReservations, and the aclSubmitReservations 
properties, which can be set per queue. Currently the supported administrative 
actions are updating and deleting reservations. An administrator may also 
submit and list *all* reservations on the queue. These properties take values 
in a format like "user1,user2 group1,group2" or " group1,group2". Actions on a 
queue are permitted if the user/group is a member of the reservation ACL. Note 
that any user can update, delete, or list their own reservations. If 
reservation ACLs are enabled but not defined, everyone will have access.
 
+### Configuring `ReservationSystem`
+
+The Fair Scheduler supports the **ReservationSystem** which allows users to 
reserve resources ahead of time. The application can request reserved resources 
at runtime by specifying the `reservationId` during submission. The following 
configuration parameters can be configured in yarn-site.xml for 
`ReservationSystem`.
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.resourcemanager.reservation-system.enable` | *Mandatory* parameter: to 
enable the `ReservationSystem` in the **ResourceManager**. Boolean value 
expected. The default value is *false*, i.e. `ReservationSystem` is not enabled 
by default. |
+| `yarn.resourcemanager.reservation-system.class` | *Optional* parameter: the 
class name of the `ReservationSystem`. The default value is picked based on the 
configured Scheduler, i.e. if `FairScheduler` is configured, then it is 
`FairReservationSystem`. |
+| `yarn.resourcemanager.reservation-system.plan.follower` | *Optional* 
parameter: the class name of the `PlanFollower` that runs on a timer, and 
synchronizes the `FairScheduler` with the `Plan` and vice versa. The default 
value is picked based on the configured Scheduler, i.e. if `FairScheduler` is 
configured, then it is `FairSchedulerPlanFollower`. |
+| `yarn.resourcemanager.reservation-system.planfollower.time-step` | 
*Optional* parameter: the frequency in milliseconds of the `PlanFollower` 
timer. Long value expected. The default value is *1000*. |
+
+The `ReservationSystem` is integrated with the Fair Scheduler queue hierarchy 
and can be configured for and only for leaf queues. The detailed instructions 
are in section `Allocation file format`.
+
 ##Administration
 
 The fair scheduler provides support for administration at runtime through a 
few mechanisms:

http://git-wip-us.apache.org/repos/asf/hadoop/blob/74aee867/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md
----------------------------------------------------------------------
diff --git 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md
 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md
index cd4ae54..ac3269d 100644
--- 
a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md
+++ 
b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/ReservationSystem.md
@@ -60,4 +60,4 @@ With reference to the figure above, a typical reservation 
proceeds as follows:
 Configuring the Reservation System
 ----------------------------------
 
-Configuring the `ReservationSystem` is simple. Currently we have added support 
for *reservations* in both `CapacityScheduler` and `FairScheduler`. You can 
mark any **leaf queue** in the **capacity-scheduler.xml** or 
**fair-scheduler.xml** as available for "reservations" (see 
[CapacityScheduler](./CapacityScheduler.html#Configuring_ReservationSystem_with_CapacityScheduler)
 and the [FairScheduler](./FairScheduler.html) for details). Then the 
capacity/fair share within that queue can be used for making reservations. Jobs 
can still be submitted to the *reservable queue* without a reservation, in 
which case they will be run in best-effort mode in whatever capacity is left 
over by the jobs running within active reservations.
+Configuring the `ReservationSystem` is simple. Currently we have added support 
for *reservations* in both `CapacityScheduler` and `FairScheduler`. You can 
mark any **leaf queue** in the **capacity-scheduler.xml** or 
**fair-scheduler.xml** as available for "reservations" (see 
[CapacityScheduler](./CapacityScheduler.html#Configuring_ReservationSystem_with_CapacityScheduler)
 and the [FairScheduler](./FairScheduler.html#Configuring_ReservationSystem) 
for details). Then the capacity/fair share within that queue can be used for 
making reservations. Jobs can still be submitted to the *reservable queue* 
without a reservation, in which case they will be run in best-effort mode in 
whatever capacity is left over by the jobs running within active reservations.


---------------------------------------------------------------------
To unsubscribe, e-mail: common-commits-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-commits-h...@hadoop.apache.org

Reply via email to