[jira] [Commented] (YARN-2986) Support hierarchical and unified scheduler configuration

[Vinod Kumar Vavilapalli (JIRA)]

    [ 
https://issues.apache.org/jira/browse/YARN-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14257310#comment-14257310
 ] 

Vinod Kumar Vavilapalli commented on YARN-2986:
-----------------------------------------------

I propose the following
 - A single unified configuration file called yarn-scheduler.xml
 - If this file is not present, old configuration files will be applied by 
individual schedulers. Otherwise, it is the only configuration file loaded. 
[Backwards compatibility]
 - Proposal for the schema of the unified file

{code:xml}
<scheduler-configuration>

  <scheduler-name><!--values fifo, capacity-scheduler, fair-scheduler. We 
should stop exposing class-names --> </scheduler-name>

  <minimum-allocation-size>memory=1024,vcores=1 <!-- Compact representation of 
resources --><Minimum-allocation-size>
  <maximum-allocation-size>memory=8192,vcores=32 <!-- Compact representation of 
resources --><maximum-allocation-size>

  <scheduler-custom-configuration>
    <!-- Scheduler specific global configuration -->
    <!-- For e.g.
      <maximum-applications>10000</maximum-applications>
      <resource-calculator>DominantResourceCalculator</resource-calculator>
    -->
  </scheduler-custom-configuration>

  <queue>

    <name>root</name>
    <state>running</state>
    <acl-submit-app>*</acl-submit-app>
    <acl-modify-app>admins</acl-modify-app>
    <properties>
       <property key="capacity" value="100" />
        <property key="maxcapacity" value="100" />
    </properties>

    <children>
      <names>default, a</names>

      <queue>
        <name>default</name>
        <state>stopped</state>
        <acl-submit-app>*</acl-submit-app>
        <acl-modify-app>*</acl-modify-app>
        <properties>
           <property key="capacity" value="60" />
            <property key="maxcapacity" value="60" />
        </properties>
      </queue>

      <queue>
            <name>a</name>
            <state>running</state>
            <acl-submit-app>alice,bob</acl-submit-app>
            <acl-modify-app>mark</acl-modify-app>
            <properties>
                <property key="capacity" value="40" />
                <property key="maxcapacity" value="50" />
            </properties>
            <children>
              <names>childOfA</names>
              <state>running</state>
              <acl-submit-app>alice</acl-submit-app>
              <acl-modify-app>mark</acl-modify-app>
              <properties>
                <property key="capacity" value="100" />
                <property key="maxcapacity" value="100" />
              </properties>
            </children>
        </queue>

    </children>
  </queue>
<scheduler-configuration>
{code}

To do:
  - Preemption monitors

Thoughts? Folks focusing on fair-scheduler, will this scheme work?

> Support hierarchical and unified scheduler configuration
> --------------------------------------------------------
>
>                 Key: YARN-2986
>                 URL: https://issues.apache.org/jira/browse/YARN-2986
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>
> Today's scheduler configuration is fragmented and non-intuitive, and needs to 
> be improved. Details in comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)