[jira] [Commented] (YARN-2986) Support hierarchical and unified scheduler configuration

[Wangda Tan (JIRA)]

    [ 
https://issues.apache.org/jira/browse/YARN-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14319147#comment-14319147
 ] 

Wangda Tan commented on YARN-2986:
----------------------------------

Thanks for [~vinodkv] reporting, a huge +1 for the proposal.

*In addition to Vinod's suggest, I think we should define scheduler conf 
standard first, my proposal is*

{code}
<scheduler>
   <global-option1>value</global-option1>
   <global-option2>value</global-option2>
   <global-option3>value</global-option3>

   <queue name="root">
      <state>running</state>

      <queue-option1>value</queue-option1>
      <queue-option2>value</queue-option2>
      <queue-option3>value</queue-option3>

      <queue name="a">
         <state>running</state>

         <queue name="a1">
                ...
         </queue>
      </queue>

      <queue name="b">
        ...
      </queue>
   </queue>
</scheduler>
{code}

There're several things need to be highlight in above example:
1) There's no "group xml node", instead of
{code}
<children>
        <queue name=1/>
        <queue name=2/>
<children>
{code}

or 
{code}
<scheduler-custom-configuration>
    <!-- Scheduler specific global configuration -->
    <!-- For e.g.
      <maximum-applications>10000</maximum-applications>
      <resource-calculator>DominantResourceCalculator</resource-calculator>
    -->
  </scheduler-custom-configuration>
{code}

I would suggest
{code}
<queue name=1/>
<queue name=2/>
{code}

2) Element vs. Attribute
(Definition of E/A, see http://www.w3schools.com/xml/xml_attributes.asp)
I think we shouldn't mix element/attribute in config file, attribute field 
should be only a limited set of properties. IMO, name/type should be only 
properties that need to put as attribute.
With this, admin will not hesitate about which property needs go to attribute 
and which needs to go Element.

*To Implement this, I think there're two steps:*

1) A parser to support easily get/set values for scheduler conf.
- Hide complexities of a generic XML parser
- Have basic functionalities to handle value inherit, etc.
For each level ({{<key>...</key>}}), data structure could as simple as:
{code}
SchedulerConfNode {
        String getName();
        String getType();
        String get(String key, bool inherit=false, default=null);
        List<Node> getChildren(String key);
}
{code}

2) Common implementation of existing Hadoop scheduler
Since there're some fields common in fair/capacity/fifo scheduler, we can 
extend the {{SchedulerConfNode}} to {{BaseSchedulerConfNode}}, which has more 
methods like:
{code}
BaseSchedulerConfNode : SchedulerConfNode {
        Resource getMinimumAllocation();
        Resource getMaximumAllocation();
        List<String> getSubmitAcls();
        // ...
}
{code}

Thoughts?

> Support hierarchical and unified scheduler configuration
> --------------------------------------------------------
>
>                 Key: YARN-2986
>                 URL: https://issues.apache.org/jira/browse/YARN-2986
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Vinod Kumar Vavilapalli
>
> Today's scheduler configuration is fragmented and non-intuitive, and needs to 
> be improved. Details in comments.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)