Wangda Tan updated YARN-2986:
    Attachment: YARN-2986.1.patch

Uploaded a WIP patch, which consists:

1) Implementation of hierarchy CS configuration:
- Basic xml parser utilities
- Base implementation of scheduler hierarchy configuration (can by reused by 
Fair and other schedulers)
- Capacity Scheduler configuration implementation (*)

(*) Capacity Scheduler configuration implementation:
- Since we need to maintain two different configuration style (plain & 
hierarchy), so we need an abstraction layer to make Capacity Scheduler less 
impacted. So now:
- CapacitySchedulerConfiguration becomes an interface, with all getter/setter 
- The original CapacitySchedulerConfiguration is renamed CSPlainConfiguration, 
implements CapacitySchedulerConfiguration
- The new hierarchy configuration is named CSHierarchyConfiguration

2) Changes in Capacity Scheduler to use the new interface -- 
- This is half way done, now calling CapacitySchedulerContext.getConfiguration 
will return the interface class.
- Other part of CapacityScheduler class is still using CSPlainConfiguration

For 2), pending works:
- Add smart detectation code choose to read configuration file
- Eleminate directly using CSPlainConfiguration, this is blocked by an issue in 
existing CapacityScheduler implementation:
In existing CS implementation, CapacitySchedulerConfiguration will 
automatically includes "yarn-site.xml", which is not correct, we need limit 
CapacitySchedulerConfiguration only use "capacity-scheduler.xml".
- Add more tests to make sure CS will work under both configuration file style.

Now config file style

       <queue name="a">

               <queue name="a1">

               <queue name="a2">

Suggestion of moving forward:
I suggest to commit existing patch *if we agree with config style and verified 
changes to CS is safe*. And we can address pending works in following ticket, 
now the patch is already huge, I'm afraid putting all things together may make 
harder patch review.

> Support hierarchical and unified scheduler configuration
> --------------------------------------------------------
>                 Key: YARN-2986
>                 URL: https://issues.apache.org/jira/browse/YARN-2986
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Wangda Tan
>         Attachments: YARN-2986.1.patch
> Today's scheduler configuration is fragmented and non-intuitive, and needs to 
> be improved. Details in comments.

This message was sent by Atlassian JIRA

Reply via email to