[
https://issues.apache.org/jira/browse/YARN-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wangda Tan updated YARN-2986:
-----------------------------
Attachment: YARN-2986.1.patch
Uploaded a WIP patch, which consists:
1) Implementation of hierarchy CS configuration:
- Basic xml parser utilities
- Base implementation of scheduler hierarchy configuration (can by reused by
Fair and other schedulers)
- Capacity Scheduler configuration implementation (*)
(*) Capacity Scheduler configuration implementation:
- Since we need to maintain two different configuration style (plain &
hierarchy), so we need an abstraction layer to make Capacity Scheduler less
impacted. So now:
- CapacitySchedulerConfiguration becomes an interface, with all getter/setter
needed
- The original CapacitySchedulerConfiguration is renamed CSPlainConfiguration,
implements CapacitySchedulerConfiguration
- The new hierarchy configuration is named CSHierarchyConfiguration
2) Changes in Capacity Scheduler to use the new interface --
CapacitySchedulerConfiguration
- This is half way done, now calling CapacitySchedulerContext.getConfiguration
will return the interface class.
- Other part of CapacityScheduler class is still using CSPlainConfiguration
For 2), pending works:
- Add smart detectation code choose to read configuration file
- Eleminate directly using CSPlainConfiguration, this is blocked by an issue in
existing CapacityScheduler implementation:
In existing CS implementation, CapacitySchedulerConfiguration will
automatically includes "yarn-site.xml", which is not correct, we need limit
CapacitySchedulerConfiguration only use "capacity-scheduler.xml".
- Add more tests to make sure CS will work under both configuration file style.
Now config file style
{code}
<scheduler>
<globals>
<global-var1>value</global-var1>
<global-var2>value</global-var2>
<global-var3>value</global-var3>
</globals>
<queues>
<queue name="a">
<queue-var1>value</queue-var1>
<queue-var2>value</queue-var2>
<queues>
<queue name="a1">
</queue>
<queue name="a2">
</queue>
</queue>
</queues>
</scheduler>
{code}
Suggestion of moving forward:
I suggest to commit existing patch *if we agree with config style and verified
changes to CS is safe*. And we can address pending works in following ticket,
now the patch is already huge, I'm afraid putting all things together may make
harder patch review.
> Support hierarchical and unified scheduler configuration
> --------------------------------------------------------
>
> Key: YARN-2986
> URL: https://issues.apache.org/jira/browse/YARN-2986
> Project: Hadoop YARN
> Issue Type: Improvement
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Wangda Tan
> Attachments: YARN-2986.1.patch
>
>
> Today's scheduler configuration is fragmented and non-intuitive, and needs to
> be improved. Details in comments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
