[
https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17247614#comment-17247614
]
Wangda Tan commented on YARN-10504:
-----------------------------------
Thanks [~zhuqi], as we discussed, we're working on the patch in parallel, it
will be great if you can let us know if you have intention to work on a patch
so we can coordinate in the future.
Also, as I shared offline to you, the patch you attached has the following
issues:
1) It changes CS config object, which makes hard for us to identify if it is a
change caused by config file change or internal change.
2) Auto created queue (which is why we add this feature) doesn't load config
from CS config.
I attached patch (see ver.2). It takes the following approach:
1) Major updates are in updateClusterResource, which will be called after
scheduler init and reinit.
2) Introduced new fields to QueueCapacities (weight, and normalized_weight).
Based on normalized_weight, we will calculate absolute resource.
3) Did a bunch of refactorings because code to handle changes of queue
efficient resource and config is everywhere, very hard to troubleshoot and make
further changes. I hope the refactoring in the patch can help code easier
maintained.
Can you please help to review it? [~zhuqi], [~bteke], [~pbacsko], [~shuzirra],
[~snemeth], [~sunilg]
> Implement weight mode in Capacity Scheduler
> -------------------------------------------
>
> Key: YARN-10504
> URL: https://issues.apache.org/jira/browse/YARN-10504
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Benjamin Teke
> Assignee: zhuqi
> Priority: Major
> Attachments: YARN-10504.001.patch, YARN-10504.ver-1.patch,
> YARN-10504.ver-2.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a
> weight mode should be introduced. The existing \{{capacity }}property should
> be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>
> The new functionality should:
> * accept and validate the new weight values
> * enforce a singular mode on the whole queue tree
> * (re)calculate the relative (percentage-based) capacities based on the
> weights during launch and every time the queue structure changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
