[jira] [Commented] (YARN-2986) (Umbrella) Support hierarchical and unified scheduler configuration
[ https://issues.apache.org/jira/browse/YARN-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14337372#comment-14337372 ] Wangda Tan commented on YARN-2986: -- Hi [~jianhe], I think it should only take effect when user puts it to correct place, we can either ignore it or throw exception when user puts it to a incorrect tag. Adding policy-properties is a double-edged sword, as you said, user has to learn which configuration should be policy-properties. But in the other hand, if we support different policy for different queues in the future (e.g. some queues are fair-based scheduling, some queues are fifo-based scheduling, some queues are priority based scheduling). Admin can get clear understanding, when policy need changed/updated, which fields need to be added/removed/updated. I more prefer adding the policy-properties. Thanks, (Umbrella) Support hierarchical and unified scheduler configuration --- Key: YARN-2986 URL: https://issues.apache.org/jira/browse/YARN-2986 Project: Hadoop YARN Issue Type: Improvement Reporter: Vinod Kumar Vavilapalli Assignee: Wangda Tan Attachments: YARN-2986.1.patch Today's scheduler configuration is fragmented and non-intuitive, and needs to be improved. Details in comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2986) (Umbrella) Support hierarchical and unified scheduler configuration
[ https://issues.apache.org/jira/browse/YARN-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14333728#comment-14333728 ] Wangda Tan commented on YARN-2986: -- Hi [~sunilg], Thanks for sharing your thoughts :). bq. Do you mean that, non repeating items are kept outside loop of queues and changing items are kept inside each queue? No, what I meant is, if a field *need* to be set in queue level, policy-properties should be a part of queue, and also global is some fields need to be global. For your 2nd point, providing a way to make policy-properties linkable. However, I think for most of the fields, like capacity, label settings, etc. should be different for different queues, adding links might make people get confused. And queues have to overwrite some fields to make their own. Alternatively, I think we can use inherit to make configuration easier, for example, a parent queue's accessible-node-labels will be inherited by its children if its children don't overwrite. Thoughts? (Umbrella) Support hierarchical and unified scheduler configuration --- Key: YARN-2986 URL: https://issues.apache.org/jira/browse/YARN-2986 Project: Hadoop YARN Issue Type: Improvement Reporter: Vinod Kumar Vavilapalli Assignee: Wangda Tan Attachments: YARN-2986.1.patch Today's scheduler configuration is fragmented and non-intuitive, and needs to be improved. Details in comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2986) (Umbrella) Support hierarchical and unified scheduler configuration
[ https://issues.apache.org/jira/browse/YARN-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14334079#comment-14334079 ] Jian He commented on YARN-2986: --- Regarding the policy-properties take , I have one doubt that what if user specify a policy-property outside of the policy-properties tag and non-policy-property inside policy-properties, will we throw exception or ignore the settings? Ignoring seems confusing to users, as user may have no idea which properties are set correctly and which properties are not. Either way forces users to learn precisely which properties should be inside the policy-properties tags and which ones should not. Each time a new property is added, users also need to be learned. That's non-trivial effort to users. (Umbrella) Support hierarchical and unified scheduler configuration --- Key: YARN-2986 URL: https://issues.apache.org/jira/browse/YARN-2986 Project: Hadoop YARN Issue Type: Improvement Reporter: Vinod Kumar Vavilapalli Assignee: Wangda Tan Attachments: YARN-2986.1.patch Today's scheduler configuration is fragmented and non-intuitive, and needs to be improved. Details in comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2986) (Umbrella) Support hierarchical and unified scheduler configuration
[
https://issues.apache.org/jira/browse/YARN-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329949#comment-14329949
]
Sunil G commented on YARN-2986:
---
Thank you [~leftnoteasy], this is much awaited ticket :)
I have few inputs on same.
1.
{noformat}
policy-properties
resource-calculator
org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
/resource-calculator
/policy-properties
{noformat}
and
{noformat}
policy-properties
user-limit-factor2/user-limit-factor
{noformat}
This is inside queue.
Do you mean that, non repeating items are kept outside loop of queues and
changing items are kept inside each queue?
Hoewever if i have only one set userlimit, node labels etc, and if i keep all
of those policy-properties outside *queue* section, then will it be applicable
for all queues?
If not, I suggest we can have policy-property name concept.
{noformat}
queue name=default
stateRUNNING/state
acl_submit_applications*/acl_submit_applications
acl_administer_queue*/acl_administer_queue
accessible-node-labelsx/accessible-node-labels
policy-propertiesgpu/policy-properties
/queue
queue name=queueA
stateRUNNING/state
acl_submit_applications*/acl_submit_applications
acl_administer_queue*/acl_administer_queue
accessible-node-labelsx/accessible-node-labels
policy-propertiesgpu/policy-properties
/queue
policy-properties name=gpu
user-limit-factor2/user-limit-factor
node-labels
node-label name=x
capacity20/capacity
maximum-capacity50/maximum-capacity
/node-label
/node-labels
{noformat}
It cab be shared as needed across queues. And will make the queue part more
readable.
(Umbrella) Support hierarchical and unified scheduler configuration
---
Key: YARN-2986
URL: https://issues.apache.org/jira/browse/YARN-2986
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Vinod Kumar Vavilapalli
Assignee: Wangda Tan
Attachments: YARN-2986.1.patch
Today's scheduler configuration is fragmented and non-intuitive, and needs to
be improved. Details in comments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2986) (Umbrella) Support hierarchical and unified scheduler configuration
[
https://issues.apache.org/jira/browse/YARN-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329917#comment-14329917
]
Wangda Tan commented on YARN-2986:
--
An update:
After an offline discussion with [~vinodkv] and [~jianhe], now proposed
configuration file looks like:
{code}
scheduler
typecapacity/type
maximum-applications/maximum-applications
queue-mappings/queue-mappings
queue-mappings-override-enable/queue-mappings-override-enable
maximum-am-resource-percent0.3/maximum-am-resource-percent
policy-properties
resource-calculator
org.apache.hadoop.yarn.util.resource.DominantResourceCalculator
/resource-calculator
/policy-properties
queue name=root
queues
queue name=default
stateRUNNING/state
acl_submit_applications*/acl_submit_applications
acl_administer_queue*/acl_administer_queue
accessible-node-labelsx/accessible-node-labels
policy-properties
user-limit-factor2/user-limit-factor
capacity50/capacity
maximum-capacity90/maximum-capacity
node-locality-delay30/node-locality-delay
node-labels
node-label name=x
capacity20/capacity
maximum-capacity50/maximum-capacity
/node-label
/node-labels
/policy-properties
/queue
/queues
/queue
/scheduler
{code}
One highlight of this proposal and previous proposal is: this contains a
policy-properties for each configuration node, which means a
scheduler-specific configurations, like capacity in CapacityScheduler and
minShare in FairScheduler, etc. (policy here means different kinds of
scheduling method).
For other common options (not belongs to a specific scheduler implementation),
should be placed outside of policy-properties.
*Please feel free to share your thoughts about this proposal :).*
To move this forward, I filed several sub ticket, YARN-3233 is targeted to
solve the configuration file (for common scheduler and capacity scheduler)
definition and parsing, I will upload a patch right now. YARN-3234 is to solve
Capacity Scheduler integration with the new config file.
(Umbrella) Support hierarchical and unified scheduler configuration
---
Key: YARN-2986
URL: https://issues.apache.org/jira/browse/YARN-2986
Project: Hadoop YARN
Issue Type: Improvement
Reporter: Vinod Kumar Vavilapalli
Assignee: Wangda Tan
Attachments: YARN-2986.1.patch
Today's scheduler configuration is fragmented and non-intuitive, and needs to
be improved. Details in comments.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2986) (Umbrella) Support hierarchical and unified scheduler configuration
[ https://issues.apache.org/jira/browse/YARN-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14329921#comment-14329921 ] Wangda Tan commented on YARN-2986: -- Ver.1 patch uploaded to YARN-3233 (Umbrella) Support hierarchical and unified scheduler configuration --- Key: YARN-2986 URL: https://issues.apache.org/jira/browse/YARN-2986 Project: Hadoop YARN Issue Type: Improvement Reporter: Vinod Kumar Vavilapalli Assignee: Wangda Tan Attachments: YARN-2986.1.patch Today's scheduler configuration is fragmented and non-intuitive, and needs to be improved. Details in comments. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
