[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17316262#comment-17316262
]
Andras Gyori edited comment on YARN-10564 at 4/7/21, 11:52 AM:
---------------------------------------------------------------
[~pbacsko] thank you for your review. I was afraid that the will be a little
bit convoluted. This was much worse in case of legacy auto created queues, but
I think wildcards do complicate things in this case as well. Storing the
children templates on the Parent itself might be more clear, I will try to
investigate that approach. What is happening in this patch:
# The template configuration class sets the properties on the queue itself
# Since we will support configurable depth of queue creation, we need to
support variable length of wildcarding as well (so in case of
QUEUE_CREATION_DEPTH=3, we need to support 2 wildcard levels eg. root.a.*.*,
thats why we have MAX_WILDCARD_LEVEL)
# For a dynamic LeafQueue *root.a.a1,* we start looking for template entries
for its parent *root.a* in the configuration and increasing the wildcard level
by 1 each iteration. An example for the iteration:
## root.a.auto-queue-creation-v2.template.weight
## root.*.auto-queue-creation-v2.template.weight
# After this, we cut the _auto-queue-creation-v2.template_ prefix, and we set
the actual value for the configuration. An example:
## from root.a.auto-queue-creation-v2.template.weight -> root.a.a1.weight
# If we set the root.a.a1.weight explicitly (this would be used for STOPPING /
RUNNING the queue), a template configuration would not overwrite the value. An
example:
## root.a.auto-queue-creation-v2.template.weight = 2
## root.a.a1.weight = 3
## root.a.a1 will have weight = 3
I hope it does clarify the patch to some extent.
was (Author: gandras):
[~pbacsko] thank you for your review. I was afraid that the will be a little
bit convoluted. This was much worse in case of legacy auto created queues.
Storing the children templates on the Parent itself might be more clear, I will
try to investigate that approach. What is happening in this patch:
# The template configuration class sets the properties on the queue itself
# Since we will support configurable depth of queue creation, we need to
support variable length of wildcarding as well (so in case of
QUEUE_CREATION_DEPTH=3, we need to support 2 wildcard levels eg. root.a.*.*,
thats why we have MAX_WILDCARD_LEVEL)
# For a dynamic LeafQueue *root.a.a1,* we start looking for template entries
for its parent *root.a* in the configuration and increasing the wildcard level
by 1 each iteration. An example for the iteration:
## root.a.auto-queue-creation-v2.template.weight
## root.*.auto-queue-creation-v2.template.weight
# After this, we cut the _auto-queue-creation-v2.template_ prefix, and we set
the actual value for the configuration. An example:
## from root.a.auto-queue-creation-v2.template.weight -> root.a.a1.weight
# If we set the root.a.a1.weight explicitly (this would be used for STOPPING /
RUNNING the queue), a template configuration would not overwrite the value. An
example:
## root.a.auto-queue-creation-v2.template.weight = 2
## root.a.a1.weight = 3
## root.a.a1 will have weight = 3
I hope it does clarify the patch to some extent.
> Support Auto Queue Creation template configurations
> ---------------------------------------------------
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Andras Gyori
> Assignee: Andras Gyori
> Priority: Major
> Attachments: YARN-10564.001.patch, YARN-10564.002.patch,
> YARN-10564.003.patch, YARN-10564.004.patch, YARN-10564.005.patch,
> YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need
> to support templates for the new auto queue creation logic. Proposition is to
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under
> root.
> We should possibly take an approach, that could support arbitrary depth of
> template configuration, because we might need to lift the limitation of auto
> queue nesting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
