[jira] [Comment Edited] (YARN-10564) Support Auto Queue Creation template configurations
[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316378#comment-17316378
]
Andras Gyori edited comment on YARN-10564 at 4/7/21, 2:25 PM:
--
Thank you [~pbacsko] for the suggestions. I have incorporated your ideas and
uploaded a new revision. Realised that we will need the template configurations
on the parents. Also, hopefully I have made the logic simpler and more readable.
was (Author: gandras):
I have uploaded a new revision. I have realised that we will need the template
configurations on the parents. Also, hopefully I have made the logic simpler
and more readable.
> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.001.patch, YARN-10564.002.patch,
> YARN-10564.003.patch, YARN-10564.004.patch, YARN-10564.005.patch,
> YARN-10564.006.patch, YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need
> to support templates for the new auto queue creation logic. Proposition is to
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under
> root.
> We should possibly take an approach, that could support arbitrary depth of
> template configuration, because we might need to lift the limitation of auto
> queue nesting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10564) Support Auto Queue Creation template configurations
[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316277#comment-17316277
]
Peter Bacsko edited comment on YARN-10564 at 4/7/21, 12:16 PM:
---
Thanks [~gandras], I think I get it. I guess the trick is the "for" loop which
modifies "queuePathParts". First we try to find the templates for the parent
explicitly, then we step back a wildcard at each iteration. By changing
"queuePathParts", the prefix changes so eventually we might find a parent which
contains templates.
Finally, we call {{setConfigFromTemplateEntries()}} where we set the collected
values for the original queue.
Is this correct?
was (Author: pbacsko):
Thanks [~gandras], I think I get it. I guess the trick is the "for" loop which
modifies "queuePathParts". First we try to find the templates for the parent
explicitly, then we step back each wildcard at a time. By changing
"queuePathParts", the prefix changes so eventually we might find a parent which
contains templates.
Finally, we call {{setConfigFromTemplateEntries()}} where we set the collected
values for the original queue.
Is this correct?
> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.001.patch, YARN-10564.002.patch,
> YARN-10564.003.patch, YARN-10564.004.patch, YARN-10564.005.patch,
> YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need
> to support templates for the new auto queue creation logic. Proposition is to
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under
> root.
> We should possibly take an approach, that could support arbitrary depth of
> template configuration, because we might need to lift the limitation of auto
> queue nesting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10564) Support Auto Queue Creation template configurations
[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316262#comment-17316262
]
Andras Gyori edited comment on YARN-10564 at 4/7/21, 11:52 AM:
---
[~pbacsko] thank you for your review. I was afraid that the will be a little
bit convoluted. This was much worse in case of legacy auto created queues, but
I think wildcards do complicate things in this case as well. Storing the
children templates on the Parent itself might be more clear, I will try to
investigate that approach. What is happening in this patch:
# The template configuration class sets the properties on the queue itself
# Since we will support configurable depth of queue creation, we need to
support variable length of wildcarding as well (so in case of
QUEUE_CREATION_DEPTH=3, we need to support 2 wildcard levels eg. root.a.*.*,
thats why we have MAX_WILDCARD_LEVEL)
# For a dynamic LeafQueue *root.a.a1,* we start looking for template entries
for its parent *root.a* in the configuration and increasing the wildcard level
by 1 each iteration. An example for the iteration:
## root.a.auto-queue-creation-v2.template.weight
## root.*.auto-queue-creation-v2.template.weight
# After this, we cut the _auto-queue-creation-v2.template_ prefix, and we set
the actual value for the configuration. An example:
## from root.a.auto-queue-creation-v2.template.weight -> root.a.a1.weight
# If we set the root.a.a1.weight explicitly (this would be used for STOPPING /
RUNNING the queue), a template configuration would not overwrite the value. An
example:
## root.a.auto-queue-creation-v2.template.weight = 2
## root.a.a1.weight = 3
## root.a.a1 will have weight = 3
I hope it does clarify the patch to some extent.
was (Author: gandras):
[~pbacsko] thank you for your review. I was afraid that the will be a little
bit convoluted. This was much worse in case of legacy auto created queues.
Storing the children templates on the Parent itself might be more clear, I will
try to investigate that approach. What is happening in this patch:
# The template configuration class sets the properties on the queue itself
# Since we will support configurable depth of queue creation, we need to
support variable length of wildcarding as well (so in case of
QUEUE_CREATION_DEPTH=3, we need to support 2 wildcard levels eg. root.a.*.*,
thats why we have MAX_WILDCARD_LEVEL)
# For a dynamic LeafQueue *root.a.a1,* we start looking for template entries
for its parent *root.a* in the configuration and increasing the wildcard level
by 1 each iteration. An example for the iteration:
## root.a.auto-queue-creation-v2.template.weight
## root.*.auto-queue-creation-v2.template.weight
# After this, we cut the _auto-queue-creation-v2.template_ prefix, and we set
the actual value for the configuration. An example:
## from root.a.auto-queue-creation-v2.template.weight -> root.a.a1.weight
# If we set the root.a.a1.weight explicitly (this would be used for STOPPING /
RUNNING the queue), a template configuration would not overwrite the value. An
example:
## root.a.auto-queue-creation-v2.template.weight = 2
## root.a.a1.weight = 3
## root.a.a1 will have weight = 3
I hope it does clarify the patch to some extent.
> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.001.patch, YARN-10564.002.patch,
> YARN-10564.003.patch, YARN-10564.004.patch, YARN-10564.005.patch,
> YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need
> to support templates for the new auto queue creation logic. Proposition is to
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under
> root.
> We should possibly take an approach, that could support arbitrary depth of
> template configuration, because we might need to lift the limitation of auto
> queue nesting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10564) Support Auto Queue Creation template configurations
[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316213#comment-17316213
]
Peter Bacsko edited comment on YARN-10564 at 4/7/21, 11:51 AM:
---
[~gandras] thanks for the patch.
>From coding POV it looks ok, this is more like a high level review.
There's are some things I just can't figure out (maybe I'm in a bad shape
today).
1. Let's say you set the capacity 6w for {{root.a.*}}. Then a dynamic queue
{{root.a.newparent.newchild}} get created. How does the weight settings
propagate to "newparent" and "newchild"? I kept looking at the code, but it's
just not obvious. I can see that "root.a" will have an entry in
{{templateEntries}}, but then what?
2. I can't deciper this part:
{noformat}
for (int i = 0; i <= wildcardLevel; ++i) {
queuePathParts.set(queuePathParts.size() - 1 - i, WILDCARD_QUEUE);
}
{noformat}
What's happening here?
3. There is a variable called "supportedWildcardLevel". What is "supported"
means in this context? Later on we set it to {{Math.min(queueHierarchyParts -
1, MAX_WILDCARD_LEVEL);}}. It seems to me that it is either 0 or 1, because
{{MAX_WILDCARD_LEVEL}} is 1. I assume most of the time it's going to be 1? I
don't understand what it is meant to represent.
was (Author: pbacsko):
[~gandras] thanks for the patch.
>From coding POV it looks ok, this is more like a high level review.
There's are some things I just can't figure out (maybe I'm in a bad shape
today).
1. Let's say you set the capacity 6w for {{root.a.*}}. Then a dynamic queue
{{root.a.newparent.newchild}} get created. How does the weight settings
propagate to "newparent" and "newchild"? I kept looking at the code, but it's
just not obvious. I can see that "root.a" will have an entry in
{{templateEntries}}, but then what?
2. I can't deciper this part:
{noformat}
for (int i = 0; i <= wildcardLevel; ++i) {
queuePathParts.set(queuePathParts.size() - 1 - i, WILDCARD_QUEUE);
}
{noformat}
What's happening here?
3. There is a variable called "supportedWildcardLevel". What is "supported"
means in this context? Later on we set it to {{Math.min(queueHierarchyParts -
1, MAX_WILDCARD_LEVEL);}}. It seems to me that it is either 0 or 1, because
{{MAX_WILDCARD_LEVEL}} is 1. I assume most of the time it's going to be 1?
Mentally I don't understand what it is meant to represent.
> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.001.patch, YARN-10564.002.patch,
> YARN-10564.003.patch, YARN-10564.004.patch, YARN-10564.005.patch,
> YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need
> to support templates for the new auto queue creation logic. Proposition is to
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under
> root.
> We should possibly take an approach, that could support arbitrary depth of
> template configuration, because we might need to lift the limitation of auto
> queue nesting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10564) Support Auto Queue Creation template configurations
[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316213#comment-17316213
]
Peter Bacsko edited comment on YARN-10564 at 4/7/21, 10:49 AM:
---
[~gandras] thanks for the patch.
>From coding POV it looks ok, this is more like a high level review.
There's are some things I just can't figure out (maybe I'm in a bad shape
today).
1. Let's say you set the capacity 6w for {{root.a.*}}. Then a dynamic queue
{{root.a.newparent.newchild}} get created. How does the weight settings
propagate to "newparent" and "newchild"? I kept looking at the code, but it's
just not obvious. I can see that "root.a" will have an entry in
{{templateEntries}}, but then what?
2. I can't deciper this part:
{noformat}
for (int i = 0; i <= wildcardLevel; ++i) {
queuePathParts.set(queuePathParts.size() - 1 - i, WILDCARD_QUEUE);
}
{noformat}
What's happening here?
3. There is a variable called "supportedWildcardLevel". What is "supported"
means in this context? Later on we set it to {{Math.min(queueHierarchyParts -
1, MAX_WILDCARD_LEVEL);}}. It seems to me that it is either 0 or 1, because
{{MAX_WILDCARD_LEVEL}} is 1. I assume most of the time it's going to be 1?
Mentally I don't understand what it is meant to represent.
was (Author: pbacsko):
[~gandras] thanks for the patch.
>From coding POV it looks ok, this is more like a high level review.
There's are some things I just can't figure out (maybe I'm in a bad shape
today).
1. Let's say you set 6w for {{root.a.*}}. Then a dynamic queue
{{root.a.newparent.newchild}} get created. How does the weight settings
propagate to "newparent" and "newchild"? I kept looking at the code, but it's
just not obvious. I can see that "root.a" will have an entry in
{{templateEntries}}, but then what?
2. I can't deciper this part:
{noformat}
for (int i = 0; i <= wildcardLevel; ++i) {
queuePathParts.set(queuePathParts.size() - 1 - i, WILDCARD_QUEUE);
}
{noformat}
What's happening here?
3. There is a variable called "supportedWildcardLevel". What is "supported"
means in this context? Later on we set it to {{Math.min(queueHierarchyParts -
1, MAX_WILDCARD_LEVEL);}} which seems to be that it is either 0 or 1, because
{{MAX_WILDCARD_LEVEL}} is 1. I assume most of the time it's going to be 1?
Mentally I don't understand what it is meant to represent.
> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.001.patch, YARN-10564.002.patch,
> YARN-10564.003.patch, YARN-10564.004.patch, YARN-10564.005.patch,
> YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need
> to support templates for the new auto queue creation logic. Proposition is to
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under
> root.
> We should possibly take an approach, that could support arbitrary depth of
> template configuration, because we might need to lift the limitation of auto
> queue nesting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10564) Support Auto Queue Creation template configurations
[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269289#comment-17269289
]
Andras Gyori edited comment on YARN-10564 at 1/21/21, 1:15 PM:
---
Uploaded a patch, that supports template configuration in the following way:
* A template config for a parent could be added as:
** {{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.capacity}}
** {{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.capacity}}
**
{{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.\*.\*.capacity}}
* Wildcarded path templates have lower precedence, than more explicit paths:
*
** {{_root.a.b.capacity > root.a.\*.capacity > root.\*.\*.capacity_}}
* Remaining questions:
*
** Do we want to support more flexible wildcard patterns, like _root.a.\*.b or
root.\*.a.\*.b_ (be able to wildcard anywhere in the path)?
** Do we want to restrict which config keys are templatable? eg. have a
specific set of keys that is supported?
was (Author: gandras):
Uploaded a patch, that supports template configuration in the following way:
* A template config for a parent could be added as:
** {{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.capacity}}
** {{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.capacity}}
**
{{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.*.capacity}}
* Wildcarded path templates have lower precedence, than more explicit paths:
*
** {{_root.a.b.capacity > root.a.*.capacity > root.*.*.capacity_}}
* Remaining questions:
*
** Do we want to support more flexible wildcard patterns, like _root.a.*.b or
root.*.a.*.b_ (be able to wildcard anywhere in the path)?
** Do we want to restrict which config keys are templatable? eg. have a
specific set of keys that is supported?
> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.001.patch, YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need
> to support templates for the new auto queue creation logic. Proposition is to
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under
> root.
> We should possibly take an approach, that could support arbitrary depth of
> template configuration, because we might need to lift the limitation of auto
> queue nesting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10564) Support Auto Queue Creation template configurations
[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269289#comment-17269289
]
Andras Gyori edited comment on YARN-10564 at 1/21/21, 1:14 PM:
---
Uploaded a patch, that supports template configuration in the following way:
* A template config for a parent could be added as:
** {{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.capacity}}
** {{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.capacity}}
**
{{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.*.capacity}}
* Wildcarded path templates have lower precedence, than more explicit paths:
*
** {{_root.a.b.capacity > root.a.*.capacity > root.*.*.capacity_}}
* Remaining questions:
*
** Do we want to support more flexible wildcard patterns, like _root.a.*.b or
root.*.a.*.b_ (be able to wildcard anywhere in the path)?
** Do we want to restrict which config keys are templatable? eg. have a
specific set of keys that is supported?
was (Author: gandras):
Uploaded a patch, that supports template configuration in the following way:
* A template config for a parent could be added as:
** _yarn.scheduler.capacity.auto-queue-creation-v2.template.root.capacity_
** _yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.capacity_
** _yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.*.capacity_
* Wildcarded path templates have lower precedence, than more explicit paths:
** _root.a.b.capacity > root.a.*.capacity > root.*.*.capacity_
* Remaining questions:
** Do we want to support more flexible wildcard patterns, like root.a.*.b or
root.*.a.*.b (be able to wildcard anywhere in the path)?
** Do we want to restrict which config keys are templatable? eg. have a
specific set of keys that is supported?
> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.001.patch, YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need
> to support templates for the new auto queue creation logic. Proposition is to
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under
> root.
> We should possibly take an approach, that could support arbitrary depth of
> template configuration, because we might need to lift the limitation of auto
> queue nesting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10564) Support Auto Queue Creation template configurations
[
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17263426#comment-17263426
]
zhuqi edited comment on YARN-10564 at 1/12/21, 3:32 PM:
[~leftnoteasy] [~gandras]
Add an auto create with template poc patch, combined with YARN-10506-008.patch
for test.
The poc just handle template related conf, next we should reuse or extend
AutoCreatedQueueManagementPolicy and realize the related logic.
The test is in :
testAutoCreateQueueWithTemplateWeight function.
Thanks.
was (Author: zhuqi):
[~leftnoteasy] [~gandras]
Add an auto create with template poc patch, combined with YARN-10506-008.patch
for test.
The poc just handle template related conf, next we should reuse or extend
AutoCreatedQueueManagementPolicy and realize the related logic.
> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need
> to support templates for the new auto queue creation logic. Proposition is to
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under
> root.
> We should possibly take an approach, that could support arbitrary depth of
> template configuration, because we might need to lift the limitation of auto
> queue nesting.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
