[jira] [Comment Edited] (YARN-10564) Support Auto Queue Creation template configurations

2021-01-21 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269289#comment-17269289
 ] 

Andras Gyori edited comment on YARN-10564 at 1/21/21, 1:15 PM:
---

Uploaded a patch, that supports template configuration in the following way:
 * A template config for a parent could be added as:
 ** {{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.capacity}}
 ** {{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.capacity}}
 ** 
{{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.\*.\*.capacity}}
 * Wildcarded path templates have lower precedence, than more explicit paths:

 * 
 ** {{_root.a.b.capacity > root.a.\*.capacity > root.\*.\*.capacity_}}
 * Remaining questions:

 * 
 ** Do we want to support more flexible wildcard patterns, like _root.a.\*.b or 
root.\*.a.\*.b_ (be able to wildcard anywhere in the path)?
 ** Do we want to restrict which config keys are templatable? eg. have a 
specific set of keys that is supported?


was (Author: gandras):
Uploaded a patch, that supports template configuration in the following way:
 * A template config for a parent could be added as:
 ** {{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.capacity}}
 ** {{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.capacity}}
 ** 
{{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.*.capacity}}
 * Wildcarded path templates have lower precedence, than more explicit paths:

 * 
 ** {{_root.a.b.capacity > root.a.*.capacity > root.*.*.capacity_}}
 * Remaining questions:

 * 
 ** Do we want to support more flexible wildcard patterns, like _root.a.*.b or 
root.*.a.*.b_ (be able to wildcard anywhere in the path)?
 ** Do we want to restrict which config keys are templatable? eg. have a 
specific set of keys that is supported?

> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.001.patch, YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need 
> to support templates for the new auto queue creation logic. Proposition is to 
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under 
> root.
> We should possibly take an approach, that could support arbitrary depth of 
> template configuration, because we might need to lift the limitation of auto 
> queue nesting.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Comment Edited] (YARN-10564) Support Auto Queue Creation template configurations

2021-01-21 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269289#comment-17269289
 ] 

Andras Gyori edited comment on YARN-10564 at 1/21/21, 1:14 PM:
---

Uploaded a patch, that supports template configuration in the following way:
 * A template config for a parent could be added as:
 ** {{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.capacity}}
 ** {{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.capacity}}
 ** 
{{yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.*.capacity}}
 * Wildcarded path templates have lower precedence, than more explicit paths:

 * 
 ** {{_root.a.b.capacity > root.a.*.capacity > root.*.*.capacity_}}
 * Remaining questions:

 * 
 ** Do we want to support more flexible wildcard patterns, like _root.a.*.b or 
root.*.a.*.b_ (be able to wildcard anywhere in the path)?
 ** Do we want to restrict which config keys are templatable? eg. have a 
specific set of keys that is supported?


was (Author: gandras):
Uploaded a patch, that supports template configuration in the following way:
 * A template config for a parent could be added as:
 ** _yarn.scheduler.capacity.auto-queue-creation-v2.template.root.capacity_
 ** _yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.capacity_
 ** _yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.*.capacity_
 * Wildcarded path templates have lower precedence, than more explicit paths:

 ** _root.a.b.capacity > root.a.*.capacity > root.*.*.capacity_
 * Remaining questions:

 ** Do we want to support more flexible wildcard patterns, like root.a.*.b or 
root.*.a.*.b (be able to wildcard anywhere in the path)?
 ** Do we want to restrict which config keys are templatable? eg. have a 
specific set of keys that is supported?

> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.001.patch, YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need 
> to support templates for the new auto queue creation logic. Proposition is to 
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under 
> root.
> We should possibly take an approach, that could support arbitrary depth of 
> template configuration, because we might need to lift the limitation of auto 
> queue nesting.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10564) Support Auto Queue Creation template configurations

2021-01-21 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269289#comment-17269289
 ] 

Andras Gyori commented on YARN-10564:
-

Uploaded a patch, that supports template configuration in the following way:
 * A template config for a parent could be added as:
 ** _yarn.scheduler.capacity.auto-queue-creation-v2.template.root.capacity_
 ** _yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.capacity_
 ** _yarn.scheduler.capacity.auto-queue-creation-v2.template.root.*.*.capacity_
 * Wildcarded path templates have lower precedence, than more explicit paths:

 ** _root.a.b.capacity > root.a.*.capacity > root.*.*.capacity_
 * Remaining questions:

 ** Do we want to support more flexible wildcard patterns, like root.a.*.b or 
root.*.a.*.b (be able to wildcard anywhere in the path)?
 ** Do we want to restrict which config keys are templatable? eg. have a 
specific set of keys that is supported?

> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.001.patch, YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need 
> to support templates for the new auto queue creation logic. Proposition is to 
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under 
> root.
> We should possibly take an approach, that could support arbitrary depth of 
> template configuration, because we might need to lift the limitation of auto 
> queue nesting.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10564) Support Auto Queue Creation template configurations

2021-01-21 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10564:

Attachment: YARN-10564.001.patch

> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.001.patch, YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need 
> to support templates for the new auto queue creation logic. Proposition is to 
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under 
> root.
> We should possibly take an approach, that could support arbitrary depth of 
> template configuration, because we might need to lift the limitation of auto 
> queue nesting.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10581) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include queue creation type for queues

2021-01-21 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17269169#comment-17269169
 ] 

Andras Gyori commented on YARN-10581:
-

Thanks [~snemeth] for the patch, I have managed to apply it on latest trunk, 
after YARN-10579 was merged in. I have some additional comment, none of them is 
critical:
 * We have 4 types of queues, but an AutoCreatedLeafQueue will be considered as 
a staticLeaf. I think this is a bit misleading, we should come up with a 
solution (maybe in a followup jira)
 * The queue type strings are repeated in the test. I think you could make them 
public and use them in CapacitySchedulerInfoHelper.
 * In TestRMWebServicesCapacitySchedDynamicConfig, csConf is set, but never 
used.
 * Really minor nit: could not find a auto queue under default, but you set the 
root.default.auto-queue-creation-v2.enabled to true

> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include 
> queue creation type for queues
> 
>
> Key: YARN-10581
> URL: https://issues.apache.org/jira/browse/YARN-10581
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-10581.001.patch, YARN-10581.002.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> Auto-queue creation has been also imlemented with YARN-10506.
> Connected to this effort, we would like to expose the type of the queue with 
> the RM's /scheduler REST endpoint.
> The queue type should hold these values: 
>  * Auto-created parent queue: *autoCreatedParent*
>  * Auto-created leaf queue: *autoCreatedLeaf*
>  * Static parent: *staticParent*
>  * Static leaf: *staticLeaf* 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10579) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include weight values for queues

2021-01-20 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10579?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17268638#comment-17268638
 ] 

Andras Gyori commented on YARN-10579:
-

I have no comments regarding this, +1 non-binding.

> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include 
> weight values for queues
> --
>
> Key: YARN-10579
> URL: https://issues.apache.org/jira/browse/YARN-10579
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-10579.001.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
>  We would like to expose the weight values for all queues with the RM's 
> /scheduler REST endpoint.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10578) Fix Auto Queue Creation parent handling

2021-01-19 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267879#comment-17267879
 ] 

Andras Gyori commented on YARN-10578:
-

Thank you [~shuzirra], it is a valid point! However, I think this lock is there 
to make sure the CS is intact while creating the queue, so that no additional 
queue handling is made during the period of auto queue creation. The 
CSAutoQueueCreationHandler itself does not contain any state apart from the 
queueManager, which is thread safe internally. 

> Fix Auto Queue Creation parent handling
> ---
>
> Key: YARN-10578
> URL: https://issues.apache.org/jira/browse/YARN-10578
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10578.001.patch, YARN-10578.002.patch
>
>
> YARN-10506 introduced the new auto queue creation logic, however a parent == 
> null check in CapacityScheduler#autoCreateLeafQueue will prevent a two levels 
> queue to be created. We need to revert it back to the normal logic, also, we 
> should wrap the auto queue handling with a lock.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10578) Fix Auto Queue Creation parent handling

2021-01-18 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267703#comment-17267703
 ] 

Andras Gyori commented on YARN-10578:
-

Thanks [~wangda] and [~zhuqi] for the review. lastSubmittedTimestamp will be 
added in the queue deletion issue, I agree, but in a different jira. Rebased on 
top of trunk and uploaded a new patch.

> Fix Auto Queue Creation parent handling
> ---
>
> Key: YARN-10578
> URL: https://issues.apache.org/jira/browse/YARN-10578
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10578.001.patch, YARN-10578.002.patch
>
>
> YARN-10506 introduced the new auto queue creation logic, however a parent == 
> null check in CapacityScheduler#autoCreateLeafQueue will prevent a two levels 
> queue to be created. We need to revert it back to the normal logic, also, we 
> should wrap the auto queue handling with a lock.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10578) Fix Auto Queue Creation parent handling

2021-01-18 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10578:

Attachment: YARN-10578.002.patch

> Fix Auto Queue Creation parent handling
> ---
>
> Key: YARN-10578
> URL: https://issues.apache.org/jira/browse/YARN-10578
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10578.001.patch, YARN-10578.002.patch
>
>
> YARN-10506 introduced the new auto queue creation logic, however a parent == 
> null check in CapacityScheduler#autoCreateLeafQueue will prevent a two levels 
> queue to be created. We need to revert it back to the normal logic, also, we 
> should wrap the auto queue handling with a lock.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10578) Fix Auto Queue Creation parent handling

2021-01-18 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10578:

Attachment: YARN-10578.001.patch

> Fix Auto Queue Creation parent handling
> ---
>
> Key: YARN-10578
> URL: https://issues.apache.org/jira/browse/YARN-10578
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10578.001.patch
>
>
> YARN-10506 introduced the new auto queue creation logic, however a parent == 
> null check in CapacityScheduler#autoCreateLeafQueue will prevent a two levels 
> queue to be created. We need to revert it back to the normal logic, also, we 
> should wrap the auto queue handling with a lock.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10578) Fix Auto Queue Creation parent handling

2021-01-18 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10578:

Parent: YARN-10496
Issue Type: Sub-task  (was: Bug)

> Fix Auto Queue Creation parent handling
> ---
>
> Key: YARN-10578
> URL: https://issues.apache.org/jira/browse/YARN-10578
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
>
> YARN-10506 introduced the new auto queue creation logic, however a parent == 
> null check in CapacityScheduler#autoCreateLeafQueue will prevent a two levels 
> queue to be created. We need to revert it back to the normal logic, also, we 
> should wrap the auto queue handling with a lock.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Created] (YARN-10578) Fix Auto Queue Creation parent handling

2021-01-18 Thread Andras Gyori (Jira)
Andras Gyori created YARN-10578:
---

 Summary: Fix Auto Queue Creation parent handling
 Key: YARN-10578
 URL: https://issues.apache.org/jira/browse/YARN-10578
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacity scheduler
Reporter: Andras Gyori
Assignee: Andras Gyori


YARN-10506 introduced the new auto queue creation logic, however a parent == 
null check in CapacityScheduler#autoCreateLeafQueue will prevent a two levels 
queue to be created. We need to revert it back to the normal logic, also, we 
should wrap the auto queue handling with a lock.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement

2021-01-18 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267076#comment-17267076
 ] 

Andras Gyori commented on YARN-10535:
-

I agree with [~snemeth]'s detailed feedback, however I see [~shuzirra]'s point 
as well. In order not to achieve the same confusion as it is now, we should 
only start the refactoring in one patch, and it is a good idea not to try to 
come up with anything new in this patch. My idea would be to introduce a 
QueuePath class, that could handle the most use cases listed here, and 
elsewhere, regarding general queue path operations (like extracting parent 
path, iterating over the path etc..)

> Make queue placement in CapacityScheduler compliant with auto-queue-placement
> -
>
> Key: YARN-10535
> URL: https://issues.apache.org/jira/browse/YARN-10535
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Wangda Tan
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10535.001.patch, YARN-10535.002.patch, 
> YARN-10535.003.patch, YARN-10535.004.patch, YARN-10535.005.patch
>
>
> Once YARN-10506 is done, we need to call the API from the queue placement 
> policy to create queues. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS

2021-01-18 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267072#comment-17267072
 ] 

Andras Gyori commented on YARN-10512:
-

Thank you [~snemeth] for the patch. I have taken an other look at it:
 * Left some unused import in TestRMWebServices, but I presume you will clear 
the checkstyle errors anyways, so it is just an FYI.

Apart from these minor suggestions, I have no additional comment,

> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include 
> mode of operation for CS
> --
>
> Key: YARN-10512
> URL: https://issues.apache.org/jira/browse/YARN-10512
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-10512.001.patch, YARN-10512.002.patch, 
> YARN-10512.003.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> We would like to expose the mode of operation with the RM's /scheduler REST 
> endpoint.
> The field name will be 'mode'.
> All queue representations in the response will be uniformly hold any of the 
> mode values of: "percentage", "absolute", "weight".



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10428) Zombie applications in the YARN queue using FAIR + sizebasedweight

2021-01-17 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17267048#comment-17267048
 ] 

Andras Gyori commented on YARN-10428:
-

Hey [~yguang11], thanks for the patch. We have encountered the same issue. Do 
you know any easy way to reproduce in a deterministic way? I would like to 
explore the possibilities and also understand the root cause.

> Zombie applications in the YARN queue using FAIR + sizebasedweight
> --
>
> Key: YARN-10428
> URL: https://issues.apache.org/jira/browse/YARN-10428
> Project: Hadoop YARN
>  Issue Type: Bug
>  Components: capacityscheduler
>Affects Versions: 2.8.5
>Reporter: Guang Yang
>Priority: Major
> Attachments: YARN-10428.001.patch, YARN-10428.002.patch
>
>
> Seeing zombie jobs in the YARN queue that uses FAIR and size based weight 
> ordering policy .
> *Detection:*
> The YARN UI shows incorrect number of "Num Schedulable Applications".
> *Impact:*
> The queue has an upper limit of number of running applications, with zombie 
> job, it hits the limit even though the number of running applications is far 
> less than the limit. 
> *Workaround:*
> **Fail-over and restart Resource Manager process.
> *Analysis:*
> **In the heap dump, we can find the zombie jobs in the `FairOderingPolicy#
> schedulableEntities` (see attachment). Take application 
> "application_1599157165858_29429" for example, it is still in the  
> `FairOderingPolicy#schedulableEntities` set, however, if we check the log of 
> resource manager, we can see RM already tried to remove the application:
>  
> ./yarn-yarn-resourcemanager-ip-172-21-153-252.log.2020-09-04-04:2020-09-04 
> 04:32:19,730 INFO 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue 
> (ResourceManager Event Processor): Application removed - appId: 
> application_1599157165858_29429 user: svc_di_data_eng queue: core-data 
> #user-pending-applications: -3 #user-active-applications: 7 
> #queue-pending-applications: 0 #queue-active-applications: 21
>  
> So it appears RM failed to removed the application from the set.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation

2021-01-15 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17266181#comment-17266181
 ] 

Andras Gyori commented on YARN-10506:
-

Thanks [~zhuqi], you are right, but I realized the whole logic was flawed 
because of the parent == null check. I have removed this check and added your 
suggestion, also, fixed the corresponding test.

> Update queue creation logic to use weight mode and allow the flexible 
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch, 
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, 
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, 
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, 
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, 
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, 
> YARN-10506.014.patch, YARN-10506.015.patch, YARN-10506.016.patch
>
>
> The queue creation logic should be updated to use weight mode and support the 
> flexible creation. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation

2021-01-15 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10506:

Attachment: YARN-10506.016.patch

> Update queue creation logic to use weight mode and allow the flexible 
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch, 
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, 
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, 
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, 
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, 
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, 
> YARN-10506.014.patch, YARN-10506.015.patch, YARN-10506.016.patch
>
>
> The queue creation logic should be updated to use weight mode and support the 
> flexible creation. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation

2021-01-15 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17266096#comment-17266096
 ] 

Andras Gyori commented on YARN-10506:
-

Thank you [~shuzirra] for the feedback, I too find out troubling and fixed this 
concern, and also found an additional line where we used short name instead of 
full path. As for the rest, I will handle them in the followup refactoring 
jira, but in general I agree with each of them.
Also thank you [~bteke] for the comments, I have removed all the unused 
variables. [~zhuqi] will reintroduce the fields that are needed in an other 
jira.

> Update queue creation logic to use weight mode and allow the flexible 
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch, 
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, 
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, 
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, 
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, 
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, 
> YARN-10506.014.patch, YARN-10506.015.patch
>
>
> The queue creation logic should be updated to use weight mode and support the 
> flexible creation. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation

2021-01-15 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10506:

Attachment: YARN-10506.015.patch

> Update queue creation logic to use weight mode and allow the flexible 
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch, 
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, 
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, 
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, 
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, 
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, 
> YARN-10506.014.patch, YARN-10506.015.patch
>
>
> The queue creation logic should be updated to use weight mode and support the 
> flexible creation. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10513) CS Flexible Auto Queue Creation RM UIv2 modifications

2021-01-15 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17266069#comment-17266069
 ] 

Andras Gyori commented on YARN-10513:
-

I will analyse the impact of AQC and weight mode on UI2 after the major parts 
got into trunk.

> CS Flexible Auto Queue Creation RM UIv2 modifications
> -
>
> Key: YARN-10513
> URL: https://issues.apache.org/jira/browse/YARN-10513
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-10513) CS Flexible Auto Queue Creation RM UIv2 modifications

2021-01-15 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori reassigned YARN-10513:
---

Assignee: Andras Gyori

> CS Flexible Auto Queue Creation RM UIv2 modifications
> -
>
> Key: YARN-10513
> URL: https://issues.apache.org/jira/browse/YARN-10513
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10535) Make queue placement in CapacityScheduler compliant with auto-queue-placement

2021-01-15 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17266068#comment-17266068
 ] 

Andras Gyori commented on YARN-10535:
-

Thank you [~shuzirra] for the patch. I do not have any major comment at this 
point and I understand that you will add more test cases in the following 
patches. One minor feedback:
 * MappingRuleValidationContextImpl#isParentQueue: I think this method is not 
about being a ParentQueue, but being a dynamic parent candidate, and I found it 
a bit ambiguous. Also, the javadoc of the method talks about child queues, but 
the ManagedParentQueues and AQC-enabled ParentQueues are different in that they 
can have dynamic child queues. This is only a nit, and only address it if you 
have enough time at hand.

> Make queue placement in CapacityScheduler compliant with auto-queue-placement
> -
>
> Key: YARN-10535
> URL: https://issues.apache.org/jira/browse/YARN-10535
> Project: Hadoop YARN
>  Issue Type: Sub-task
>  Components: capacity scheduler
>Reporter: Wangda Tan
>Assignee: Gergely Pollak
>Priority: Major
> Attachments: YARN-10535.001.patch
>
>
> Once YARN-10506 is done, we need to call the API from the queue placement 
> policy to create queues. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS

2021-01-15 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17266033#comment-17266033
 ] 

Andras Gyori commented on YARN-10512:
-

I agree with this approach, +1 LGTM.

> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include 
> mode of operation for CS
> --
>
> Key: YARN-10512
> URL: https://issues.apache.org/jira/browse/YARN-10512
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-10512.001.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> We would like to expose the mode of operation with the RM's /scheduler REST 
> endpoint.
> The field name will be 'mode'.
> All queue representations in the response will be uniformly hold any of the 
> mode values of: "percentage", "absolute", "weight".



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS

2021-01-15 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265986#comment-17265986
 ] 

Andras Gyori commented on YARN-10512:
-

While ParentQueue#getCapacityConfigurationTypeForQueues has a 
Collection signature, it is used for single queues as well, by calling 
it:
{code:java}
getCapacityConfigurationTypeForQueues(ImmutableList.of(queue))
{code}

> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include 
> mode of operation for CS
> --
>
> Key: YARN-10512
> URL: https://issues.apache.org/jira/browse/YARN-10512
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-10512.001.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> We would like to expose the mode of operation with the RM's /scheduler REST 
> endpoint.
> The field name will be 'mode'.
> All queue representations in the response will be uniformly hold any of the 
> mode values of: "percentage", "absolute", "weight".



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10512) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include mode of operation for CS

2021-01-15 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265984#comment-17265984
 ] 

Andras Gyori commented on YARN-10512:
-

Thank you [~snemeth] for the patch! I have some comment, but I am not entirely 
sure if this could be done in this patch.
 * First is simple. I would advise to use ParentQueue's QueueCapacityType(it 
has the weight mode) in CapacitySchedulerInfoHelper#getMode. That said, 
QueueCapacityType and AbstractCSQueue.CapacityConfigType is almost a same, we 
might need to merge them together. I would advise to store its string 
representation in a similar fashion:

{code:java}
private enum QueueCapacityType {
  WEIGHT("weight"),
  ABSOLUTE_RESOURCE("absolute"),
  PERCENT("percentage");

  private String internal;

  QueueCapacityType(String internal) {
this.internal = internal;
  }

  @Override
  public String toString() {
return internal;
  }

}
{code}

 * I think this is a more serious issue, and we should think about it. The way 
getMode discovers the mode might be problematic. Dynamic queues do not have a 
weight mode config, therefore they will be considered as they were in weight 
mode. There is a method ParentQueue#getCapacityConfigurationTypeForQueues that 
is used internally, and handles all corner cases as well. My suggestion would 
be to put it in AbstractCSQueue and use it instead.

> CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to include 
> mode of operation for CS
> --
>
> Key: YARN-10512
> URL: https://issues.apache.org/jira/browse/YARN-10512
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: YARN-10512.001.patch
>
>
> Under this umbrella (YARN-10496), weight-mode has been implemented for CS 
> with YARN-10504.
> We would like to expose the mode of operation with the RM's /scheduler REST 
> endpoint.
> The field name will be 'mode'.
> All queue representations in the response will be uniformly hold any of the 
> mode values of: "percentage", "absolute", "weight".



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10490) "yarn top" command not quitting completely with ctrl+c

2021-01-15 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265890#comment-17265890
 ] 

Andras Gyori commented on YARN-10490:
-

Thank you [~akshink], the patch looks good to me +1.

> "yarn top" command not quitting completely with ctrl+c
> --
>
> Key: YARN-10490
> URL: https://issues.apache.org/jira/browse/YARN-10490
> Project: Hadoop YARN
>  Issue Type: Improvement
>  Components: yarn
>Reporter: Agshin Kazimli
>Assignee: Agshin Kazimli
>Priority: Minor
> Attachments: YARN-10490-001.patch, YARN-10490-001.patch, 
> YARN-10490-002.patch
>
>
> When we quit "yarn top" command using ctrl+c, even though the command prompt 
> returns, the old statistics info is still visible below the prompt.
> And we need to manually type clear command or multiple return keys to get 
> around it. It is reported as a inconvenience for the customer from a 
> usability perspective.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation

2021-01-15 Thread Andras Gyori (Jira)


[ 
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17265868#comment-17265868
 ] 

Andras Gyori commented on YARN-10506:
-

Thank you [~shuzirra] [~pbacsko] [~zhuqi] [~wangda] for the patches, reviews 
and discussion details. I have fixed some minor things (comments and findbug 
fixes), also reintroduced the locking on in autoCreateLeafQueue at LoC 3403, I 
think it is needed here, but might be subject to change. Apart from this, I 
have no additional comments +1.

> Update queue creation logic to use weight mode and allow the flexible 
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch, 
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, 
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, 
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, 
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, 
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, 
> YARN-10506.014.patch
>
>
> The queue creation logic should be updated to use weight mode and support the 
> flexible creation. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Updated] (YARN-10506) Update queue creation logic to use weight mode and allow the flexible static/dynamic creation

2021-01-15 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori updated YARN-10506:

Attachment: YARN-10506.014.patch

> Update queue creation logic to use weight mode and allow the flexible 
> static/dynamic creation
> -
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Benjamin Teke
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10506-006-10504-010.patch, 
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506-010.patch, 
> YARN-10506-012.patch, YARN-10506-013.patch, YARN-10506.001.patch, 
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch, 
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch, 
> YARN-10506.007.patch, YARN-10506.009.patch, YARN-10506.011.patch, 
> YARN-10506.014.patch
>
>
> The queue creation logic should be updated to use weight mode and support the 
> flexible creation. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Assigned] (YARN-10564) Support Auto Queue Creation template configurations

2021-01-14 Thread Andras Gyori (Jira)


 [ 
https://issues.apache.org/jira/browse/YARN-10564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andras Gyori reassigned YARN-10564:
---

Assignee: Andras Gyori  (was: zhuqi)

> Support Auto Queue Creation template configurations
> ---
>
> Key: YARN-10564
> URL: https://issues.apache.org/jira/browse/YARN-10564
> Project: Hadoop YARN
>  Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10564.poc.001.patch
>
>
> Similar to how the template configuration works for ManagedParents, we need 
> to support templates for the new auto queue creation logic. Proposition is to 
> allow wildcards in template configs such as:
> {noformat}
> yarn.scheduler.capacity.root.*.*.weight 10{noformat}
> which would mean, that set weight to 10 of every leaf of every parent under 
> root.
> We should possibly take an approach, that could support arbitrary depth of 
> template configuration, because we might need to lift the limitation of auto 
> queue nesting.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org



[jira] [Commented] (YARN-10564) Support Auto Queue Creation template configurations

2021-01-14 Thread Andras Gyori (Jira)