[jira] [Commented] (YARN-10428) Zombie applications in the YARN queue using FAIR + sizebasedweight
[ https://issues.apache.org/jira/browse/YARN-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273377#comment-17273377 ] Andras Gyori commented on YARN-10428: - Thank you [~yguang11] for your contribution! I am going to try this on a dev cluster and help push this forward. > Zombie applications in the YARN queue using FAIR + sizebasedweight > -- > > Key: YARN-10428 > URL: https://issues.apache.org/jira/browse/YARN-10428 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.8.5 >Reporter: Guang Yang >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10428.001.patch, YARN-10428.002.patch > > > Seeing zombie jobs in the YARN queue that uses FAIR and size based weight > ordering policy . > *Detection:* > The YARN UI shows incorrect number of "Num Schedulable Applications". > *Impact:* > The queue has an upper limit of number of running applications, with zombie > job, it hits the limit even though the number of running applications is far > less than the limit. > *Workaround:* > **Fail-over and restart Resource Manager process. > *Analysis:* > **In the heap dump, we can find the zombie jobs in the `FairOderingPolicy# > schedulableEntities` (see attachment). Take application > "application_1599157165858_29429" for example, it is still in the > `FairOderingPolicy#schedulableEntities` set, however, if we check the log of > resource manager, we can see RM already tried to remove the application: > > ./yarn-yarn-resourcemanager-ip-172-21-153-252.log.2020-09-04-04:2020-09-04 > 04:32:19,730 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue > (ResourceManager Event Processor): Application removed - appId: > application_1599157165858_29429 user: svc_di_data_eng queue: core-data > #user-pending-applications: -3 #user-active-applications: 7 > #queue-pending-applications: 0 #queue-active-applications: 21 > > So it appears RM failed to removed the application from the set. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10428) Zombie applications in the YARN queue using FAIR + sizebasedweight
[ https://issues.apache.org/jira/browse/YARN-10428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andras Gyori reassigned YARN-10428: --- Assignee: Andras Gyori > Zombie applications in the YARN queue using FAIR + sizebasedweight > -- > > Key: YARN-10428 > URL: https://issues.apache.org/jira/browse/YARN-10428 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.8.5 >Reporter: Guang Yang >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10428.001.patch, YARN-10428.002.patch > > > Seeing zombie jobs in the YARN queue that uses FAIR and size based weight > ordering policy . > *Detection:* > The YARN UI shows incorrect number of "Num Schedulable Applications". > *Impact:* > The queue has an upper limit of number of running applications, with zombie > job, it hits the limit even though the number of running applications is far > less than the limit. > *Workaround:* > **Fail-over and restart Resource Manager process. > *Analysis:* > **In the heap dump, we can find the zombie jobs in the `FairOderingPolicy# > schedulableEntities` (see attachment). Take application > "application_1599157165858_29429" for example, it is still in the > `FairOderingPolicy#schedulableEntities` set, however, if we check the log of > resource manager, we can see RM already tried to remove the application: > > ./yarn-yarn-resourcemanager-ip-172-21-153-252.log.2020-09-04-04:2020-09-04 > 04:32:19,730 INFO > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue > (ResourceManager Event Processor): Application removed - appId: > application_1599157165858_29429 user: svc_di_data_eng queue: core-data > #user-pending-applications: -3 #user-active-applications: 7 > #queue-pending-applications: 0 #queue-active-applications: 21 > > So it appears RM failed to removed the application from the set. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273287#comment-17273287 ] zhuqi commented on YARN-10532: -- [~gandras] Thanks for your review and valid suggestions. It makes sense: 1. I will remove the old auto created leaf deletion related logic. 2. Actually, the event triggered by an edit policy thread, which already a background thread that periodically checks , i will make it more readable or just add a new thread. 3. I will handle deletion of ParentQueues. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
[
https://issues.apache.org/jira/browse/YARN-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273121#comment-17273121
]
Hadoop QA commented on YARN-10599:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red}{color} | {color:red} YARN-10599 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10599 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13019521/YARN-10599-002.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/557/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all
> parents
> -
>
> Key: YARN-10599
> URL: https://issues.apache.org/jira/browse/YARN-10599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Fix For: 3.4.0
>
> Attachments: YARN-10599-001.patch, YARN-10599-002.patch
>
>
> The property
> {{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is
> not enabled by default for parent queues. However, users who migrate from FS
> need this property enabled for all parents queues, because FS allows them to
> have dynamic children.
> Note that this is only relevant if we convert directly to weights, it's not
> needed in percentage mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10598: -- Comment: was deleted (was: Okay, since the patch is already committed and findbugs issue is not caused by this one, I'm resolving this jira.) > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10598.001.patch, YARN-10598.002.patch, > YARN-10598.003.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273119#comment-17273119 ] Szilard Nemeth commented on YARN-10598: --- Okay, since the patch is already committed and findbugs issue is not caused by this one, I'm resolving this jira. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10598.001.patch, YARN-10598.002.patch, > YARN-10598.003.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[
https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273117#comment-17273117
]
Hadoop QA commented on YARN-10598:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
8s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 3
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
0s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 9s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
48s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
46s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/555/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html{color}
| {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant findbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 41s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/555/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 6 new + 116 unchanged - 1 fixed = 122 total (was 117)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} |
[jira] [Commented] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
[
https://issues.apache.org/jira/browse/YARN-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273018#comment-17273018
]
Hadoop QA commented on YARN-10599:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s{color}
| {color:red}{color} | {color:red} YARN-10599 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10599 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13019521/YARN-10599-002.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/556/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all
> parents
> -
>
> Key: YARN-10599
> URL: https://issues.apache.org/jira/browse/YARN-10599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Fix For: 3.4.0
>
> Attachments: YARN-10599-001.patch, YARN-10599-002.patch
>
>
> The property
> {{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is
> not enabled by default for parent queues. However, users who migrate from FS
> need this property enabled for all parents queues, because FS allows them to
> have dynamic children.
> Note that this is only relevant if we convert directly to weights, it's not
> needed in percentage mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273017#comment-17273017 ] Szilard Nemeth edited comment on YARN-10598 at 1/27/21, 5:15 PM: - Thanks [~bteke], Latest patch LGTM, committed to trunk. Finbugs was not introduced by this patch and the checkstyle issues can be ignored. Thanks [~gandras] for the review. was (Author: snemeth): Thanks [~bteke], Latest patch LGTM, committed to trunk. Thanks [~gandras] for the review. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10598.001.patch, YARN-10598.002.patch, > YARN-10598.003.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273017#comment-17273017 ] Szilard Nemeth commented on YARN-10598: --- Thanks [~bteke], Latest patch LGTM, committed to trunk. Thanks [~gandras] for the review. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10598.001.patch, YARN-10598.002.patch, > YARN-10598.003.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10598: -- Fix Version/s: 3.4.0 > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10598.001.patch, YARN-10598.002.patch, > YARN-10598.003.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[
https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273007#comment-17273007
]
Hadoop QA commented on YARN-10598:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 3
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
27s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
4s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 24s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
47s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/552/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html{color}
| {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant findbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/552/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 4 new + 116 unchanged - 1 fixed = 120 total (was 117)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} |
[jira] [Commented] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272997#comment-17272997 ] Benjamin Teke commented on YARN-10598: -- [~snemeth] Thanks for the review, both of them are valid points. Uploaded a new patch. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10598.001.patch, YARN-10598.002.patch, > YARN-10598.003.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10598: - Attachment: YARN-10598.003.patch > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10598.001.patch, YARN-10598.002.patch, > YARN-10598.003.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272968#comment-17272968 ] Andras Gyori edited comment on YARN-10532 at 1/27/21, 4:38 PM: --- Thank you [~zhuqi] for the patch. I have come up with several points regarding this approach * In my opinion, implementing auto queue deletion for the legacy auto queue logic is not justified. Old CS users have their own way of keeping their queue hierarchy clean, thus providing this feature would be of little use for them. As for new CS users, they are encouraged to use the new auto queue creation. We should encourage the userbase to move away from ManagedParents, as the code is hard to maintain and very hard to reason about. All in all, I would refrain from adding new features to the old auto queue creation. * I think the approach chosen for this patch is hard to maintain because: ** Does not have a central point where the dynamic queue deletion happens (this was a major pain point of weight calculation as well, we should not repeat this mistake again). QueueManagementChanges and updateQueues both have twisted logic, which reduces readability. ** It does not cover all cases. If I understand correctly, the auto deletion only triggered if CS is reinitialised or a queue management change occurs. In my opinion, we should not rely on events of the users, which may, or may not happen. ** It does not handle deletion of ParentQueues. I think childless ParentQueues should get removed as well. My idea of implementing automatic queue deletion somewhat similar to a garbage collector: # Run a background thread, that periodically checks the whole queue hierarchy (maybe we could store the references of all the dynamic queues, in order to eliminate the cost of traversing the hierarchy) # Store the timestamp of the last submitted application immediately after receiving the queue in CS # Store the timestamp when a dynamic reaches 0 application (either in the queue itself or in an external map) # If a queue still has 0 active application after a duration of a configured expiration time: ## Check, if the last submitted application is also above this threshold ## If yes, delete it ## If no, it means, that an application has been submitted, but not active yet, thus we need to reset the expiration timer for this queue # Remove dynamic ParentQueues the same way, but instead of checking active applications, check the number of children To avoid any race condition in the scheduler, we should use the ReadWriteLock of CS during queue deletion. was (Author: gandras): Thank you [~zhuqi] for the patch. I have come up with several points regarding this approach * In my opinion, implementing auto queue deletion for the legacy auto queue logic is not justified. Old CS users have their own way of keeping their queue hierarchy clean, thus providing this feature would be of little use for them. As for new CS users, they are encouraged to use the new auto queue creation. We should encourage the userbase to move away from ManagedParents, as the code is hard to maintain and very hard to reason about. All in all, I would refrain from adding new features to the old auto queue creation. * I think the approach chosen for this patch is hard to maintain because: ** Does not have a central point where the dynamic queue deletion happens (this was a major pain point of weight calculation as well, we should not repeat this mistake again). QueueManagementChanges and updateQueues both have twisted logic, which reduces readability. ** It does not cover all cases. If I understand correctly, the auto deletion only triggered if CS is reinitialised or a queue management change occurs. In my opinion, we should not rely on events of the users, which may, or may not happen. ** It does not handle deletion of ParentQueues. I think childless ParentQueues should get removed as well. My idea of implementing automatic queue deletion somewhat similar to a garbage collector: # Run a background thread, that periodically checks the whole queue hierarchy (maybe we could store the references of all the dynamic queues, in order to eliminate the cost of traversing the hierarchy) # Store the timestamp when a dynamic reaches 0 application (either in the queue itself or in an external map) # Mark the queues for deletion, that has been without application for a configured time ## Marking introduces a grace period, to avoid race conditions (namely, delete a queue in the same as an application has been submitted ## Application submission to marked queues should be rejected or make the mapping rules step to the next rule # After the grace period, check that the marked queues does not have any application running, and: # ## Delete, if active application number is still == 0 ## Remove mark and timestamp if active
[jira] [Comment Edited] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272968#comment-17272968 ] Andras Gyori edited comment on YARN-10532 at 1/27/21, 4:28 PM: --- Thank you [~zhuqi] for the patch. I have come up with several points regarding this approach * In my opinion, implementing auto queue deletion for the legacy auto queue logic is not justified. Old CS users have their own way of keeping their queue hierarchy clean, thus providing this feature would be of little use for them. As for new CS users, they are encouraged to use the new auto queue creation. We should encourage the userbase to move away from ManagedParents, as the code is hard to maintain and very hard to reason about. All in all, I would refrain from adding new features to the old auto queue creation. * I think the approach chosen for this patch is hard to maintain because: ** Does not have a central point where the dynamic queue deletion happens (this was a major pain point of weight calculation as well, we should not repeat this mistake again). QueueManagementChanges and updateQueues both have twisted logic, which reduces readability. ** It does not cover all cases. If I understand correctly, the auto deletion only triggered if CS is reinitialised or a queue management change occurs. In my opinion, we should not rely on events of the users, which may, or may not happen. ** It does not handle deletion of ParentQueues. I think childless ParentQueues should get removed as well. My idea of implementing automatic queue deletion somewhat similar to a garbage collector: # Run a background thread, that periodically checks the whole queue hierarchy (maybe we could store the references of all the dynamic queues, in order to eliminate the cost of traversing the hierarchy) # Store the timestamp when a dynamic reaches 0 application (either in the queue itself or in an external map) # Mark the queues for deletion, that has been without application for a configured time ## Marking introduces a grace period, to avoid race conditions (namely, delete a queue in the same as an application has been submitted ## Application submission to marked queues should be rejected or make the mapping rules step to the next rule # After the grace period, check that the marked queues does not have any application running, and: # ## Delete, if active application number is still == 0 ## Remove mark and timestamp if active application number > 0 # Remove dynamic ParentQueues the same way, but instead of checking active applications, check the number of children Now, I see that marking would introduce a surprising behaviour, but I can not come up with a way that is less disruptive and solves the race condition at the same time. was (Author: gandras): Thank you [~zhuqi] for the patch. I have come up with several points regarding this approach * In my opinion, implementing auto queue deletion for the legacy auto queue logic is not justified. Old CS users have their own way of keeping their queue hierarchy clean, thus providing this feature would be of little use for them. As for new CS users, they are encouraged to use the new auto queue creation. We should encourage the userbase to move away from ManagedParents, as the code is hard to maintain and very hard to reason about. * I think the approach chosen for this patch is hard to maintain because: ** Does not have a central point where the dynamic queue deletion happens (this was a major pain point of weight calculation as well, we should not repeat this mistake again). QueueManagementChanges and updateQueues both have twisted logic, which reduces readability. ** It does not cover all cases. If I understand correctly, the auto deletion only triggered if CS is reinitialised or a queue management change occurs. In my opinion, we should not rely on events of the users, which may, or may not happen. ** It does not handle deletion of ParentQueues. I think childless ParentQueues should get removed as well. My idea of implementing automatic queue deletion somewhat similar to a garbage collector: # Run a background thread, that periodically checks the whole queue hierarchy (maybe we could store the references of all the dynamic queues, in order to eliminate the cost of traversing the hierarchy) # Store the timestamp when a dynamic reaches 0 application (either in the queue itself or in an external map) # Mark the queues for deletion, that has been without application for a configured time ## Marking introduces a grace period, to avoid race conditions (namely, delete a queue in the same as an application has been submitted ## Application submission to marked queues should be rejected or make the mapping rules step to the next rule # After the grace period, check that the marked queues does not have any application running, and:
[jira] [Commented] (YARN-10532) Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is not being used
[ https://issues.apache.org/jira/browse/YARN-10532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272968#comment-17272968 ] Andras Gyori commented on YARN-10532: - Thank you [~zhuqi] for the patch. I have come up with several points regarding this approach * In my opinion, implementing auto queue deletion for the legacy auto queue logic is not justified. Old CS users have their own way of keeping their queue hierarchy clean, thus providing this feature would be of little use for them. As for new CS users, they are encouraged to use the new auto queue creation. We should encourage the userbase to move away from ManagedParents, as the code is hard to maintain and very hard to reason about. * I think the approach chosen for this patch is hard to maintain because: ** Does not have a central point where the dynamic queue deletion happens (this was a major pain point of weight calculation as well, we should not repeat this mistake again). QueueManagementChanges and updateQueues both have twisted logic, which reduces readability. ** It does not cover all cases. If I understand correctly, the auto deletion only triggered if CS is reinitialised or a queue management change occurs. In my opinion, we should not rely on events of the users, which may, or may not happen. ** It does not handle deletion of ParentQueues. I think childless ParentQueues should get removed as well. My idea of implementing automatic queue deletion somewhat similar to a garbage collector: # Run a background thread, that periodically checks the whole queue hierarchy (maybe we could store the references of all the dynamic queues, in order to eliminate the cost of traversing the hierarchy) # Store the timestamp when a dynamic reaches 0 application (either in the queue itself or in an external map) # Mark the queues for deletion, that has been without application for a configured time ## Marking introduces a grace period, to avoid race conditions (namely, delete a queue in the same as an application has been submitted ## Application submission to marked queues should be rejected or make the mapping rules step to the next rule # After the grace period, check that the marked queues does not have any application running, and: ## Delete, if active application number is still == 0 ## Remove mark and timestamp if active application number > 0 # Remove dynamic ParentQueues the same way, but instead of checking active applications, check the number of children Now, I see that marking would introduce a surprising behaviour, but I can not come up with a way that is less disruptive and solves the race condition at the same time. > Capacity Scheduler Auto Queue Creation: Allow auto delete queue when queue is > not being used > > > Key: YARN-10532 > URL: https://issues.apache.org/jira/browse/YARN-10532 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: zhuqi >Priority: Major > Attachments: YARN-10532.001.patch, YARN-10532.002.patch, > YARN-10532.003.patch, YARN-10532.004.patch, YARN-10532.005.patch, > YARN-10532.006.patch, YARN-10532.007.patch > > > It's better if we can delete auto-created queues when they are not in use for > a period of time (like 5 mins). It will be helpful when we have a large > number of auto-created queues (e.g. from 500 users), but only a small subset > of queues are actively used. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272939#comment-17272939 ] Szilard Nemeth commented on YARN-10598: --- Hi [~bteke], Latest patch looks good to me. Two minor things only: # This method is not used: org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySchedDynamicConfig#submitApp # It would be useful to add javadoc for all static final variables that are among valid values of autoCreationType, queueType and auto creation eligibility: org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.helper.CapacitySchedulerInfoHelper I can also accept this in a follow-up fix as it's not critical. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10598.001.patch, YARN-10598.002.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
[
https://issues.apache.org/jira/browse/YARN-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272937#comment-17272937
]
Hadoop QA commented on YARN-10599:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s{color}
| {color:red}{color} | {color:red} YARN-10599 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-10599 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13019521/YARN-10599-002.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/554/console |
| versions | git=2.17.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all
> parents
> -
>
> Key: YARN-10599
> URL: https://issues.apache.org/jira/browse/YARN-10599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Fix For: 3.4.0
>
> Attachments: YARN-10599-001.patch, YARN-10599-002.patch
>
>
> The property
> {{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is
> not enabled by default for parent queues. However, users who migrate from FS
> need this property enabled for all parents queues, because FS allows them to
> have dynamic children.
> Note that this is only relevant if we convert directly to weights, it's not
> needed in percentage mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
[
https://issues.apache.org/jira/browse/YARN-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272929#comment-17272929
]
Hadoop QA commented on YARN-10599:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 37m
51s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color}
| {color:red}{color} | {color:red} YARN-10599 does not apply to trunk. Rebase
required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for
help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/553/artifact/out/Dockerfile
|
| JIRA Issue | YARN-10599 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13019521/YARN-10599-002.patch |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/553/console |
| versions | git=2.25.1 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all
> parents
> -
>
> Key: YARN-10599
> URL: https://issues.apache.org/jira/browse/YARN-10599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Fix For: 3.4.0
>
> Attachments: YARN-10599-001.patch, YARN-10599-002.patch
>
>
> The property
> {{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is
> not enabled by default for parent queues. However, users who migrate from FS
> need this property enabled for all parents queues, because FS allows them to
> have dynamic children.
> Note that this is only relevant if we convert directly to weights, it's not
> needed in percentage mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
[
https://issues.apache.org/jira/browse/YARN-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-10599:
--
Fix Version/s: 3.4.0
> fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all
> parents
> -
>
> Key: YARN-10599
> URL: https://issues.apache.org/jira/browse/YARN-10599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Fix For: 3.4.0
>
> Attachments: YARN-10599-001.patch, YARN-10599-002.patch
>
>
> The property
> {{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is
> not enabled by default for parent queues. However, users who migrate from FS
> need this property enabled for all parents queues, because FS allows them to
> have dynamic children.
> Note that this is only relevant if we convert directly to weights, it's not
> needed in percentage mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
[
https://issues.apache.org/jira/browse/YARN-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272904#comment-17272904
]
Szilard Nemeth commented on YARN-10599:
---
Thanks [~pbacsko],
Latest patch LGTM, committed to trunk.
> fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all
> parents
> -
>
> Key: YARN-10599
> URL: https://issues.apache.org/jira/browse/YARN-10599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Fix For: 3.4.0
>
> Attachments: YARN-10599-001.patch, YARN-10599-002.patch
>
>
> The property
> {{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is
> not enabled by default for parent queues. However, users who migrate from FS
> need this property enabled for all parents queues, because FS allows them to
> have dynamic children.
> Note that this is only relevant if we convert directly to weights, it's not
> needed in percentage mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
[
https://issues.apache.org/jira/browse/YARN-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10599:
Attachment: YARN-10599-002.patch
> fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all
> parents
> -
>
> Key: YARN-10599
> URL: https://issues.apache.org/jira/browse/YARN-10599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10599-001.patch, YARN-10599-002.patch
>
>
> The property
> {{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is
> not enabled by default for parent queues. However, users who migrate from FS
> need this property enabled for all parents queues, because FS allows them to
> have dynamic children.
> Note that this is only relevant if we convert directly to weights, it's not
> needed in percentage mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
[
https://issues.apache.org/jira/browse/YARN-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272888#comment-17272888
]
Hadoop QA commented on YARN-10599:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
28s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 3
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
29s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 13s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
47s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
45s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/551/artifact/out/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-warnings.html{color}
| {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
in trunk has 1 extant findbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 7s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green}
[jira] [Commented] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272881#comment-17272881 ] Benjamin Teke commented on YARN-10598: -- Thank [~gandras] for the review. Fixed the suggestions and the checkstyle issues that doesn't need a larger refactor. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10598.001.patch, YARN-10598.002.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10598: - Attachment: YARN-10598.002.patch > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10598.001.patch, YARN-10598.002.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
[
https://issues.apache.org/jira/browse/YARN-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272852#comment-17272852
]
Andras Gyori commented on YARN-10599:
-
Thank you [~pbacsko] for the patch. LGTM +1.
> fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all
> parents
> -
>
> Key: YARN-10599
> URL: https://issues.apache.org/jira/browse/YARN-10599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10599-001.patch
>
>
> The property
> {{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is
> not enabled by default for parent queues. However, users who migrate from FS
> need this property enabled for all parents queues, because FS allows them to
> have dynamic children.
> Note that this is only relevant if we convert directly to weights, it's not
> needed in percentage mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10598) CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the creation type with additional information
[ https://issues.apache.org/jira/browse/YARN-10598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272822#comment-17272822 ] Andras Gyori commented on YARN-10598: - Thank you [~bteke] for the patch! My additions: * You do not need to separate the flexible dynamic queue parts in getAutoCreationEligibility. ParentQueue#isEligibleForAutoQueueCreation handles exactly those two cases, so it could be merged. * TestRMWebServicesCapacitySchedDynamicConfig has unused imports and a method named submitApp, but I suppose they are referenced in the checkstyle error list The overall logic seems good to me. > CS Flexible Auto Queue Creation: Modify RM /scheduler endpoint to extend the > creation type with additional information > -- > > Key: YARN-10598 > URL: https://issues.apache.org/jira/browse/YARN-10598 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10598.001.patch > > > Under this umbrella (YARN-10496), weight-mode has been implemented for CS > with YARN-10504. > Auto-queue creation has been also implemented with YARN-10506. > Connected to this effort, we would like to expose the type of the queue with > the RM's /scheduler REST endpoint. > To extend/modify the values added in YARN-10581 these 3 fields will describe > a queue: > * queueType : *parent/leaf* > * creationMethod : *static/dynamicLegacy/dynamicFlexible* > * autoCreationEligibility : *off/legacy/flexible* > After this change here are some example cases: > * Static parent queue which has the auto-creation-enabled-v2 false: > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *off* > * Static managed parent (can have dynamic children): > ** queueType : *parent* > ** creationMethod : *static* > ** autoCreationEligibility : *legacy* > * Legacy auto-created leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicLegacy* > ** autoCreationEligibility : *off* > * Auto-created (v2) parent queue, (implicitly) auto-creation-enabled-v2 > true: > ** queueType : *parent* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *flexible* > * Auto-created (v2) leaf queue (cannot have children): > ** queueType : *leaf* > ** creationMethod : *dynamicFlexible* > ** autoCreationEligibility : *off* -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
[
https://issues.apache.org/jira/browse/YARN-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10599:
Attachment: YARN-10599-001.patch
> fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all
> parents
> -
>
> Key: YARN-10599
> URL: https://issues.apache.org/jira/browse/YARN-10599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10599-001.patch
>
>
> The property
> {{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is
> not enabled by default for parent queues. However, users who migrate from FS
> need this property enabled for all parents queues, because FS allows them to
> have dynamic children.
> Note that this is only relevant if we convert directly to weights, it's not
> needed in percentage mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[ https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272778#comment-17272778 ] Peter Bacsko edited comment on YARN-10596 at 1/27/21, 11:17 AM: Just for proper documentation: this solution means that auto-create-v2 property must be enabled for all parents when converting from FS. See YARN-10599. was (Author: pbacsko): Just for proper documentation: this solution means that auto-create-v2 property must be enabled for all parents. See YARN-10599. > Allow static definition of childless ParentQueues with auto-queue-creation-v2 > enabled > - > > Key: YARN-10596 > URL: https://issues.apache.org/jira/browse/YARN-10596 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10596.001.patch, YARN-10596.002.patch > > > The old auto queue creation/managed queue logic allowed the definition of > childless parents to be created statically, if the auto-create-child-queue > flag was turned on the parent (thus making it a ManagedParentQueue). > Since it is not an edge case, we also need to support the creation of a > ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, > even when no child queue is defined under the parent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[ https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272778#comment-17272778 ] Peter Bacsko commented on YARN-10596: - Just for proper documentation: this solution means that auto-create-v2 property must be enabled for all parents. See YARN-10599. > Allow static definition of childless ParentQueues with auto-queue-creation-v2 > enabled > - > > Key: YARN-10596 > URL: https://issues.apache.org/jira/browse/YARN-10596 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10596.001.patch, YARN-10596.002.patch > > > The old auto queue creation/managed queue logic allowed the definition of > childless parents to be created statically, if the auto-create-child-queue > flag was turned on the parent (thus making it a ManagedParentQueue). > Since it is not an edge case, we also need to support the creation of a > ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, > even when no child queue is defined under the parent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10599) fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all parents
[
https://issues.apache.org/jira/browse/YARN-10599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Bacsko updated YARN-10599:
Description:
The property
{{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is not
enabled by default for parent queues. However, users who migrate from FS need
this property enabled for all parents queues, because FS allows them to have
dynamic children.
Note that this is only relevant if we convert directly to weights, it's not
needed in percentage mode.
was:
The property
{{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is not
enabled by default for parent queues. However, users who migrate from FS need
this property enabled for all parents queues, because FS allows them to have
dynamic queues.
Note that this is only relevant if we convert directly to weights, it's not
needed in percentage mode.
> fs2cs should generate new "auto-queue-creation-v2.enabled" properties for all
> parents
> -
>
> Key: YARN-10599
> URL: https://issues.apache.org/jira/browse/YARN-10599
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
>
> The property
> {{yarn.scheduler.capacity..auto-queue-creation-v2.enabled}} is
> not enabled by default for parent queues. However, users who migrate from FS
> need this property enabled for all parents queues, because FS allows them to
> have dynamic children.
> Note that this is only relevant if we convert directly to weights, it's not
> needed in percentage mode.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
