[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[ https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272778#comment-17272778 ] Peter Bacsko commented on YARN-10596: - Just for proper documentation: this solution means that auto-create-v2 property must be enabled for all parents. See YARN-10599. > Allow static definition of childless ParentQueues with auto-queue-creation-v2 > enabled > - > > Key: YARN-10596 > URL: https://issues.apache.org/jira/browse/YARN-10596 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10596.001.patch, YARN-10596.002.patch > > > The old auto queue creation/managed queue logic allowed the definition of > childless parents to be created statically, if the auto-create-child-queue > flag was turned on the parent (thus making it a ManagedParentQueue). > Since it is not an edge case, we also need to support the creation of a > ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, > even when no child queue is defined under the parent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[
https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272247#comment-17272247
]
Hadoop QA commented on YARN-10596:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
42s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
52s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
7s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 34s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
46s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 50s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} the
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[ https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272161#comment-17272161 ] Szilard Nemeth commented on YARN-10596: --- Hi [~gandras], Thanks for working on this. Latest patch looks good to me, committed to trunk. Thanks [~pbacsko] and [~bteke] for the reviews. > Allow static definition of childless ParentQueues with auto-queue-creation-v2 > enabled > - > > Key: YARN-10596 > URL: https://issues.apache.org/jira/browse/YARN-10596 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10596.001.patch, YARN-10596.002.patch > > > The old auto queue creation/managed queue logic allowed the definition of > childless parents to be created statically, if the auto-create-child-queue > flag was turned on the parent (thus making it a ManagedParentQueue). > Since it is not an edge case, we also need to support the creation of a > ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, > even when no child queue is defined under the parent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[ https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272123#comment-17272123 ] Andras Gyori commented on YARN-10596: - Thank you [~bteke] and [~pbacsko] for the review. Fixed the test and checkstyle errors, also incorporated [~bteke]'s suggestion. We have decided to defer the introduction of a global property until we have enough feedback that it is indeed a popular request among users. > Allow static definition of childless ParentQueues with auto-queue-creation-v2 > enabled > - > > Key: YARN-10596 > URL: https://issues.apache.org/jira/browse/YARN-10596 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10596.001.patch, YARN-10596.002.patch > > > The old auto queue creation/managed queue logic allowed the definition of > childless parents to be created statically, if the auto-create-child-queue > flag was turned on the parent (thus making it a ManagedParentQueue). > Since it is not an edge case, we also need to support the creation of a > ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, > even when no child queue is defined under the parent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[
https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271563#comment-17271563
]
Hadoop QA commented on YARN-10596:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
8s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
12s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 28s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
48s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.18.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~18.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 32s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/542/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 32s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[
https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271419#comment-17271419
]
Benjamin Teke commented on YARN-10596:
--
[~gandras] thanks for the patch. I have a suggestion for making the condition a
bit more readable (for me at least). Consider the following part:
{code:java}
boolean isEligibleForAutoQueueCreation = false;
// Auto created parent queues might not have static children, but they
// must be kept as a ParentQueue
CSQueue oldQueue = oldQueues.get(fullQueueName);
if (oldQueue instanceof ParentQueue) {
isEligibleForAutoQueueCreation = ((ParentQueue)
oldQueue).isDynamicQueue();
}
// if a queue is eligible for auto queue creation v2
// it must be a ParentQueue
if (!isEligibleForAutoQueueCreation) {
isEligibleForAutoQueueCreation =
conf.isAutoQueueCreationV2Enabled(fullQueueName);
}
{code}
Basically the condition is: if the queue is a dynamic parent, or it has the the
AutoQueueCreationV2 enabled then it can be an empty parent. As the old
autoQueueEnabled flag is parsed and stored as a boolean I think it would be
nicer if you got the conf.isAutoQueueCreationV2Enabled there and kept the
isDynamicParent intact (as isDynamicParent induces the eligibility for auto
creation, however it is not the same). You could skip the (bit) misleading if
(!isEligibleForAutoQueueCreation) part, and instead of the if
(childQueueNames.size() == 0 && !isEligibleForAutoQueueCreation) condition you
could say that the queue should not have 0 childQueues, should not be a dynamic
parent and should not be eligible for auto creation based on the config.
> Allow static definition of childless ParentQueues with auto-queue-creation-v2
> enabled
> -
>
> Key: YARN-10596
> URL: https://issues.apache.org/jira/browse/YARN-10596
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10596.001.patch
>
>
> The old auto queue creation/managed queue logic allowed the definition of
> childless parents to be created statically, if the auto-create-child-queue
> flag was turned on the parent (thus making it a ManagedParentQueue).
> Since it is not an edge case, we also need to support the creation of a
> ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled,
> even when no child queue is defined under the parent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[ https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271355#comment-17271355 ] Peter Bacsko commented on YARN-10596: - Ok, I've just read this sentence: _Since it is not an edge case, we also need to support the creation of a ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, even when no child queue is defined under the parent._ My understanding was that by default, all parents are allowed of having dynamic queues created. If this is not true, that means: 1. If someone wants all parents to have dynamic queues (which is standard for FS users who are migrating), then this property must be set for all parents. 2. This affects fs2cs tool, because it does not set "auto-queue-creation-v2" at all. I pointed this out in my comment https://issues.apache.org/jira/browse/YARN-10506?focusedCommentId=17264903=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17264903. See paragraph which begins with "We also have to think about the following scenario" and conclusion point #2. > Allow static definition of childless ParentQueues with auto-queue-creation-v2 > enabled > - > > Key: YARN-10596 > URL: https://issues.apache.org/jira/browse/YARN-10596 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Attachments: YARN-10596.001.patch > > > The old auto queue creation/managed queue logic allowed the definition of > childless parents to be created statically, if the auto-create-child-queue > flag was turned on the parent (thus making it a ManagedParentQueue). > Since it is not an edge case, we also need to support the creation of a > ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled, > even when no child queue is defined under the parent. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10596) Allow static definition of childless ParentQueues with auto-queue-creation-v2 enabled
[
https://issues.apache.org/jira/browse/YARN-10596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271342#comment-17271342
]
Peter Bacsko commented on YARN-10596:
-
[~gandras] I have a question.
{noformat}
// Auto created parent queues might not have static children, but they
// must be kept as a ParentQueue
CSQueue oldQueue = oldQueues.get(fullQueueName);
if (oldQueue instanceof ParentQueue) {
isDynamicParent = ((ParentQueue) oldQueue).isDynamicQueue();
isEligibleForAutoQueueCreation = ((ParentQueue) oldQueue).isDynamicQueue();
}
{noformat}
Without explicitly saying if a queue is a parent or not, how do you distinguish
between a parent with no leafs and an ordinary leaf queu? How do you detect
that a queue with no children is intended to be a parent?
> Allow static definition of childless ParentQueues with auto-queue-creation-v2
> enabled
> -
>
> Key: YARN-10596
> URL: https://issues.apache.org/jira/browse/YARN-10596
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Attachments: YARN-10596.001.patch
>
>
> The old auto queue creation/managed queue logic allowed the definition of
> childless parents to be created statically, if the auto-create-child-queue
> flag was turned on the parent (thus making it a ManagedParentQueue).
> Since it is not an edge case, we also need to support the creation of a
> ParentQueue instead of a LeafQueue, if auto-queue-creation-v2 is enabled,
> even when no child queue is defined under the parent.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
