[jira] [Commented] (YARN-10620) fs2cs: parentQueue for certain placement rules are not set during conversion
[
https://issues.apache.org/jira/browse/YARN-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282426#comment-17282426
]
Hadoop QA commented on YARN-10620:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 22m
12s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
12s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 42s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
54s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_282-8u282-b08-0ubuntu1~20.04-b08 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 52s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.10+9-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green}{color} | {color
[jira] [Commented] (YARN-10620) fs2cs: parentQueue for certain placement rules are not set during conversion
[
https://issues.apache.org/jira/browse/YARN-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282357#comment-17282357
]
Szilard Nemeth commented on YARN-10620:
---
Hi [~pbacsko],
Thanks for working on this.
Latest patch LGTM, just committed to trunk.
Thanks [~gandras] for the review.
> fs2cs: parentQueue for certain placement rules are not set during conversion
>
>
> Key: YARN-10620
> URL: https://issues.apache.org/jira/browse/YARN-10620
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Fix For: 3.4.0
>
> Attachments: YARN-10620-001.patch, YARN-10620-002.patch
>
>
> There are some placement rules in FS which are currently not handled properly
> by fs2cs:
> {noformat}
>
>
>
>
> {noformat}
> The first rule means that if the user queue doesn't exist, it should be
> created as {{root.}}.
> The second means the same thing, except refers to the primary group instead
> of the submitting user: {{root.}}.
> The problem is that in order for the create="true" setting to take effect, we
> must set the parent queue in the generated JSON:
> Current:
> {noformat}
> {
> "rules" : [ {
> "type" : "user",
> "matches" : "*",
> "policy" : "user",
> "fallbackResult" : "skip",
> "create" : true
> }, {
> "type" : "user",
> "matches" : "*",
> "policy" : "primaryGroup",
> "fallbackResult" : "skip",
> "create" : true
> } ]
> }
> {noformat}
> Expected:
> {noformat}
> {
> "rules" : [ {
> "type" : "user",
> "matches" : "*",
> "policy" : "user",
> "fallbackResult" : "skip",
> "parentQueue": "root",
> "create" : true
> }, {
> "type" : "user",
> "matches" : "*",
> "policy" : "primaryGroup",
> "fallbackResult" : "skip",
> "parentQueue": "root",
> "create" : true
> } ]
> {noformat}
> This is missing right now and it need to be fixed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10620) fs2cs: parentQueue for certain placement rules are not set during conversion
[
https://issues.apache.org/jira/browse/YARN-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282323#comment-17282323
]
Andras Gyori commented on YARN-10620:
-
[~pbacsko] I agree with your solution, the set is more appropriate and
efficient here +1.
> fs2cs: parentQueue for certain placement rules are not set during conversion
>
>
> Key: YARN-10620
> URL: https://issues.apache.org/jira/browse/YARN-10620
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10620-001.patch, YARN-10620-002.patch
>
>
> There are some placement rules in FS which are currently not handled properly
> by fs2cs:
> {noformat}
>
>
>
>
> {noformat}
> The first rule means that if the user queue doesn't exist, it should be
> created as {{root.}}.
> The second means the same thing, except refers to the primary group instead
> of the submitting user: {{root.}}.
> The problem is that in order for the create="true" setting to take effect, we
> must set the parent queue in the generated JSON:
> {noformat}
> {
> "rules" : [ {
> "type" : "user",
> "matches" : "*",
> "policy" : "user",
> "fallbackResult" : "skip",
> "create" : true
> }, {
> "type" : "user",
> "matches" : "*",
> "policy" : "primaryGroup",
> "fallbackResult" : "skip",
> "create" : true
> } ]
> }
> {noformat}
> This is missing right now and it need to be fixed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10620) fs2cs: parentQueue for certain placement rules are not set during conversion
[
https://issues.apache.org/jira/browse/YARN-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282322#comment-17282322
]
Peter Bacsko commented on YARN-10620:
-
[~gandras] thanks, I modified the code a little bit, used a set instead of an
array.
> fs2cs: parentQueue for certain placement rules are not set during conversion
>
>
> Key: YARN-10620
> URL: https://issues.apache.org/jira/browse/YARN-10620
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10620-001.patch, YARN-10620-002.patch
>
>
> There are some placement rules in FS which are currently not handled properly
> by fs2cs:
> {noformat}
>
>
>
>
> {noformat}
> The first rule means that if the user queue doesn't exist, it should be
> created as {{root.}}.
> The second means the same thing, except refers to the primary group instead
> of the submitting user: {{root.}}.
> The problem is that in order for the create="true" setting to take effect, we
> must set the parent queue in the generated JSON:
> {noformat}
> {
> "rules" : [ {
> "type" : "user",
> "matches" : "*",
> "policy" : "user",
> "fallbackResult" : "skip",
> "create" : true
> }, {
> "type" : "user",
> "matches" : "*",
> "policy" : "primaryGroup",
> "fallbackResult" : "skip",
> "create" : true
> } ]
> }
> {noformat}
> This is missing right now and it need to be fixed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10620) fs2cs: parentQueue for certain placement rules are not set during conversion
[
https://issues.apache.org/jira/browse/YARN-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282314#comment-17282314
]
Andras Gyori commented on YARN-10620:
-
Thank you [~pbacsko] for the patch. It looks good to me, I have one minor
addition to this:
* Checking the policy would be more readable, if the policies were stored in a
constant array, and checked, if the policy is contained in this array. This
would reduce the expression to !usePercentages && ArrayUtils.contains(policies,
policy).
> fs2cs: parentQueue for certain placement rules are not set during conversion
>
>
> Key: YARN-10620
> URL: https://issues.apache.org/jira/browse/YARN-10620
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Attachments: YARN-10620-001.patch
>
>
> There are some placement rules in FS which are currently not handled properly
> by fs2cs:
> {noformat}
>
>
>
>
> {noformat}
> The first rule means that if the user queue doesn't exist, it should be
> created as {{root.}}.
> The second means the same thing, except refers to the primary group instead
> of the submitting user: {{root.}}.
> The problem is that in order for the create="true" setting to take effect, we
> must set the parent queue in the generated JSON:
> {noformat}
> {
> "rules" : [ {
> "type" : "user",
> "matches" : "*",
> "policy" : "user",
> "fallbackResult" : "skip",
> "create" : true
> }, {
> "type" : "user",
> "matches" : "*",
> "policy" : "primaryGroup",
> "fallbackResult" : "skip",
> "create" : true
> } ]
> }
> {noformat}
> This is missing right now and it need to be fixed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10620) fs2cs: parentQueue for certain placement rules are not set during conversion
[
https://issues.apache.org/jira/browse/YARN-10620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17282020#comment-17282020
]
Hadoop QA commented on YARN-10620:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
14s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
19s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 50s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
50s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs
config; considering switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
48s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_275-8u275-b01-0ubuntu1~20.04-b01 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
48s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 2s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.9.1+1-Ubuntu-0ubuntu1.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green}{color} | {c
