[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064779#comment-17064779
]
Hudson commented on YARN-10198:
---
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #18073 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/18073/])
YARN-10198. Remove Validation of Queue Exist under Managed Parent Queue
(pjoseph: rev d312991ead37a5f840577a39420a18f0621b09dc)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/placement/TestUserGroupMappingPlacementRule.java
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/placement/UserGroupMappingPlacementRule.java
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: YARN-10198-001.patch, YARN-10198-002.patch,
> YARN-10198-003.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064768#comment-17064768
]
Prabhu Joseph commented on YARN-10198:
--
Thanks [~pbacsko], have pushed the [^YARN-10198-003.patch] to trunk.
Thanks [~maniraj...@gmail.com] for the review.
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch, YARN-10198-002.patch,
> YARN-10198-003.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064750#comment-17064750
]
Peter Bacsko commented on YARN-10198:
-
Due to infra issues, we can't see the comment about the Precommit builds.
Here's a build about patch v3:
https://builds.apache.org/job/PreCommit-YARN-Build/25730/console
{noformat}
+1 overall
| Vote |Subsystem | Runtime | Comment
| 0 | reexec | 0m 48s | Docker mode activated.
| | || Prechecks
| +1 | @author | 0m 0s | The patch does not contain any @author
| | || tags.
| +1 | test4tests | 0m 0s | The patch appears to include 1 new or
| | || modified test files.
| | || trunk Compile Tests
| +1 | mvninstall | 20m 57s | trunk passed
| +1 | compile | 0m 43s | trunk passed
| +1 | checkstyle | 0m 34s | trunk passed
| +1 | mvnsite | 0m 46s | trunk passed
| +1 |shadedclient | 15m 27s | branch has no errors when building and
| | || testing our client artifacts.
| +1 |findbugs | 1m 34s | trunk passed
| +1 | javadoc | 0m 29s | trunk passed
| | || Patch Compile Tests
| +1 | mvninstall | 0m 42s | the patch passed
| +1 | compile | 0m 37s | the patch passed
| +1 | javac | 0m 37s | the patch passed
| +1 | checkstyle | 0m 28s | the patch passed
| +1 | mvnsite | 0m 41s | the patch passed
| +1 | whitespace | 0m 0s | The patch has no whitespace issues.
| +1 |shadedclient | 14m 2s | patch has no errors when building and
| | || testing our client artifacts.
| +1 |findbugs | 1m 37s | the patch passed
| +1 | javadoc | 0m 28s | the patch passed
| | || Other Tests
| +1 |unit | 89m 8s | hadoop-yarn-server-resourcemanager in
| | || the patch passed.
| +1 | asflicense | 0m 26s | The patch does not generate ASF
| | || License warnings.
| | | 149m 17s |
{noformat}
[~sunilg] [~prabhujoseph] could you guys commit this change?
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch, YARN-10198-002.patch,
> YARN-10198-003.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063433#comment-17063433
]
Peter Bacsko commented on YARN-10198:
-
[~maniraj...@gmail.com] I was thinking about adding a comment about null-check.
In short: it's not necessary, because instanceof simply returns false if the
object is null.
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch, YARN-10198-002.patch,
> YARN-10198-003.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063380#comment-17063380
]
Manikandan R commented on YARN-10198:
-
Should we do NULL check for {{parent}} after Line No. 163 to address mappings
like u:%user:%primary_group ?
Other than this, Looks good to me. Thanks [~pbacsko] and [~prabhujoseph].
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch, YARN-10198-002.patch,
> YARN-10198-003.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063301#comment-17063301
]
Prabhu Joseph commented on YARN-10198:
--
Thanks [~pbacsko] for the patch. The patch looks good to me, +1.
Will wait for [~maniraj...@gmail.com] to review the latest patch.
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch, YARN-10198-002.patch,
> YARN-10198-003.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062750#comment-17062750
]
Manikandan R commented on YARN-10198:
-
[~pbacsko] Thanks for extending the patch.
1. Yes, Unlike other placement rules in FS,
"SecondaryGroupExistingPlacementRule" expects Queue exist. Except
"SecondaryGroupExistingPlacementRule", all other rules (even
PrimaryGroupExistingPlacementRule) depends on "FSPlacementRule#createQueue"
flag as well in conjunction with FSPlacementRule#configuredQueue.
Modified patch is in line with above FS flow.
Line No.161 can make use of getPrimaryGroup() method.
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch, YARN-10198-002.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062531#comment-17062531
]
Peter Bacsko commented on YARN-10198:
-
I uploaded patch v2.
Couple of things:
1. I was reasoning about the validation in case of {{%secondary_group}} and
based on the existing code, this cannot have a managed parent. The queue has to
exist, see {{getSecondaryGroup()}}. This also seems to be in line with Fair
Scheduler, where this placement rule is called
"SecondaryGroupExistingPlacementRule".
2. I had to do some refactor in {{UserGroupMappingPlacementRule}} because
readability is becoming more of a concern with the added features. It will be
addressed by YARN-10199 hopefully.
3. Added extra unit tests. Existing tests are not broken by this change (at
least not the ones in {{TestUserGroupMappingPlacementRule}}).
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch, YARN-10198-002.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062414#comment-17062414
]
Peter Bacsko commented on YARN-10198:
-
[~sunilg] the patch needs to be extended. In its current form, it's not enough.
I'm going to attach a new version today.
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17062320#comment-17062320
]
Sunil G commented on YARN-10198:
[~prabhujoseph] [~maniraj...@gmail.com] [~pbacsko] Do we have a consensus in
this attached patch and the approach.
I think patch looks fine. Thoughts?
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061596#comment-17061596
]
Prabhu Joseph commented on YARN-10198:
--
Had a discussion with [~pbacsko] and the above two cases will be handled as
part of this Jira.
1. Validation of leaf queue present has to happen for regular parent queue for
both [regularParentQueue].%primary_group and
[regularParentQueue].%secondary_group
2. Validation will be skipped for managed parent queue alone.
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17061033#comment-17061033
]
Manikandan R commented on YARN-10198:
-
[~pbacsko] Thanks for the patch.
1. Are we planning to have this check only for regular parent queue in separate
jira?
2. Are we going to handle [managedParentQueue].%secondary_group in separate
jira?
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060958#comment-17060958
]
Peter Bacsko commented on YARN-10198:
-
Test failure is unrelated.
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060942#comment-17060942
]
Hadoop QA commented on YARN-10198:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
46s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
21m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 0s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
32s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}180m 4s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerOvercommit |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.8 Server=19.03.8 Image:yetus/hadoop:c44943d1fc3 |
| JIRA Issue | YARN-10198 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12996906/YARN-10198-001.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 5b10f876dd8c 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 1975479 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_242 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-YARN-Build/25700/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/25700/testReport/ |
| Max. process+thread count | 824 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060857#comment-17060857
]
Peter Bacsko commented on YARN-10198:
-
Thanks [~akhilpb].
[~prabhujoseph] could you please review patch 001?
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Attachments: YARN-10198-001.patch
>
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060683#comment-17060683
]
Akhil PB commented on YARN-10198:
-
Expected behavior:
# *[managedParent].%primary_group*
** Should not create queue name *%primary_group* under *[managedParent]* if
leaf queue named *%primary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%primary_group* does not exists.
# *[managedParent].%secondary_group*
** Should not create queue name *%secondary_group* under *[managedParent]* if
leaf queue named *%secondary_group* already exists.
** Should create new child queue under *[managedParent]* if leaf queue named
*%secondary_group* does not exists.
Note that for current static mappings like *u:%user:%primary_group* and
*u:%user:%secondary_group*, following is the behavior and works as expected.
# *u:%user:%primary_group*
** App submitted when leaf queue named *%primary_group*( is present
** App failed when leaf queue named *%primary_group* is not present or stopped
# *u:%user:%secondary_group*
** App submitted when leaf queue named *%secondary_group* is present
** App submitted to "default" queue when leaf queue named *%secondary_group* is
not present
Please share your thoughts if I am missing something.
cc: [~sunilg] [~prabhujoseph] [~pbacsko]
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060409#comment-17060409
]
Peter Bacsko commented on YARN-10198:
-
[~maniraj...@gmail.com] I think so. YARN-9868 introduced a regression. With the
new changes, a leaf queue named {{%primary_group}} has to exist which was not
the case before - if {{managedQueue}} exists with
{{auto-create-child-queue=true}}, it was dynamically created. Since an existing
behavior changed, that part of the patch should be reverted.
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10198) [managedParent].%primary_group mapping rule doesn't work after YARN-9868
[
https://issues.apache.org/jira/browse/YARN-10198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059196#comment-17059196
]
Manikandan R commented on YARN-10198:
-
Above mentioned NULL check has been added to be in line with YARN-9840 changes.
With above check in place, client side thinks there is no queue mapping rule as
ApplicationPlacementContext is NULL. Without above checks,
ApplicationPlacementContext is not null but queue name is NULL. In both cases,
"default" queue would be used.
Should we do this checks only for regular parent queue?
> [managedParent].%primary_group mapping rule doesn't work after YARN-9868
>
>
> Key: YARN-10198
> URL: https://issues.apache.org/jira/browse/YARN-10198
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
>
> YARN-9868 introduced an unnecessary check if we have the following placement
> rule:
> [managedParentQueue].%primary_group
> Here, {{%primary_group}} is expected to be created if it doesn't exist.
> However, there is this validation code which is not necessary:
> {noformat}
> } else if (mapping.getQueue().equals(PRIMARY_GROUP_MAPPING)) {
> if (this.queueManager
> .getQueue(groups.getGroups(user).get(0)) != null) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> } else {
> return null;
> }
> {noformat}
> We should revert this part to the original version:
> {noformat}
> } else if (mapping.queue.equals(PRIMARY_GROUP_MAPPING)) {
> return getPlacementContext(mapping,
> groups.getGroups(user).get(0));
> }
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
