[
https://issues.apache.org/jira/browse/YARN-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962111#comment-16962111
]
Peter Bacsko commented on YARN-9865:
------------------------------------
Thanks for the update.
I just realized two things:
1) The two testcases are *very* similar, they almost perform the same thing. To
avoid code duplication, we can do the following:
{noformat}
@Test
public void testNestedUserQueueWithPrimaryGroupAsDynamicParentQueue()
throws Exception {
/**
* Mapping order: 1. u:%user:%primary_group.%user 2.
* u:%user:%secondary_group.%user
*
* Expected parent queue is primary group of the user
*/
// set queue mapping
List<UserGroupMappingPlacementRule.QueueMapping> queueMappingsForUG =
new ArrayList<>();
// u:%user:%primary_group.%user
UserGroupMappingPlacementRule.QueueMapping userQueueMapping1 =
new UserGroupMappingPlacementRule.QueueMapping(
UserGroupMappingPlacementRule.QueueMapping.MappingType.USER,
"%user", getQueueMapping("%primary_group", "%user"));
// u:%user:%secondary_group.%user
UserGroupMappingPlacementRule.QueueMapping userQueueMapping2 =
new UserGroupMappingPlacementRule.QueueMapping(
UserGroupMappingPlacementRule.QueueMapping.MappingType.USER,
"%user", getQueueMapping("%secondary_group", "%user"));
queueMappingsForUG.add(userQueueMapping1);
queueMappingsForUG.add(userQueueMapping2);
_testNestedUserQueueWithDynamicParentQueue(queueMappingsForUG, true);
}
@Test
public void testNestedUserQueueWithSecondaryGroupAsDynamicParentQueue()
throws Exception {
/**
* Mapping order: 1. u:%user:%secondary_group.%user 2.
* u:%user:%primary_group.%user
*
* Expected parent queue is secondary group of the user
*/
// set queue mapping
List<UserGroupMappingPlacementRule.QueueMapping> queueMappingsForUG =
new ArrayList<>();
// u:%user:%primary_group.%user
UserGroupMappingPlacementRule.QueueMapping userQueueMapping1 =
new UserGroupMappingPlacementRule.QueueMapping(
UserGroupMappingPlacementRule.QueueMapping.MappingType.USER,
"%user", getQueueMapping("%primary_group", "%user"));
// u:%user:%secondary_group.%user
UserGroupMappingPlacementRule.QueueMapping userQueueMapping2 =
new UserGroupMappingPlacementRule.QueueMapping(
UserGroupMappingPlacementRule.QueueMapping.MappingType.USER,
"%user", getQueueMapping("%secondary_group", "%user"));
queueMappingsForUG.add(userQueueMapping2);
queueMappingsForUG.add(userQueueMapping1);
_testNestedUserQueueWithDynamicParentQueue(queueMappingsForUG, false);
}
private void _testNestedUserQueueWithDynamicParentQueue(
List<UserGroupMappingPlacementRule.QueueMapping> mapping,
boolean primary) throws Exception {
CapacitySchedulerConfiguration conf = new CapacitySchedulerConfiguration();
setupQueueConfiguration(conf);
conf.setClass(YarnConfiguration.RM_SCHEDULER, CapacityScheduler.class,
ResourceScheduler.class);
conf.setClass(CommonConfigurationKeys.HADOOP_SECURITY_GROUP_MAPPING,
SimpleGroupsMapping.class, GroupMappingServiceProvider.class);
List<String> queuePlacementRules = new ArrayList<>();
queuePlacementRules.add(QUEUE_MAPPING_RULE_USER_GROUP);
conf.setQueuePlacementRules(queuePlacementRules);
List<UserGroupMappingPlacementRule.QueueMapping> existingMappingsForUG =
conf.getQueueMappings();
existingMappingsForUG.addAll(mapping);
conf.setQueueMappings(existingMappingsForUG);
// override with queue mappings
conf.setOverrideWithQueueMappings(true);
mockRM = new MockRM(conf);
CapacityScheduler cs = (CapacityScheduler) mockRM.getResourceScheduler();
cs.updatePlacementRules();
mockRM.start();
cs.start();
ApplicationSubmissionContext asc =
Records.newRecord(ApplicationSubmissionContext.class);
asc.setQueue("default");
List<PlacementRule> rules =
cs.getRMContext().getQueuePlacementManager().getPlacementRules();
UserGroupMappingPlacementRule r =
(UserGroupMappingPlacementRule) rules.get(0);
ApplicationPlacementContext ctx = r.getPlacementForApp(asc, "a");
assertEquals("Queue", "a", ctx.getQueue());
if (primary) {
assertEquals("Primary Group", "agroup", ctx.getParentQueue());
} else {
assertEquals("Secondary Group", "asubgroup1", ctx.getParentQueue());
}
mockRM.close();
}
{noformat}
There's also a test called {{testNestedUserQueueWithDynamicParentQueue()}},
isn't it the same as
{{testNestedUserQueueWithPrimaryGroupAsDynamicParentQueue()}}? If so, then
let's remove {{testNestedUserQueueWithDynamicParentQueue()}}.
> Capacity scheduler: add support for combined %user + %secondary_group mapping
> -----------------------------------------------------------------------------
>
> Key: YARN-9865
> URL: https://issues.apache.org/jira/browse/YARN-9865
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Manikandan R
> Assignee: Manikandan R
> Priority: Major
> Attachments: YARN-9865.001.patch, YARN-9865.002.patch,
> YARN-9865.003.patch
>
>
> Similiar to YARN-9841, but for secondary group.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
