[
https://issues.apache.org/jira/browse/YARN-10506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17263961#comment-17263961
]
zhuqi commented on YARN-10506:
------------------------------
[~wangda] [~gandras]
Thanks [~wangda] for review.
I update a patch for:
According the comment by [~wangda] above:
1. CapacityScheduler
1.1 Fix the not return lq in Capacity.
1.2 Add 008 patch remaining include to fix testautocreateleaf:
As above in 008 patch, Fix autoCreateLeafQueue to use path , the logic change ,
the 009 not include this:
{code:java}
@VisibleForTesting
protected LeafQueue autoCreateLeafQueue(
ApplicationPlacementContext placementContext)
throws IOException, YarnException {
String leafQueueName = placementContext.getQueue();
String parentQueueName = placementContext.getParentQueue();
if (!StringUtils.isEmpty(parentQueueName)) {
CSQueue parentQueue = getQueue(parentQueueName);
if (parentQueue == null) {
throw new SchedulerDynamicEditException(
"Could not auto-create leaf queue for " + leafQueueName
+ ". Queue mapping specifies an invalid parent queue "
+ "which does not exist " + parentQueueName);
}
if (conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) {
// Case 1: Handle ManagedParentQueue
AutoCreatedLeafQueue autoCreatedLeafQueue = null;
ManagedParentQueue autoCreateEnabledParentQueue =
(ManagedParentQueue) parentQueue;
autoCreatedLeafQueue = new AutoCreatedLeafQueue(this, leafQueueName,
autoCreateEnabledParentQueue);
addQueue(autoCreatedLeafQueue);
return autoCreatedLeafQueue;
} ...
}
{code}
2. CapacitySchedulerAutoQueueHandler
2.1 Back create flag in ApplicationPlacementContext, and use it in
autoCreateLeafQueue placement check :
{code:java}
private LeafQueue autoCreateLeafQueue(
ApplicationPlacementContext placementContext)
throws IOException, YarnException {
...
if (!StringUtils.isEmpty(parentQueueName)) {
...
if (conf.isAutoCreateChildQueueEnabled(parentQueue.getQueuePath())) {
...
} else {
ApplicationPlacementContext apc = placementContext.clone();
// Set default true now
// TODO get by queue placement policy
apc.setCreateParentQueue(true);
apc.setCreateLeafQueue(true);
if (placementContext.isCreateLeafQueue()
|| placementContext.isCreateParentQueue()) {
} else {
...
}
}
}
{code}
2.2 -2.3
Remove autoCreateParentHierarchy, the logic change to autoCreateQueuePath. And
the autoCreateQueuePath to autoCreateQueue, and make it atomic, and fix related
test.
{code:java}
public LeafQueue autoCreateQueue(ApplicationPlacementContext queue)
throws YarnException {
...
// The existingParentQueue is already ParentQueue
LeafQueue leafQueue = existingParentQueue.addDynamicLeafQueue(
queue.getFullQueuePath());
queueManager.addQueue(leafQueue.getQueuePath(), leafQueue);
return leafQueue;
}
{code}
3. CapacitySchedulerConfiguration change:
Remaining todo.
4. ParentQueue#addDynamicChildQueue, change the logic handle static queue with
empty childs:
{code:java}
private QueueCapacityType getCapacityConfigurationTypeForQueues(
Collection<CSQueue> queues) throws IOException {
...
else {
if (isDynamicQueue()) {
// If it is an empty dynamic queue, consider weight mode instead
return QueueCapacityType.WEIGHT;
} else {
// Double check queues should be empty
// When static queue with queues empty, return parent ConfigurationType
if (queues.isEmpty()) {
return getCapacityConfigurationTypeForQueues(ImmutableList.of(this));
} else {
throw new
IOException("Unknown case when child queues not empty! " +
"But no child config type!");
}
}
}
{code}
5. Change the submit test to trigger the placement:
{code:java}
@Test
public void testAutoQueueCreationOnAppSubmission() throws Exception {
startScheduler();
createBasicQueueStructureAndValidate();
submitApp(cs, USER0, USER0, "root.e-auto");
AbstractCSQueue e = (AbstractCSQueue) cs.getQueue("root.e-auto");
Assert.assertNotNull(e);
Assert.assertTrue(e.isDynamicQueue());
AbstractCSQueue user0 = (AbstractCSQueue) cs.getQueue("root.e-auto." + USER0);
Assert.assertNotNull(user0);
Assert.assertTrue(user0.isDynamicQueue());
}
{code}
6. Fix clone finding bugs.
Remaining to do
minor things:
# AutoCreateLeafQueue should be moved to autoQueueHandler.
# CSQueueUtils#extractQueuePath should be moved to CSAutoQueueHandler. (It
won't be used by other classes). And rename extractQueuePath to
extractApplicationPlacementContext (we don't just extract path).
# Add test case that static queues with empty childs.
Major things:
CapacitySchedulerConfiguration change.
Thanks.
> Update queue creation logic to use weight mode and allow the flexible
> static/dynamic creation
> ---------------------------------------------------------------------------------------------
>
> Key: YARN-10506
> URL: https://issues.apache.org/jira/browse/YARN-10506
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Benjamin Teke
> Assignee: Andras Gyori
> Priority: Major
> Attachments: YARN-10506-006-10504-010.patch,
> YARN-10506-007-10504-010.patch, YARN-10506-008.patch, YARN-10506.001.patch,
> YARN-10506.002.patch, YARN-10506.003.patch, YARN-10506.004.patch,
> YARN-10506.005.patch, YARN-10506.006-combined.patch, YARN-10506.006.patch,
> YARN-10506.007.patch, YARN-10506.009.patch
>
>
> The queue creation logic should be updated to use weight mode and support the
> flexible creation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
