[jira] [Commented] (YARN-3764) CapacityScheduler should properly handle moving LeafQueue from one parent to another

[Vinod Kumar Vavilapalli (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571479#comment-14571479
 ] 

Vinod Kumar Vavilapalli commented on YARN-3764:
---

bq. A short term fix is don't allow remove queue under parentQueue.
We never supported removing queues. So this is not just a short-term fix, this 
is the right fix for now.

 CapacityScheduler should properly handle moving LeafQueue from one parent to 
 another
 

 Key: YARN-3764
 URL: https://issues.apache.org/jira/browse/YARN-3764
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Wangda Tan
Assignee: Wangda Tan
Priority: Blocker

 Currently CapacityScheduler doesn't handle the case well, for example:
 A queue structure:
 {code}
 root
   |
   a (100)
 /   \
x y
   (50)   (50)
 {code}
 And reinitialize using following structure:
 {code}
  root
  /   \ 
 (50)a x (50)
 |
 y
(100)
 {code}
 The actual queue structure after reinitialize is:
 {code}
  root
 /\
a (50) x (50)
   /  \
  xy
 (50)  (100)
 {code}
 We should handle this case better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3764) CapacityScheduler should properly handle moving LeafQueue from one parent to another

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571477#comment-14571477
 ] 

Wangda Tan commented on YARN-3764:
--

Following test case can verify this issue:

{code}
  @Test
  public void testQueueParsingWithMoveQueue()
  throws IOException {
YarnConfiguration conf = new YarnConfiguration();
CapacitySchedulerConfiguration csConf =
new CapacitySchedulerConfiguration(conf);
csConf.setQueues(root, new String[] { a });
csConf.setQueues(root.a, new String[] { x, y });
csConf.setCapacity(root.a, 100);
csConf.setCapacity(root.a.x, 50);
csConf.setCapacity(root.a.y, 50);

CapacityScheduler capacityScheduler = new CapacityScheduler();
RMContextImpl rmContext =
new RMContextImpl(null, null, null, null, null, null,
new RMContainerTokenSecretManager(csConf),
new NMTokenSecretManagerInRM(csConf),
new ClientToAMTokenSecretManagerInRM(), null);
rmContext.setNodeLabelManager(nodeLabelManager);
capacityScheduler.setConf(csConf);
capacityScheduler.setRMContext(rmContext);
capacityScheduler.init(csConf);
capacityScheduler.start();

csConf.setQueues(root, new String[] { a, x });
csConf.setQueues(root.a, new String[] { y });
csConf.setCapacity(root.x, 50);
csConf.setCapacity(root.a, 50);
csConf.setCapacity(root.a.y, 100);

capacityScheduler.reinitialize(csConf, rmContext);

Assert.assertEquals(1, ((ParentQueue) capacityScheduler.getQueue(a))
.getChildQueues().size());
  }
{code}

 CapacityScheduler should properly handle moving LeafQueue from one parent to 
 another
 

 Key: YARN-3764
 URL: https://issues.apache.org/jira/browse/YARN-3764
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Wangda Tan
Assignee: Wangda Tan
Priority: Blocker

 Currently CapacityScheduler doesn't handle the case well, for example:
 A queue structure:
 {code}
 root
   |
   a (100)
 /   \
x y
   (50)   (50)
 {code}
 And reinitialize using following structure:
 {code}
  root
  /   \ 
 (50)a x (50)
 |
 y
(100)
 {code}
 The actual queue structure after reinitialize is:
 {code}
  root
 /\
a (50) x (50)
   /  \
  xy
 (50)  (100)
 {code}
 We should handle this case better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (YARN-3764) CapacityScheduler should properly handle moving LeafQueue from one parent to another

[Wangda Tan (JIRA)]

[ 
https://issues.apache.org/jira/browse/YARN-3764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14571473#comment-14571473
 ] 

Wangda Tan commented on YARN-3764:
--

CS's reinitialize logic creates new queues, but only copies configuration 
properties to old queue, and new queue will be discarded after reinitialization.

A comprehensive fix for this is, copy old queue's run time information to new 
queue, including runningApplications, etc. And discard old queue after 
reinitialization.

A short term fix is don't allow remove queue under parentQueue. IAW, CS will 
throw exception if a LeafQueue is moved from one parent to another. I prefer to 
do comprehensive fix for 2.8.0, and short term fix for 2.7.1/2.6.1 (if 
required).

Thoughts?

 CapacityScheduler should properly handle moving LeafQueue from one parent to 
 another
 

 Key: YARN-3764
 URL: https://issues.apache.org/jira/browse/YARN-3764
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Wangda Tan
Assignee: Wangda Tan
Priority: Blocker

 Currently CapacityScheduler doesn't handle the case well, for example:
 A queue structure:
 {code}
 root
   |
   a (100)
 /   \
x y
   (50)   (50)
 {code}
 And reinitialize using following structure:
 {code}
  root
  /   \ 
 (50)a x (50)
 |
 y
(100)
 {code}
 The actual queue structure after reinitialize is:
 {code}
  root
 /\
a (50) x (50)
   /  \
  xy
 (50)  (100)
 {code}
 We should handle this case better.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)