[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662716#comment-14662716 ] Karthik Kambatla commented on YARN-3176: If the child queue is going to be limited by the value set at the parent queue, it seems reasonable to show that value. In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662408#comment-14662408 ] Ashwin Shankar commented on YARN-3176: -- Yes agreed. In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662228#comment-14662228 ] Ashwin Shankar commented on YARN-3176: -- Yeah, thats a problem. Maybe we should display the child queue's maxApp to say Not configured(or something along those lines) in those cases instead of Integer.MAX_VALUE ? Ideally we would want to show maxApps = 50 on the parent queue on the UI, but unfortunately I believe the scheduler page supports Queue status box(the box that shows up when we click on the arrow on the left of a leaf queue) only for leaf queues and not parent queues. In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662122#comment-14662122 ] Siqi Li commented on YARN-3176: --- If we don't allow a queue inherit properties from its parent, I am not sure what is the right way to fix the UI issue. For example, 1. queueMaxAppsDefault is not set 2. parent queue have MaxAppRunning set to 50 3. child queue doesn't have MaxAppRunning set What value should we display for the child queue, should it inherit 50 from its parent? In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662056#comment-14662056 ] Ashwin Shankar commented on YARN-3176: -- There is still the UI issue with maxApp that you brought up. Feel free to fix it in this jira, or raise another jira and mark this as Wont fix. In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[
https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14662391#comment-14662391
]
Siqi Li commented on YARN-3176:
---
From FairSchedulerPage, it looks like the UI has already handle the
Integer.MAX_VALUE case.
{code}
protected void render(Block html) {
ResponseInfo ri = info(\' + qinfo.getQueueName() + \' Queue Status).
_(Used Resources:, qinfo.getUsedResources().toString()).
_(Num Active Applications:, qinfo.getNumActiveApplications()).
_(Num Pending Applications:, qinfo.getNumPendingApplications()).
_(Min Resources:, qinfo.getMinResources().toString()).
_(Max Resources:, qinfo.getMaxResources().toString());
int maxApps = qinfo.getMaxApplications();
if (maxApps Integer.MAX_VALUE) {
ri._(Max Running Applications:, qinfo.getMaxApplications());
}
ri._(STEADY_FAIR_SHARE + :, qinfo.getSteadyFairShare().toString());
ri._(INSTANTANEOUS_FAIR_SHARE + :, qinfo.getFairShare().toString());
html._(InfoBlock.class);
// clear the info contents so this queue's info doesn't accumulate into
another queue's info
ri.clear();
}
{code}
I think the best way to fix the UI is also displaying the information(i.e.
Queue status box) for parent queue. By doing this, there will be no more
confusion
In Fair Scheduler, child queue should inherit maxApp from its parent
Key: YARN-3176
URL: https://issues.apache.org/jira/browse/YARN-3176
Project: Hadoop YARN
Issue Type: Bug
Reporter: Siqi Li
Assignee: Siqi Li
Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch
if the child queue does not have a maxRunningApp limit, it will use the
queueMaxAppsDefault. This behavior is not quite right, since
queueMaxAppsDefault is normally a small number, whereas some parent queues do
have maxRunningApp set to be more than the default
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660787#comment-14660787 ] Siqi Li commented on YARN-3176: --- One example could be queueMaxAppsDefault is set to 50, and parent has a maxRunningApps of 25. But if the child queues of the parent do not have maxRunningApps set, they are going to get 50 as their max instead of 25. In this case, MaxRunningAppEnforcer could end up letting more apps running in the child queues. Also, I don't think there is a direct way to find out the maxRunningApps number of a parent queue from the scheduler UI. In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660760#comment-14660760 ] Ashwin Shankar commented on YARN-3176: -- [~l201514], Just trying to understand, what is the point of inheriting maxApp from parent when it is anyway going to be enforced at the parent by looking at number of applications running at parent(which includes number of applications running in child) ? just because you inherit maxApp at child, that doesn't mean you can always run $maxApp at child, rather it depends on how many apps are running in its siblings, which would be looked at and enforced at the parent. Maybe if you can give an example illustrating your case, that would be helpful. I can see that the UI needs to be fixed. In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660895#comment-14660895 ] Siqi Li commented on YARN-3176: --- Considering an other example, grandparent - parent - child 1. grandparent queue has a MaxRunningApps of 500 2. queueMaxAppsDefault is set to 25 3. parent queue doesn't have MaxAppRunning set 4. child queue has a max of 50 In this case, child queue cannot exceed the 25, since its parent will get the queueMaxAppsDefault as its max instead of the max from grandparent. Although, this kind of queue structure is uncommon, it represents the problem. In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[
https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660859#comment-14660859
]
Ashwin Shankar commented on YARN-3176:
--
bq. In this case, MaxRunningAppEnforcer could end up letting more apps running
in the child queues.
Are you sure about this ? if queueMaxAppsDefault is set to 50 and the child
does not have maxApps set, then their maxApps would be taken as 50, which is
true like you said, but that doesn't mean they can run 50. They wouldn't be
able to run more than 25, which is their parent's maxApps, since every
application submitted (FS.addApplicationAttempt()) has to go through
MaxRunningAppEnforcer.canAppBeRunnable(), which would recursively look and see
if the new app would violate maxApp at any of its queue's ancestors, which
means the child would be limited to 25 apps, not 50, no ?
{code:java}
// Check queue and all parent queues
while (queue != null) {
int queueMaxApps = allocConf.getQueueMaxApps(queue.getName());
if (queue.getNumRunnableApps() = queueMaxApps) {
return false;
}
queue = queue.getParent();
}
{code}
Yes I agree that we do need something to fix the UI, but I don't believe we are
fixing the an incorrect behavior of maxapps logic.
In Fair Scheduler, child queue should inherit maxApp from its parent
Key: YARN-3176
URL: https://issues.apache.org/jira/browse/YARN-3176
Project: Hadoop YARN
Issue Type: Bug
Reporter: Siqi Li
Assignee: Siqi Li
Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch
if the child queue does not have a maxRunningApp limit, it will use the
queueMaxAppsDefault. This behavior is not quite right, since
queueMaxAppsDefault is normally a small number, whereas some parent queues do
have maxRunningApp set to be more than the default
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661251#comment-14661251 ] Siqi Li commented on YARN-3176: --- That makes sense. If you feel like this change is unnecessary, I can mark it as won't fix In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661153#comment-14661153 ] Ashwin Shankar commented on YARN-3176: -- It depends on the use case, if a user has lots of queues, and wants the default value to be take effect, then based on your proposal he has to configure maxRunningApps=default for every one of them, which can be painful. Also I don't think we want to inherit queue properties like minshare, weight which are queue specific properties and won't make sense inheriting. My 2 cents : in the example you described above( grandparent - parent - child), if you think that's a corner case and if its just about configuring one queue, would it work for you to just set maxApp at parent to 50, so that child queue can run 50 apps ? [~ka...@cloudera.com], [~asuresh] thoughts on all of this ? In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[
https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660948#comment-14660948
]
Siqi Li commented on YARN-3176:
---
for that, we can maybe introduce something like a default as follows, which
would let the queue use the default value.
{code}
queue name=queue
maxRunningAppsdefaultmaxRunningApps
/queue
{code}
Also, we need to do the same thing for other queue properties if we agree to
add such queue inherit behavior.
In Fair Scheduler, child queue should inherit maxApp from its parent
Key: YARN-3176
URL: https://issues.apache.org/jira/browse/YARN-3176
Project: Hadoop YARN
Issue Type: Bug
Reporter: Siqi Li
Assignee: Siqi Li
Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch
if the child queue does not have a maxRunningApp limit, it will use the
queueMaxAppsDefault. This behavior is not quite right, since
queueMaxAppsDefault is normally a small number, whereas some parent queues do
have maxRunningApp set to be more than the default
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660925#comment-14660925 ] Ashwin Shankar commented on YARN-3176: -- Yes, this is valid case when inheriting would be useful. I'm just mildly concerned about couple things : what if someone doesn't want to inherit, but wants to use the default. Also since we don't inherit other properties like minshare, would this behavior cause confusion due to inconsistency. In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660651#comment-14660651 ] Siqi Li commented on YARN-3176: --- [~ashwinshankar77], I don't think the sample alloc is a good example. The maxRunningApps for root is usually set to a very high value, since it is the limit of concurrent running apps of the whole cluster. Also, using your sample alloc file still has some problem, the fairScheduler UI would show a Integer.MAX_VALUE as the max running apps of a child. Which is confusing for the user, since MaxRunningAppsEnforcer.canAppBeRunnable() does limit the running apps of a child to be less than their parent. [~asuresh], I could open several related JIRA to broaden the scope of this one, but I think directly inherit parent property is simple and straightforward. Since the real problem is if the queueMaxAppsDefault is not set or set to a large value, the child could end up with more running apps than its parent. In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[
https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660598#comment-14660598
]
Ashwin Shankar commented on YARN-3176:
--
I believe [~ka...@cloudera.com] is right about this jira looking like a
duplicate. Please look at MaxRunningAppsEnforcer.canAppBeRunnable().
[~l201514], you could already do the following for maxApp inheritance(including
having a default value) :
1. Remove queueMaxAppsDefault element from your alloc file.
2. Configure root queue in the alloc file and set maxRunningApps to the value
you used to specify in queueMaxAppsDefault.
3. Configure maxRunningApps at whichever parent you want, which would be
applied at the parent level and to its children which don't have it set.
Now for any child queue, if there is no maxRunningApps set on it, FS would
recursively look at its parent's maxRunningApps and try to enforce it on the
child.
If maxRunningApps is not set for any of the parent node in its ancestry, then
root's maxRunningApps would take effect, which is basically your default
maxRunningApps if that makes sense.
Sample alloc :
{code:xml}
queue name=root
maxRunningApps$defaultValuemaxRunningApps
queue name=parent1
maxRunningApps5maxRunningApps
queue name=childOfParent1 /
/queue
queue name=parent2
queue name=childOfParent2 /
/queue
{code}
In the alloc above, queue childOfParent1 will inherit maxRunningApps(=5) from
parent1, while queue childOfParent2 will inherit maxRunningApps from root
queue.(FYI, root queue and root.default queue are auto-created by Fair
scheduler irrespective of whether user configures it or not)
In Fair Scheduler, child queue should inherit maxApp from its parent
Key: YARN-3176
URL: https://issues.apache.org/jira/browse/YARN-3176
Project: Hadoop YARN
Issue Type: Bug
Reporter: Siqi Li
Assignee: Siqi Li
Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch
if the child queue does not have a maxRunningApp limit, it will use the
queueMaxAppsDefault. This behavior is not quite right, since
queueMaxAppsDefault is normally a small number, whereas some parent queues do
have maxRunningApp set to be more than the default
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660603#comment-14660603 ] Ashwin Shankar commented on YARN-3176: -- Sry, in my comment above I missed a closing /queue element at the end of the alloc xml. In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14660320#comment-14660320 ] Karthik Kambatla commented on YARN-3176: Using the instantaneous fairshare might be reasonable. In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14654130#comment-14654130 ] Arun Suresh commented on YARN-3176: --- Thanks for the patch [~l201514], The patch itself looks fine. But, currently, I see that quite a lot of queue properties do not inherit from the parent. for eg. Min(Max)Resources, Preemption Timeouts etc. Should we broaden the scope of the JIRA to include these as well ? Also, I was also thinking, is it right simply inherit the maxApp ? The queue in question can hog all the apps and not leave its siblings any. Should we use the queue share to determine the max apps ? Thoughts ? In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14653038#comment-14653038 ] Karthik Kambatla commented on YARN-3176: The patch looks reasonable. I forget where exactly, but we do have code that figures out parent queue for a queue that does things similar to what the patch adds. Can we please check that and avoid duplication if possible. [~asuresh] - does the patch look reasonable to you? In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649466#comment-14649466 ] Akira AJISAKA commented on YARN-3176: - +1 pending Jenkins. https://builds.apache.org/job/PreCommit-YARN-Build/8738 In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[
https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14649663#comment-14649663
]
Hadoop QA commented on YARN-3176:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 25s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 8m 26s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 12m 13s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 28s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 59s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 41s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 44s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 55s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 56m 22s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 100m 16s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.yarn.server.resourcemanager.resource.TestResources
|
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12740742/YARN-3176.v2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 93d50b7 |
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8738/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8738/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8738/console |
This message was automatically generated.
In Fair Scheduler, child queue should inherit maxApp from its parent
Key: YARN-3176
URL: https://issues.apache.org/jira/browse/YARN-3176
Project: Hadoop YARN
Issue Type: Bug
Reporter: Siqi Li
Assignee: Siqi Li
Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch
if the child queue does not have a maxRunningApp limit, it will use the
queueMaxAppsDefault. This behavior is not quite right, since
queueMaxAppsDefault is normally a small number, whereas some parent queues do
have maxRunningApp set to be more than the default
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14636244#comment-14636244 ] Joep Rottinghuis commented on YARN-3176: For the record, we're running with this patch in production at Twitter. In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[ https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14596257#comment-14596257 ] Siqi Li commented on YARN-3176: --- Hi [~djp], can you take a look at patch v2. The checkstyle issues and test errors do not seems to apply to this patch In Fair Scheduler, child queue should inherit maxApp from its parent Key: YARN-3176 URL: https://issues.apache.org/jira/browse/YARN-3176 Project: Hadoop YARN Issue Type: Bug Reporter: Siqi Li Assignee: Siqi Li Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch if the child queue does not have a maxRunningApp limit, it will use the queueMaxAppsDefault. This behavior is not quite right, since queueMaxAppsDefault is normally a small number, whereas some parent queues do have maxRunningApp set to be more than the default -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[
https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14594500#comment-14594500
]
Hadoop QA commented on YARN-3176:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 27s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 43s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 51s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 0m 32s | The applied patch generated
10 new checkstyle issues (total was 0, now 10). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 28s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | yarn tests | 51m 7s | Tests failed in
hadoop-yarn-server-resourcemanager. |
| | | 89m 43s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests |
hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12740742/YARN-3176.v2.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 20c03c9 |
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/8297/artifact/patchprocess/diffcheckstylehadoop-yarn-server-resourcemanager.txt
|
| hadoop-yarn-server-resourcemanager test log |
https://builds.apache.org/job/PreCommit-YARN-Build/8297/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/8297/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/8297/console |
This message was automatically generated.
In Fair Scheduler, child queue should inherit maxApp from its parent
Key: YARN-3176
URL: https://issues.apache.org/jira/browse/YARN-3176
Project: Hadoop YARN
Issue Type: Bug
Reporter: Siqi Li
Assignee: Siqi Li
Attachments: YARN-3176.v1.patch, YARN-3176.v2.patch
if the child queue does not have a maxRunningApp limit, it will use the
queueMaxAppsDefault. This behavior is not quite right, since
queueMaxAppsDefault is normally a small number, whereas some parent queues do
have maxRunningApp set to be more than the default
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3176) In Fair Scheduler, child queue should inherit maxApp from its parent
[
https://issues.apache.org/jira/browse/YARN-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316797#comment-14316797
]
Hadoop QA commented on YARN-3176:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12698137/YARN-3176.v1.patch
against trunk revision 22441ab.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/6596//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/6596//console
This message is automatically generated.
In Fair Scheduler, child queue should inherit maxApp from its parent
Key: YARN-3176
URL: https://issues.apache.org/jira/browse/YARN-3176
Project: Hadoop YARN
Issue Type: Bug
Reporter: Siqi Li
Assignee: Siqi Li
Attachments: YARN-3176.v1.patch
if the child queue does not have a maxRunningApp limit, it will use the
queueMaxAppsDefault. This behavior is not quite right, since
queueMaxAppsDefault is normally a small number, whereas some parent queues do
have maxRunningApp set to be more than the default
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
