[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[
https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158395#comment-16158395
]
Hadoop QA commented on YARN-7107:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
54s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
19s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
37s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
42s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 42s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 59s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 434 unchanged - 0 fixed = 435 total (was 434) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
39s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
22s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
22s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
17s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
30s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 36s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 21s{color}
| {color:red} hadoop-yarn-client in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7107 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12886020/YARN-7107.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 3ed3e37259d9 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
|
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16158318#comment-16158318 ] YunFan Zhou commented on YARN-7107: --- Hi, [~templedf]. I have uploaded my v1 patch. Please help me review the patch. Thank you. > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > Attachments: YARN-7107.001.patch > > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151842#comment-16151842 ] YunFan Zhou commented on YARN-7107: --- Hi, [~templedf] I think we can split the queue into three types: 1. Normal queue(The queue we currently use). 2. Disabled queue(The attributes are exactly the same as you think). 3. Exclusive queue(Make all sibling queues disabled). I think this can perfectly meet my requirements, What do you think? > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > Attachments: YARN-7107.001.preview.patch > > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151841#comment-16151841 ] YunFan Zhou commented on YARN-7107: --- Thank [~templedf] for your review, I benefited a lot from your advice. Your solution is so perfect for: * Do not allocate resources to disabled queue. * The disabled status of the queue can be viewed with the CLI way. Basically, can meet our requirements. But the current solution is more difficult to solve the problem we face in our production environment. Because in our production, we have hundreds of queue, but in the emergency case, we only want to make sure a few queues (because of this few queues be used to produce important reports) can be normally assigned resources. If we use the way you suggest, we have to set the remaining hundreds of queues to be disabled. This makes the operation very cumbersome. We need a better way to solve this problem， what do you think? > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > Attachments: YARN-7107.001.preview.patch > > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[
https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151833#comment-16151833
]
Daniel Templeton commented on YARN-7107:
Thanks for the patch. I love that you started with unit tests. We got off on
a tangent about override queues (my fault) and never got around to discussing
how to implement disabling a queue. To my understanding, disabling a queue
should still allow applications to be submitted to the queue, but it should
prevent them from running, even if the cluster is otherwise idle.
The patch you've posted almost does that, except that the jobs in disabled
queues will run if there's nothing else in the cluster that wants resources.
Instead of turning down the min share and weight (which will also cause
reporting to look odd), I'd add a new property to {{FSQueue}} called
{{disabled}}. In {{FSQueue.assignContainerPreCheck()}}, I'd immediately return
false is {{disabled}} is {{true}}. In {{FSQueue.getQueueInfo()}}, I'd also add
{{disabled}} to the reported state. As a performance optimization, I'd also
modify {{FSParentQueue}} to keep a disabled queues in a list separate from
{{childQueues}}, and then in {{FSParentQueue.getQueueInfo()}}, I'd iterate
through both {{childQueues}} and the list of disabled queues. I think that's
all that would be needed in order to completely shutdown execution from the
disabled queues. (Actually, the {{disabled}} is probably overkill, but it's
nice to have it for reporting.)
What do you think?
> add ability in Fair Scheduler to configure whether disable a queue
> --
>
> Key: YARN-7107
> URL: https://issues.apache.org/jira/browse/YARN-7107
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: YunFan Zhou
>Assignee: YunFan Zhou
> Labels: fairscheduler
> Attachments: YARN-7107.001.preview.patch
>
>
> In a production environment, emergency situations (such as the need to
> calculate the important reports) as soon as possible we need to disable all
> other queues, only allows the *RM* 's resources assigned to emergency queue
> and other queue only at the end of the urgent tasks before allowing them to
> be scheduled properly.
> At present, our approach is to write a script, in the case of an emergency
> manual changes all other queues' *minResources *and *maxResources * to *0mb,
> 0vcores* and then rebase it.This is very troublesome and easy to make
> mistakes.
> So we need to add a configuration in the *FairScheduler* configuration to
> indicate whether the queue is disabled, and if it is disabled, then *RM *will
> not allocate resources to the queue.
> * The child queue will integrate this property of the parent queue.
> * If the child queue is configured with this property, the value of the child
> queue configuration overrides the attributes of the parent queue.
> * The default value of the root queue is *enabled*.
> This will satisfy our needs, and I think other users will encounter such a
> scenario.I think this is very applicable to everyone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[
https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151830#comment-16151830
]
Hadoop QA commented on YARN-7107:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-7107 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7107 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885140/YARN-7107.001.preview.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17272/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> add ability in Fair Scheduler to configure whether disable a queue
> --
>
> Key: YARN-7107
> URL: https://issues.apache.org/jira/browse/YARN-7107
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: YunFan Zhou
>Assignee: YunFan Zhou
> Labels: fairscheduler
> Attachments: YARN-7107.001.preview.patch
>
>
> In a production environment, emergency situations (such as the need to
> calculate the important reports) as soon as possible we need to disable all
> other queues, only allows the *RM* 's resources assigned to emergency queue
> and other queue only at the end of the urgent tasks before allowing them to
> be scheduled properly.
> At present, our approach is to write a script, in the case of an emergency
> manual changes all other queues' *minResources *and *maxResources * to *0mb,
> 0vcores* and then rebase it.This is very troublesome and easy to make
> mistakes.
> So we need to add a configuration in the *FairScheduler* configuration to
> indicate whether the queue is disabled, and if it is disabled, then *RM *will
> not allocate resources to the queue.
> * The child queue will integrate this property of the parent queue.
> * If the child queue is configured with this property, the value of the child
> queue configuration overrides the attributes of the parent queue.
> * The default value of the root queue is *enabled*.
> This will satisfy our needs, and I think other users will encounter such a
> scenario.I think this is very applicable to everyone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151827#comment-16151827 ] YunFan Zhou commented on YARN-7107: --- [~templedf] Please help me review my code, thank you. > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > Attachments: YARN-7107.001.preview.patch > > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[
https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151823#comment-16151823
]
Hadoop QA commented on YARN-7107:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-7107 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7107 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885138/YARN-7107.001.preview.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17271/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> add ability in Fair Scheduler to configure whether disable a queue
> --
>
> Key: YARN-7107
> URL: https://issues.apache.org/jira/browse/YARN-7107
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: YunFan Zhou
>Assignee: YunFan Zhou
> Labels: fairscheduler
> Attachments: YARN-7107.001.preview.patch
>
>
> In a production environment, emergency situations (such as the need to
> calculate the important reports) as soon as possible we need to disable all
> other queues, only allows the *RM* 's resources assigned to emergency queue
> and other queue only at the end of the urgent tasks before allowing them to
> be scheduled properly.
> At present, our approach is to write a script, in the case of an emergency
> manual changes all other queues' *minResources *and *maxResources * to *0mb,
> 0vcores* and then rebase it.This is very troublesome and easy to make
> mistakes.
> So we need to add a configuration in the *FairScheduler* configuration to
> indicate whether the queue is disabled, and if it is disabled, then *RM *will
> not allocate resources to the queue.
> * The child queue will integrate this property of the parent queue.
> * If the child queue is configured with this property, the value of the child
> queue configuration overrides the attributes of the parent queue.
> * The default value of the root queue is *enabled*.
> This will satisfy our needs, and I think other users will encounter such a
> scenario.I think this is very applicable to everyone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[
https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16151518#comment-16151518
]
Hadoop QA commented on YARN-7107:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
31s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
31s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 31s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 24s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 6 new + 68 unchanged - 0 fixed = 74 total (was 68) {color}
|
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
31s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
18s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch
failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 31s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:71bbb86 |
| JIRA Issue | YARN-7107 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12885081/YARN-7107.001.preview.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux a3b535c594ab 3.13.0-119-generic #166-Ubuntu SMP Wed May 3
12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 275980b |
| Default Java | 1.8.0_144 |
| findbugs | v3.1.0-RC1 |
| mvninstall |
https://builds.apache.org/job/PreCommit-YARN-Build/17270/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| compile |
https://builds.apache.org/job/PreCommit-YARN-Build/17270/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| javac |
https://builds.apache.org/job/PreCommit-YARN-Build/17270/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
|
| checkstyle |
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16149258#comment-16149258 ] Daniel Templeton commented on YARN-7107: Go for it. I'm happy to review when you've got a patch. > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[
https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16148376#comment-16148376
]
YunFan Zhou commented on YARN-7107:
---
{noformat}
The trick is how to zero out non-sibling queue without making a mess
{noformat}
Thank [~templedf]. There is no better way to do this, so the implementation
does make the code look messy. So, we just have to zero sibling queue.
I think we've basically figured out what we want to achieve, and if there's no
problem, I can start doing it this weekend.
> add ability in Fair Scheduler to configure whether disable a queue
> --
>
> Key: YARN-7107
> URL: https://issues.apache.org/jira/browse/YARN-7107
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: YunFan Zhou
>Assignee: YunFan Zhou
> Labels: fairscheduler
>
> In a production environment, emergency situations (such as the need to
> calculate the important reports) as soon as possible we need to disable all
> other queues, only allows the *RM* 's resources assigned to emergency queue
> and other queue only at the end of the urgent tasks before allowing them to
> be scheduled properly.
> At present, our approach is to write a script, in the case of an emergency
> manual changes all other queues' *minResources *and *maxResources * to *0mb,
> 0vcores* and then rebase it.This is very troublesome and easy to make
> mistakes.
> So we need to add a configuration in the *FairScheduler* configuration to
> indicate whether the queue is disabled, and if it is disabled, then *RM *will
> not allocate resources to the queue.
> * The child queue will integrate this property of the parent queue.
> * If the child queue is configured with this property, the value of the child
> queue configuration overrides the attributes of the parent queue.
> * The default value of the root queue is *enabled*.
> This will satisfy our needs, and I think other users will encounter such a
> scenario.I think this is very applicable to everyone.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16147551#comment-16147551 ] Daniel Templeton commented on YARN-7107: Good point on min resources. Sounds like a reasonable approach. The trick is how to zero out non-sibling queue without making a mess. > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16146890#comment-16146890 ] YunFan Zhou commented on YARN-7107: --- Thank [~templedf], Here are some attributes I think the "override" queue needs to have: * All queues default are inactive queues unless they become active. * The inactive queue is exactly the same as the way we now use the queue. * If the queue becomes active, it will convert the *minResources* to "*0mb, 0vcores*" and *weight* to 0 of all non-ancestor queues. The reason why I also set *minResource* to 0 because the *weight* of the queue only works when the queue *minResource* is satisfied. But what we expect is that after the queue becomes active, the queue gets as many resources as possible. After the resources for the queue are completely satisfied, then consider assigning resources to non-active queues. Please give me some more suggestion, thank you. > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7107) add ability in Fair Scheduler to configure whether disable a queue
[ https://issues.apache.org/jira/browse/YARN-7107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16144085#comment-16144085 ] Daniel Templeton commented on YARN-7107: +1 on the need to disable queues. I was just saying last week that we need the feature. It sounds to me, though, that what you're trying to do is implement an override queue, which is something we could support more explicitly. We could add a special "override" queue weight that makes all other queues effectively have 0 weight if the override queue is active. Just a thought. > add ability in Fair Scheduler to configure whether disable a queue > -- > > Key: YARN-7107 > URL: https://issues.apache.org/jira/browse/YARN-7107 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: YunFan Zhou >Assignee: YunFan Zhou > Labels: fairscheduler > > In a production environment, emergency situations (such as the need to > calculate the important reports) as soon as possible we need to disable all > other queues, only allows the *RM* 's resources assigned to emergency queue > and other queue only at the end of the urgent tasks before allowing them to > be scheduled properly. > At present, our approach is to write a script, in the case of an emergency > manual changes all other queues' *minResources *and *maxResources * to *0mb, > 0vcores* and then rebase it.This is very troublesome and easy to make > mistakes. > So we need to add a configuration in the *FairScheduler* configuration to > indicate whether the queue is disabled, and if it is disabled, then *RM *will > not allocate resources to the queue. > * The child queue will integrate this property of the parent queue. > * If the child queue is configured with this property, the value of the child > queue configuration overrides the attributes of the parent queue. > * The default value of the root queue is *enabled*. > This will satisfy our needs, and I think other users will encounter such a > scenario.I think this is very applicable to everyone. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
