[jira] [Comment Edited] (YARN-8647) Add a flag to disable move app between queues
[
https://issues.apache.org/jira/browse/YARN-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577397#comment-16577397
]
sarun singla edited comment on YARN-8647 at 8/12/18 3:04 AM:
-
[~abmodi]
Use Case: There are around ~200 + application developers who submit the jobs to
this cluster. Majority of these jobs are submitted as batch users, which is why
we want to disable the feature of move queue on cluster level instead of
disabling for few users. Maybe something like a flag on cluster level.
{code:java}
move_to_queue=false{code}
Please let me know if this explains the use case. cc [~wangda]
was (Author: saruntek):
[~abmodi]
Use Case: There are around ~200 + application developers who submit the jobs to
this cluster. Majority of these jobs are submitted as batch users, which is why
we want to disable the feature of move queue on cluster level instead of
disabling for few users. Maybe something like a flag on cluster level.
{code:java}
move_to_queue=false{code}
Please let me know if this explains the use case.
> Add a flag to disable move app between queues
> -
>
> Key: YARN-8647
> URL: https://issues.apache.org/jira/browse/YARN-8647
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.3
>Reporter: sarun singla
>Assignee: Abhishek Modi
>Priority: Critical
>
> For large clusters where we have a number of users submitting application, we
> can result into scenarios where app developers try to move the queues for
> their applications using something like
> {code:java}
> yarn application -movetoqueue -queue {code}
> Today there is no way of disabling the feature if one does not want
> application developers to use the feature.
> *Solution:*
> We should probably add an option to disable move queue feature from RM side
> on the cluster level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8647) Add a flag to disable move app between queues
[
https://issues.apache.org/jira/browse/YARN-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577397#comment-16577397
]
sarun singla commented on YARN-8647:
[~abmodi]
Use Case: There are around ~200 + application developers who submit the jobs to
this cluster. Majority of these jobs are submitted as batch users, which is why
we want to disable the feature of move queue on cluster level instead of
disabling for few users. Maybe something like a flag on cluster level.
{code:java}
move_to_queue=false{code}
Please let me know if this explains the use case.
> Add a flag to disable move app between queues
> -
>
> Key: YARN-8647
> URL: https://issues.apache.org/jira/browse/YARN-8647
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.3
>Reporter: sarun singla
>Assignee: Abhishek Modi
>Priority: Critical
>
> For large clusters where we have a number of users submitting application, we
> can result into scenarios where app developers try to move the queues for
> their applications using something like
> {code:java}
> yarn application -movetoqueue -queue {code}
> Today there is no way of disabling the feature if one does not want
> application developers to use the feature.
> *Solution:*
> We should probably add an option to disable move queue feature from RM side
> on the cluster level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7953) [GQ] Data structures for federation global queues calculations
[
https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577205#comment-16577205
]
genericqa commented on YARN-7953:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
40s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} YARN-7402 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 28m
28s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
34s{color} | {color:green} YARN-7402 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} YARN-7402 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
35s{color} | {color:green} hadoop-yarn-server-globalpolicygenerator in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
27s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 18s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:ba1ab08 |
| JIRA Issue | YARN-7953 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12935243/YARN-7953-YARN-7402.v8.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml findbugs checkstyle |
| uname | Linux 2f4449bc4236 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-7402 / b8e7180 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_171 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/21575/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 1) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator
|
| Console output |
[jira] [Commented] (YARN-7953) [GQ] Data structures for federation global queues calculations
[ https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577183#comment-16577183 ] Abhishek Modi commented on YARN-7953: - Thanks [~kkaranasos] for review. I have attached v8 patch with fixes for comments. > [GQ] Data structures for federation global queues calculations > -- > > Key: YARN-7953 > URL: https://issues.apache.org/jira/browse/YARN-7953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7953-YARN-7402.v1.patch, > YARN-7953-YARN-7402.v2.patch, YARN-7953-YARN-7402.v3.patch, > YARN-7953-YARN-7402.v4.patch, YARN-7953-YARN-7402.v5.patch, > YARN-7953-YARN-7402.v6.patch, YARN-7953-YARN-7402.v7.patch, > YARN-7953-YARN-7402.v8.patch, YARN-7953.v1.patch > > > This Jira tracks data structures and helper classes used by the core > algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7953) [GQ] Data structures for federation global queues calculations
[ https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhishek Modi updated YARN-7953: Attachment: YARN-7953-YARN-7402.v8.patch > [GQ] Data structures for federation global queues calculations > -- > > Key: YARN-7953 > URL: https://issues.apache.org/jira/browse/YARN-7953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Abhishek Modi >Priority: Major > Attachments: YARN-7953-YARN-7402.v1.patch, > YARN-7953-YARN-7402.v2.patch, YARN-7953-YARN-7402.v3.patch, > YARN-7953-YARN-7402.v4.patch, YARN-7953-YARN-7402.v5.patch, > YARN-7953-YARN-7402.v6.patch, YARN-7953-YARN-7402.v7.patch, > YARN-7953-YARN-7402.v8.patch, YARN-7953.v1.patch > > > This Jira tracks data structures and helper classes used by the core > algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8647) Add a flag to disable move app between queues
[
https://issues.apache.org/jira/browse/YARN-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577081#comment-16577081
]
Abhishek Modi commented on YARN-8647:
-
[~bibinchundatt] I am also thinking on similar lines. It also checks whether
user has permission to submit in the target queue. So, IMO we shouldn't require
another flag to disable it as we are already checking for all the permissions.
[~saruntek] could you please tell the use case that you wanted to target.
> Add a flag to disable move app between queues
> -
>
> Key: YARN-8647
> URL: https://issues.apache.org/jira/browse/YARN-8647
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.3
>Reporter: sarun singla
>Assignee: Abhishek Modi
>Priority: Critical
>
> For large clusters where we have a number of users submitting application, we
> can result into scenarios where app developers try to move the queues for
> their applications using something like
> {code:java}
> yarn application -movetoqueue -queue {code}
> Today there is no way of disabling the feature if one does not want
> application developers to use the feature.
> *Solution:*
> We should probably add an option to disable move queue feature from RM side
> on the cluster level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-8647) Add a flag to disable move app between queues
[
https://issues.apache.org/jira/browse/YARN-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577074#comment-16577074
]
Bibin A Chundatt edited comment on YARN-8647 at 8/11/18 6:38 AM:
-
[~abmodi]
Only application belongs to user or if the user has admin ACL for queue he will
be able to move the application.
Also target queue should have acl for user. Addition flag to disable required??
was (Author: bibinchundatt):
[~abmodi]
Only application belongs to him or if the user has admin ACL for queue he will
be able to move the application.
Also target queue should have acl for user. Addition flag to disable required??
> Add a flag to disable move app between queues
> -
>
> Key: YARN-8647
> URL: https://issues.apache.org/jira/browse/YARN-8647
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.3
>Reporter: sarun singla
>Assignee: Abhishek Modi
>Priority: Critical
>
> For large clusters where we have a number of users submitting application, we
> can result into scenarios where app developers try to move the queues for
> their applications using something like
> {code:java}
> yarn application -movetoqueue -queue {code}
> Today there is no way of disabling the feature if one does not want
> application developers to use the feature.
> *Solution:*
> We should probably add an option to disable move queue feature from RM side
> on the cluster level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8647) Add a flag to disable move app between queues
[
https://issues.apache.org/jira/browse/YARN-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16577074#comment-16577074
]
Bibin A Chundatt commented on YARN-8647:
[~abmodi]
Only application belongs to him or if the user has admin ACL for queue he will
be able to move the application.
Also target queue should have acl for user. Addition flag to disable required??
> Add a flag to disable move app between queues
> -
>
> Key: YARN-8647
> URL: https://issues.apache.org/jira/browse/YARN-8647
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.7.3
>Reporter: sarun singla
>Assignee: Abhishek Modi
>Priority: Critical
>
> For large clusters where we have a number of users submitting application, we
> can result into scenarios where app developers try to move the queues for
> their applications using something like
> {code:java}
> yarn application -movetoqueue -queue {code}
> Today there is no way of disabling the feature if one does not want
> application developers to use the feature.
> *Solution:*
> We should probably add an option to disable move queue feature from RM side
> on the cluster level.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
