[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16433665#comment-16433665
]
Weiwei Yang commented on YARN-7791:
---
Hi [~sunilg]
I think there is no code needs to be done for this one now. Once
{{RM_PLACEMENT_CONSTRAINTS_HANDLER}} is configured to \{{scheduler}}, PCs will
be submitted to scheduler and handled by \{{AppPlacementAllocator}}, no matter
inter or intra app constraints. Closing this now, please let me know if you
disagrees. Thanks.
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
>Reporter: Weiwei Yang
>Assignee: Sunil G
>Priority: Major
> Labels: distributedshell
> Attachments: YARN-7791-YARN-6592.001.patch
>
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337170#comment-16337170
]
Sunil G commented on YARN-7791:
---
bq.Since by default, it is interpreted as intra-app right ?
Yes. Thats correct. We need not have to rush this patch to get in. Once we have
inter app support etc, we can make use of this. Hence keeping it open and
changing priority.
YARN-7807 will solve current issue.
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
> Environment: Set
> {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
>Reporter: Weiwei Yang
>Assignee: Sunil G
>Priority: Major
> Labels: distributedshell
> Attachments: YARN-7791-YARN-6592.001.patch
>
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16337151#comment-16337151
]
Wangda Tan commented on YARN-7791:
--
[~asuresh] / [~cheersyang], I think I messed up here. Yes, you're correct, now
null targe_allocation_key is interpreted as intra-app. I just created YARN-7807
and uploaded patch, I think we don't need this change for now.
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
> Environment: Set
> {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
>Reporter: Weiwei Yang
>Assignee: Sunil G
>Priority: Major
> Labels: distributedshell
> Attachments: YARN-7791-YARN-6592.001.patch
>
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336868#comment-16336868
]
Arun Suresh commented on YARN-7791:
---
I agree with [~cheersyang] - until we support inter-app, we should assume
intra-app and not need users to specify a type.
I am wondering why the default behavior does not work for the AppAllocator.
Both approaches use the same {{PlacementConstraintUtils#canSatisfy()}} method
right ?
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
> Environment: Set
> {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
>Reporter: Weiwei Yang
>Assignee: Sunil G
>Priority: Major
> Labels: distributedshell
> Attachments: YARN-7791-YARN-6592.001.patch
>
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336857#comment-16336857
]
Arun Suresh commented on YARN-7791:
---
Can you check if this works for the Processor as well ?
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
> Environment: Set
> {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
>Reporter: Weiwei Yang
>Assignee: Sunil G
>Priority: Major
> Labels: distributedshell
> Attachments: YARN-7791-YARN-6592.001.patch
>
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16336634#comment-16336634
]
Wangda Tan commented on YARN-7791:
--
+1, will commit by today if no objections.
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
> Environment: Set
> {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
>Reporter: Weiwei Yang
>Assignee: Sunil G
>Priority: Major
> Labels: distributedshell
> Attachments: YARN-7791-YARN-6592.001.patch
>
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335671#comment-16335671
]
Wangda Tan commented on YARN-7791:
--
I'd prefer to add this option to unblock tests. And we need to support this for
the placement processor approach as well since intra-app is part of the API.
Distributed shell is more like an app for testing YARN functionalities instead
of an app which is heavily used in production, I would not too worry about
usability, I want to get this one committed to unblock test efforts.
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
> Environment: Set
> {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
>Reporter: Weiwei Yang
>Assignee: Sunil G
>Priority: Major
> Labels: distributedshell
> Attachments: YARN-7791-YARN-6592.001.patch
>
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335664#comment-16335664
]
Sunil G commented on YARN-7791:
---
The way I see this is a bit different. We will be supporting inter_app very
soon and more such keys could come. Hence such an option to specify a key will
help w/o worry abt CS supports allocator/algorithm.
Now what you mentioned is correct, we need not have to specify this check for
algorithm. But do u see its better to have a key which could be common in both
case?
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
> Environment: Set
> {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
>Reporter: Weiwei Yang
>Assignee: Sunil G
>Priority: Major
> Labels: distributedshell
> Attachments: YARN-7791-YARN-6592.001.patch
>
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335653#comment-16335653
]
Weiwei Yang commented on YARN-7791:
---
Hi [~sunilg]
As an end user, do I have to specify {{intra_app}} or {{inter_app}} argument
when using placement constraint? I thought we could hide this option from user.
When I use the placement processor approach, it is not required so it is a bit
odd to me that to add this for the allocator approach.
Thanks
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
> Environment: Set
> {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
>Reporter: Weiwei Yang
>Assignee: Sunil G
>Priority: Major
> Labels: distributedshell
> Attachments: YARN-7791-YARN-6592.001.patch
>
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335617#comment-16335617
]
genericqa commented on YARN-7791:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} YARN-6592 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
25s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
14s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
24s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 27s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
30s{color} | {color:green} YARN-6592 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} YARN-6592 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 18s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 11m
11s{color} | {color:green} hadoop-yarn-applications-distributedshell in the
patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 7s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 |
| JIRA Issue | YARN-7791 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907261/YARN-7791-YARN-6592.001.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux be8ee9102c35 3.13.0-135-generic #184-Ubuntu SMP Wed Oct 18
11:55:51 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | YARN-6592 / 904e023 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/19394/testReport/ |
| Max. process+thread count | 640 (vs. ulimit of 5000) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell
|
| Console output |
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335567#comment-16335567
]
Sunil G commented on YARN-7791:
---
Patch was not named correctly for branch, reattaching same
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
> Environment: Set
> {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
>Reporter: Weiwei Yang
>Assignee: Sunil G
>Priority: Major
> Labels: distributedshell
> Attachments: YARN-7791-YARN-6592.001.patch
>
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335564#comment-16335564
]
genericqa commented on YARN-7791:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 4s{color}
| {color:red} YARN-7791 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-7791 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12907259/YARN-7791.001.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/19393/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
> Environment: Set
> {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
>Reporter: Weiwei Yang
>Assignee: Sunil G
>Priority: Major
> Labels: distributedshell
> Attachments: YARN-7791.001.patch
>
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335562#comment-16335562
]
Sunil G commented on YARN-7791:
---
Attaching patch to solve this. I tested this and works fine as expected when
placement allocator is used.
Help for this --placement_spec need to be improved, i ll handle that in another
jira.
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
> Environment: Set
> {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
>Reporter: Weiwei Yang
>Assignee: Sunil G
>Priority: Major
> Labels: distributedshell
> Attachments: YARN-7791.001.patch
>
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335563#comment-16335563
]
Sunil G commented on YARN-7791:
---
cc/ [~cheersyang] [~leftnoteasy] [~asuresh]
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
> Environment: Set
> {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
>Reporter: Weiwei Yang
>Assignee: Sunil G
>Priority: Major
> Labels: distributedshell
> Attachments: YARN-7791.001.patch
>
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335472#comment-16335472
]
Sunil G commented on YARN-7791:
---
Yes [~cheersyang]. I am working on a patch. WIll share a patch after doing some
more tests.
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
> Environment: Set
> {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
>Reporter: Weiwei Yang
>Assignee: Sunil G
>Priority: Major
> Labels: distributedshell
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7791) Support submit intra-app placement constraint in Distributed Shell to AppPlacementAllocator
[
https://issues.apache.org/jira/browse/YARN-7791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16335434#comment-16335434
]
Weiwei Yang commented on YARN-7791:
---
+ [~leftnoteasy], [~sunilg]
> Support submit intra-app placement constraint in Distributed Shell to
> AppPlacementAllocator
> ---
>
> Key: YARN-7791
> URL: https://issues.apache.org/jira/browse/YARN-7791
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: client
> Environment: Set
> {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
>Reporter: Weiwei Yang
>Priority: Major
> Labels: distributedshell
>
> Set {{yarn.resourcemanager.placement-constraints.enabled}} to {{false}}
> Submit a job with placement constraint spec, e.g
> {code}
> in/yarn org.apache.hadoop.yarn.applications.distributedshell.Client -jar
> share/hadoop/yarn/hadoop-yarn-applications-distributedshell-3.1.0-SNAPSHOT.jar
> -shell_command sleep -shell_args 30 -num_containers 2 -master_memory 500
> -container_memory 200 -placement_spec foo=4,NOTIN,NODE,foo
> {code}
> got following errors in RM log
> {noformat}
> Exception message:As of now, the only accepted target key for targetKey of
> allocation_tag target expression is: [yarn_application_label/%intra_app%].
> Please make changes to placement constraints accordingly.
> {noformat}
> Looks like DS needs some modification to support submitting proper scheduling
> requests to app placement allocators.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
