[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424766#comment-16424766
]
Konstantinos Karanasos commented on YARN-8013:
--
Sure, will commit this now.
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch,
> YARN-8013.006.patch, YARN-8013.007.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424724#comment-16424724
]
Weiwei Yang commented on YARN-8013:
---
Hi [~kkaranasos], thanks. Do you want me to take care of committing this?
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch,
> YARN-8013.006.patch, YARN-8013.007.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423577#comment-16423577
]
Konstantinos Karanasos commented on YARN-8013:
--
+1, thanks [~cheersyang].
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch,
> YARN-8013.006.patch, YARN-8013.007.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423544#comment-16423544
]
Weiwei Yang commented on YARN-8013:
---
Hi [~kkaranasos], does v7 patch look good to you?
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch,
> YARN-8013.006.patch, YARN-8013.007.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423483#comment-16423483
]
genericqa commented on YARN-8013:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
18s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 50 unchanged - 2 fixed = 50 total (was 52) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 6s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
44s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 66m
21s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}149m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8013 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12917292/YARN-8013.007.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 3c8a2995e4b3 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c78cb18 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
|
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423400#comment-16423400
]
Weiwei Yang commented on YARN-8013:
---
Hi [~kkaranasos]
Thanks, I've fixed the checkstyle issues in v7 patch, just uploaded. The
failing UT \{{TestIncreaseAllocationExpirer}} is not caused by this patch, it
runs fine on my laptop (looks like an intermittent failure).
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch,
> YARN-8013.006.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423394#comment-16423394
]
Konstantinos Karanasos commented on YARN-8013:
--
+1, thanks [~cheersyang].
(There seem to be a couple of checkstyle issues and a failing test, but
otherwise +1 from me).
BTW, since you are on this part of the code, I was checking some things today
and found that we allow the application namespace to be null. Then once we
check it, we substitute it with Self as the default in case of null. I think we
should instead make it Self by default in the PlacementConstraints and never
let it be null. This way also simplify the checking code. I just mention it
here in case you are at it -- of course it is not part of this JIRA.
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch,
> YARN-8013.006.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423383#comment-16423383
]
genericqa commented on YARN-8013:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
47s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
18s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
23s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 18s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 4 new + 50 unchanged - 2 fixed = 54 total (was 52) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
46s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 52s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}150m 14s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8013 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12917275/YARN-8013.006.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 3a06611dbe8c 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
|
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423280#comment-16423280
]
Weiwei Yang commented on YARN-8013:
---
Hi [~kkaranasos]
{quote}I would call the class {{TargetApplicationsNamespace}}. How does this
sound?
{quote}
Yes, I think this is a better name. I have renamed the class to this name in v6
patch.
{quote}It is small enough to be folded into the evaluate to be honest, and I
don't think the functionality of this class will be extended. Makes sense?
{quote}
Are you suggesting to remove this class? I prefer not to. Since I have renamed
the class to \{{TargetApplicationsNamespace}}, it looks pretty straightforward
to me to let the class be evaluable by a \{{TargetApplications}}
{code:java}
public abstract class TargetApplicationsNamespace implements
Evaluable
{code}
and \{{TargetApplications}} is used in a few places in UT as well. Second, this
way we keep our interface clean in case we need to extend what a namespace
needs to be eval against. For example, what if we want to support target types
by app? That will end up modifying the eval contract in
\{{TargetApplicationsNamespace}}. Does it make sense?
Uploaded v6 patch.
Thank you.
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16422820#comment-16422820
]
Konstantinos Karanasos commented on YARN-8013:
--
Thanks for the new patch, [~cheersyang].
I think it is cleaner that you call the evaluate inside the tags manager in
this version.
{quote}I was hoping the java doc explains the bit. Any suggested name?
{quote}
Javadoc is totally fine. I would call the class
{{TargetApplicationsNamespace}}. How does this sound?
{quote}I can even remove this class by letting namespace to be eval'd by two
args 1) current appId 2) all appIds.
{quote}
Now I see what this class does, yes. It is small enough to be folded into the
evaluate to be honest, and I don't think the functionality of this class will
be extended. Makes sense?
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421572#comment-16421572
]
Weiwei Yang commented on YARN-8013:
---
Hi [~kkaranasos], please take a look at the latest patch. Thanks
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch, YARN-8013.005.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421171#comment-16421171
]
genericqa commented on YARN-8013:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
23s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 43s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
20s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
18s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 47 unchanged - 2 fixed = 47 total (was 49) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 11s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
45s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m
9s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}148m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8013 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12917079/YARN-8013.005.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 11a61724da6e 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 6590747 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
|
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421128#comment-16421128
]
Weiwei Yang commented on YARN-8013:
---
Hi [~kkaranasos]
Thanks for providing your feedback.
{quote}AllocationTagNamespace is slightly misleading
{quote}
That's the best name I came up with so far, I was hoping the java doc explains
the bit (likely not :( )
{noformat}
Class to describe the namespace of an allocation tag. Each namespace can be
evaluated against a set of applications.After evaluation, the namespace should
have an implicit set of applications which defines its scope.
{noformat}
Any suggested name? The reason I create this class because a tags' scope is
lazy-binding, on the other word, we cannot get the scope during the parsing
phase of a PC expression. That's why we need a eval phase, that can help to
separate different "eval" logic in the different implementation of the
namespace class.
{quote}Why not have a method inside the tags manager that takes an nsScope and
returns only the target applications for that nsScope
{quote}
The best I can do is to move the namespace eval step into allocation tags
manager, that will help to avoid exposing
\{{AllocationTagsManager#getApplicationIdToTags}}. The reason I pass in
\{{AllocationTags}} instead of just target application IDs to #getCardinality
methods was because when namespace is ALL, it needs to use the global mapping,
which means it needs to know the namespace type of the tags.
{quote}It seems that this will simplify the code of {{TargetApplications}} too.
{quote}
I can even remove this class by letting namespace to be eval'd by two args 1)
current appId 2) all appIds. But I thought this is better encapsulated could be
easier for reading. What do you think?
{quote} perAppNodeMappings.keySet().contains(){quote}
Fixed.
Thanks
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16421040#comment-16421040
]
Konstantinos Karanasos commented on YARN-8013:
--
Thanks again for working on this, [~cheersyang].
Patch looks mostly fine, but from a quick look it feels that the
AllocationTagsManager exposes a bit too much information at the moment. So we
have the {{getApplicationIdToTags}} method that returns all existing
applications and their tags (which can be too many). Then we create
{{TargetApplications}} with all this information, although I think the nsScope
is given, right?
Why not have a method inside the tags manager that takes an nsScope and returns
only the target applications for that nsScope, rather than exposing all this?
It seems that this will simplify the code of {{TargetApplications}} too.
Other than that, a couple of minor things:
* AllocationTagNamespace is a slightly misleading name, as it does not reveal
that it essentially defines the applications to which a constraint will refer
* When defining {{getApplicationIdToTags}}:
perAppNodeMappings.keySet().contains() -> perAppNodeMappings.containsKey()
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418099#comment-16418099
]
Konstantinos Karanasos commented on YARN-8013:
--
Thanks [~cheersyang] for working on this and [~leftnoteasy] for the review.
Please give me a day to check this too.
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16417579#comment-16417579
]
Wangda Tan commented on YARN-8013:
--
Thanks [~cheersyang], sounds good. Will wait 2 more days to see if
[~kkaranasos]/[~asuresh] have any other suggestions and answer about:
bq. 3) Not related to the JIRA, but I'm not sure why LocalAllocationTagsManager
extends AllocationTagsManager but still has a ref to AllocationTagsManager.
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch, YARN-8013.004.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416842#comment-16416842
]
genericqa commented on YARN-8013:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
14s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 46 unchanged - 1 fixed = 46 total (was 47) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 58s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
38s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 67m
47s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | YARN-8013 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12916520/YARN-8013.004.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux fe886da0 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20
11:50:30 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a71656c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_151 |
| findbugs | v3.1.0-RC1 |
|
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416736#comment-16416736
]
Weiwei Yang commented on YARN-8013:
---
Hi [~leftnoteasy]
Thanks for the review. I have addressed your #1 and #2 comments in v4 patch.
About #3, maybe [~asuresh] can provide a better answer. My guess is that this
way, RM context can just hold {{AllocationTagsManager}} and only to instantiate
a {{LocalAllocationTagsManager}} when necessary. Note, currently
{{LocalAllocationTagsManager}} is only used in the processor path, not in the
app allocator path. Hope that helps. Thanks.
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16416505#comment-16416505
]
Wangda Tan commented on YARN-8013:
--
Thanks [~cheersyang], comments/questions:
1) getAllApplicationIds => getApplicationIdToTags
2) Java docs doesn't match.
{code}
/**
* Namespace to all applications in the cluster.
*/
public static class AppTag extends AllocationTagNamespace {
{code}
3) Not related to the JIRA, but I'm not sure why LocalAllocationTagsManager
extends AllocationTagsManager but still has a ref to AllocationTagsManager.
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409291#comment-16409291
]
Weiwei Yang commented on YARN-8013:
---
The UT failure is not related to this patch. [~leftnoteasy], [~asuresh],
[~kkaranasos], could you please help to review. Thanks a lot!
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409190#comment-16409190
]
genericqa commented on YARN-8013:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 47s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
12s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s{color} | {color:green} hadoop-yarn-project/hadoop-yarn: The patch generated
0 new + 46 unchanged - 1 fixed = 46 total (was 47) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
40s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 32s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}131m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | YARN-8013 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915593/YARN-8013.003.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 8960752c5598 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision |
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16408961#comment-16408961
]
Weiwei Yang commented on YARN-8013:
---
Forgot to update LocalAllocationTagsManager which caused the UT failure, fixed
in v3 patch.
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch,
> YARN-8013.003.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407968#comment-16407968
]
genericqa commented on YARN-8013:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
53s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
7s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api in
trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
33s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 9s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 46 unchanged - 1 fixed = 47 total (was 47) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
9m 34s{color} | {color:green} patch has no errors when building and testing our
client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
44s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 42s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 33s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.constraint.TestPlacementProcessor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:d4cc50f |
| JIRA Issue | YARN-8013 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12915467/YARN-8013.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux b37acdb00b5a 4.4.0-89-generic #112-Ubuntu SMP Mon Jul 31
19:38:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407788#comment-16407788
]
Weiwei Yang commented on YARN-8013:
---
V2 patch uploaded. The patch basically has following changes
* Renamed {{app-label}} to {{app-tag}}
* Implemented {{AllocationTagNamespace#AppTag}}
* Removed the check in {{SingleConstraintAppPlacementAllocator}} as now we
support all namespace types
* Added tests in {{TestAllocationTagsNamespace}} and
{{TestPlacementConstraintsUtil}}
Once this task is done, next step I will work on YARN-8015 to completely
support inter-app constraints in app allocator. During we should be able to
remove most of the checks in {{SingleConstraintAppPlacementAllocator}} and
delegate all placement constraint checks to {{PlacementConstraintsUtil}}.
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch, YARN-8013.002.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[
https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16392546#comment-16392546
]
Weiwei Yang commented on YARN-8013:
---
Based on the work of YARN-8002, it doesn't need big change to support this.
Attached v1 patch, note this patch will only build after applying the patch in
YARN-8002.
> Support APP-TAG namespace for allocation tags
> -
>
> Key: YARN-8013
> URL: https://issues.apache.org/jira/browse/YARN-8013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
>Priority: Major
> Attachments: YARN-8013.001.patch
>
>
> YARN-1461 adds *Application Tag* concept to Yarn applications, user is able
> to annotate application with multiple tags to classify apps. We can leverage
> this to represent a namespace for a certain group of apps. So instead of
> calling *app-label*, propose to call it *app-tag*.
> A typical use case is,
> There are a lot of TF jobs running on Yarn, and some of them are consuming
> resources heavily. So we want to limit number of PS on each node for such BIG
> players but ignore those SMALL ones. To achieve this, we can do following
> steps:
> # Add application tag "big-tf" to these big TF jobs
> # For each PS request, we add "ps" source tag and map it to constraint
> "{color:#d04437}notin, node, tensorflow/ps{color}" or
> "{color:#d04437}cardinality, node, tensorflow/ps{color}{color:#d04437}, 0,
> 2{color}" for finer grained controls.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8013) Support APP-TAG namespace for allocation tags
[ https://issues.apache.org/jira/browse/YARN-8013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16391109#comment-16391109 ] Weiwei Yang commented on YARN-8013: --- Loop [~leftnoteasy], [~asuresh], [~kkaranasos] for discussion, please let me know your thoughts, thanks! > Support APP-TAG namespace for allocation tags > - > > Key: YARN-8013 > URL: https://issues.apache.org/jira/browse/YARN-8013 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Weiwei Yang >Priority: Major > > YARN-1461 adds *Application Tag* concept to Yarn applications, user is able > to annotate application with multiple tags to classify apps. We can leverage > this to represent a namespace for a certain group of apps. So instead of > calling *app-label*, propose to call it *app-tag*. > A typical use case is, > There are a lot of TF jobs running on Yarn, and some of them are consuming > resources heavily. So we want to limit number of PS on each node for such BIG > players but ignore those SMALL ones. To achieve this, we can do following > steps: > # Add application tag "big-tf" to these big TF jobs > # For each PS request, we add "ps" source tag and map it to constraint > "notin, node, tensorflow/ps" or "cardinality, node, tensorflow/ps, 0, 2" for > finer grained controls. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
