[jira] [Commented] (YARN-6251) Add support for async Container release to prevent deadlock during container updates
[ https://issues.apache.org/jira/browse/YARN-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135556#comment-16135556 ] Wangda Tan commented on YARN-6251: -- [~asuresh], it looks like this patch need rebase after YARN-6979. Could you check? > Add support for async Container release to prevent deadlock during container > updates > > > Key: YARN-6251 > URL: https://issues.apache.org/jira/browse/YARN-6251 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6251.001.patch, YARN-6251.002.patch, > YARN-6251.003.patch, YARN-6251.004.patch, YARN-6251.005.patch, > YARN-6251.006.patch > > > Opening to track a locking issue that was uncovered when running a custom SLS > AMSimulator. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135553#comment-16135553
]
ASF GitHub Bot commented on YARN-2162:
--
Github user flyrain commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134298360
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
---
@@ -158,26 +158,33 @@ public Resource getMinShare() {
return minShare;
}
- public void setMaxShare(Resource maxShare){
+ public void setMaxShare(ResourceConfiguration maxShare){
this.maxShare = maxShare;
}
+ @Override
+ public Resource getMaxShare() {
+Resource maxResource =
maxShare.getResource(scheduler.getClusterResource());
+// Set max resource to min resource if min resource is greater than max
+// resource
+if(Resources.greaterThan(scheduler.getResourceCalculator(),
+scheduler.getClusterResource(), minShare, maxResource)) {
--- End diff --
Didn't see the formatting issue, but will reformat in next patch anyway to
make sure it is right.
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135547#comment-16135547
]
ASF GitHub Bot commented on YARN-2162:
--
Github user flyrain commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134297962
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
---
@@ -343,10 +343,10 @@ public void updateDemand() {
readLock.unlock();
}
// Cap demand to maxShare to limit allocation to maxShare
-demand = Resources.componentwiseMin(tmpDemand, maxShare);
+demand = Resources.componentwiseMin(tmpDemand, getMaxShare());
--- End diff --
Yes. It is verified. Besides, FSQueue#maxShare is private now.
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6979) Add flag to notify all types of container updates to NM via NodeHeartbeatResponse
[
https://issues.apache.org/jira/browse/YARN-6979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135546#comment-16135546
]
Wangda Tan commented on YARN-6979:
--
[~asuresh], is there any compatibility issue of the patch? I saw some protocols
are added since 2.8.0. See YARN-1644.
> Add flag to notify all types of container updates to NM via
> NodeHeartbeatResponse
> -
>
> Key: YARN-6979
> URL: https://issues.apache.org/jira/browse/YARN-6979
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Arun Suresh
>Assignee: kartheek muthyala
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6979.001.patch, YARN-6979.002.patch,
> YARN-6979.003.patch, YARN-6979.addendum-001.patch
>
>
> Currently, only the Container Resource decrease command is sent to the NM via
> NodeHeartbeat response. This JIRA proposes to add a flag in the RM to allow
> ALL container updates (increase, decrease, promote and demote) to be
> initiated via node HB.
> The AM is still free to use the ContainerManagementPrototol's
> {{updateContainer}} API in cases where for instance, the Node HB frequency is
> very low and the AM needs to update the container as soon as possible. In
> these situations, if the Node HB arrives before the updateContainer API call,
> the call would error out, due to a version mismatch and the AM is required to
> handle it.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135536#comment-16135536
]
ASF GitHub Bot commented on YARN-2162:
--
Github user flyrain commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134296734
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java
---
@@ -321,8 +310,7 @@ Resource getMaxResources(String queue) {
* @param queue the target queue's name
* @return the max allocation on this queue or null if not set
*/
- @VisibleForTesting
- Resource getMaxChildResources(String queue) {
+ @VisibleForTesting ResourceConfiguration getMaxChildResources(String
queue) {
--- End diff --
Will do.
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7043) Cleanup ResourceProfileManager
[ https://issues.apache.org/jira/browse/YARN-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135532#comment-16135532 ] Wangda Tan commented on YARN-7043: -- Thanks [~sunilg]. [~templedf], could you check the latest patch? I plan to commit it today. > Cleanup ResourceProfileManager > -- > > Key: YARN-7043 > URL: https://issues.apache.org/jira/browse/YARN-7043 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-7043.YARN-3926.001.patch, > YARN-7043.YARN-3926.002.patch, YARN-7043.YARN-3926.003.patch, > YARN-7043.YARN-3926.004.patch > > > Several cleanups we can do for ResourceProfileManager: > 1) Move GetResourceTypesInfo from profile manager to ResourceUtils. > 2) Move logics to check profile enabled, etc. from ClientRMService to > ResourceUtils. > 3) Throw exception when resource profile is disabled and method accessed by > other modules. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7053) Move curator transaction support to ZKCuratorManager
[ https://issues.apache.org/jira/browse/YARN-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135523#comment-16135523 ] Jonathan Hung commented on YARN-7053: - 002 patch fixes checkstyle > Move curator transaction support to ZKCuratorManager > > > Key: YARN-7053 > URL: https://issues.apache.org/jira/browse/YARN-7053 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-7053.001.patch, YARN-7053.002.patch > > > HADOOP-14741 moves curator functionality to ZKCuratorManager. ZKRMStateStore > has some curator transaction support which can be reused, so this can be > moved to ZKCuratorManager as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7053) Move curator transaction support to ZKCuratorManager
[ https://issues.apache.org/jira/browse/YARN-7053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-7053: Attachment: YARN-7053.002.patch > Move curator transaction support to ZKCuratorManager > > > Key: YARN-7053 > URL: https://issues.apache.org/jira/browse/YARN-7053 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung > Attachments: YARN-7053.001.patch, YARN-7053.002.patch > > > HADOOP-14741 moves curator functionality to ZKCuratorManager. ZKRMStateStore > has some curator transaction support which can be reused, so this can be > moved to ZKCuratorManager as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135420#comment-16135420
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
18s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 13 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 4m
7s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
51s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
12s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
5s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
44s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 1m 44s{color} |
{color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 44s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 49s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 1 new + 24 unchanged - 3 fixed = 25 total (was 27) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
51s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
0s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 62m 17s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 53s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
36s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}116m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6623 |
| JIRA Patch URL |
[jira] [Commented] (YARN-7058) Add null check in AMRMClientImpl#getMatchingRequest
[
https://issues.apache.org/jira/browse/YARN-7058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135415#comment-16135415
]
Hadoop QA commented on YARN-7058:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} branch-2 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
13s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} branch-2 passed with JDK v1.7.0_151 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} branch-2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} branch-2 passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
16s{color} | {color:green} branch-2 passed with JDK v1.7.0_151 {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 14s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 9 new +
61 unchanged - 9 fixed = 70 total (was 70) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
12s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed with JDK v1.7.0_151 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}109m 16s{color}
| {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_151.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}144m 55s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.7.0_151 Timed out junit tests |
org.apache.hadoop.yarn.client.TestRMFailover |
| | org.apache.hadoop.yarn.client.api.impl.TestDistributedScheduling |
| | org.apache.hadoop.yarn.client.cli.TestYarnCLI |
| | org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA |
| | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient |
| | org.apache.hadoop.yarn.client.api.impl.TestYarnClient |
| | org.apache.hadoop.yarn.client.api.impl.TestNMClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:5e40efe |
| JIRA Issue | YARN-7058 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882881/YARN-7058-branch-2.001.patch
|
| Optional Tests | asflicense compile
[jira] [Commented] (YARN-6861) Reader API for sub application entities
[ https://issues.apache.org/jira/browse/YARN-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135414#comment-16135414 ] Varun Saxena commented on YARN-6861: Thanks [~rohithsharma] for your contribution and [~vrushalic] for reviews. > Reader API for sub application entities > --- > > Key: YARN-6861 > URL: https://issues.apache.org/jira/browse/YARN-6861 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: atsv2-subapp, yarn-5355-merge-blocker > Fix For: YARN-5355, YARN-5355-branch-2 > > Attachments: YARN-6861-YARN-5355.001.patch, > YARN-6861-YARN-5355.002.patch, YARN-6861-YARN-5355.003.patch, > YARN-6861-YARN-5355.004.patch, YARN-6861-YARN-5355.005.patch, > YARN-6861-YARN-5355.006.patch, YARN-6861-YARN-5355_branch2.001.patch > > > YARN-6733 and YARN-6734 writes data into sub application table. There should > be a way to read those entities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6861) Reader API for sub application entities
[ https://issues.apache.org/jira/browse/YARN-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135413#comment-16135413 ] Varun Saxena commented on YARN-6861: Pushed it to YARN-5355_branch2 too. Test failure is unrelated and YARN-7063 tracks it. > Reader API for sub application entities > --- > > Key: YARN-6861 > URL: https://issues.apache.org/jira/browse/YARN-6861 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: atsv2-subapp, yarn-5355-merge-blocker > Attachments: YARN-6861-YARN-5355.001.patch, > YARN-6861-YARN-5355.002.patch, YARN-6861-YARN-5355.003.patch, > YARN-6861-YARN-5355.004.patch, YARN-6861-YARN-5355.005.patch, > YARN-6861-YARN-5355.006.patch, YARN-6861-YARN-5355_branch2.001.patch > > > YARN-6733 and YARN-6734 writes data into sub application table. There should > be a way to read those entities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6992) "Kill application" button is present even if the application is FINISHED in RM UI
[ https://issues.apache.org/jira/browse/YARN-6992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135393#comment-16135393 ] Sunil G commented on YARN-6992: --- It seems like only final state are validated in UI/client end. No issues, lets keep the same states (KILLED, FAILED and FINISHED) > "Kill application" button is present even if the application is FINISHED in > RM UI > - > > Key: YARN-6992 > URL: https://issues.apache.org/jira/browse/YARN-6992 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Sumana Sathish >Assignee: Suma Shivaprasad > Attachments: YARN-6992.001.patch > > > Kill button should not be displayed for FAILED, KILLED and FINISHED apps in > Application specific landing page -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6323) Rolling upgrade/config change is broken on timeline v2.
[ https://issues.apache.org/jira/browse/YARN-6323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135383#comment-16135383 ] Vrushali C commented on YARN-6323: -- Thanks [~haibo.chen] , please go ahead with the commit. > Rolling upgrade/config change is broken on timeline v2. > > > Key: YARN-6323 > URL: https://issues.apache.org/jira/browse/YARN-6323 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Li Lu >Assignee: Vrushali C > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: YARN-6323.001.patch, YARN-6323-YARN-5355.0001.patch, > YARN-6323-YARN-5355.0002.patch > > > Found this issue when deploying on real clusters. If there are apps running > when we enable timeline v2 (with work preserving restart enabled), node > managers will fail to start due to missing app context data. We should > probably assign some default names to these "left over" apps. I believe it's > suboptimal to let users clean up the whole cluster before enabling timeline > v2. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6861) Reader API for sub application entities
[ https://issues.apache.org/jira/browse/YARN-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135378#comment-16135378 ] Vrushali C commented on YARN-6861: -- Filed YARN-7063 for branch2. [~varun_saxena] please feel free to resolve jira > Reader API for sub application entities > --- > > Key: YARN-6861 > URL: https://issues.apache.org/jira/browse/YARN-6861 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: atsv2-subapp, yarn-5355-merge-blocker > Attachments: YARN-6861-YARN-5355.001.patch, > YARN-6861-YARN-5355.002.patch, YARN-6861-YARN-5355.003.patch, > YARN-6861-YARN-5355.004.patch, YARN-6861-YARN-5355.005.patch, > YARN-6861-YARN-5355.006.patch, YARN-6861-YARN-5355_branch2.001.patch > > > YARN-6733 and YARN-6734 writes data into sub application table. There should > be a way to read those entities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-7063) branch2 for tsv2 has NoClassDefFound exception for AuthenticationWithProxyUserFilter
Vrushali C created YARN-7063:
Summary: branch2 for tsv2 has NoClassDefFound exception for
AuthenticationWithProxyUserFilter
Key: YARN-7063
URL: https://issues.apache.org/jira/browse/YARN-7063
Project: Hadoop YARN
Issue Type: Bug
Components: timelineserver
Reporter: Vrushali C
Seeing NoClassDefFound on the branch2 at runtime
Stack trace
{code}
java.lang.NoClassDefFoundError:
org/apache/hadoop/security/AuthenticationWithProxyUserFilter
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at
org.apache.hadoop.yarn.server.timelineservice.reader.security.TimelineReaderAuthenticationFilterInitializer.initFilter(TimelineReaderAuthenticationFilterInitializer.java:49)
at
org.apache.hadoop.http.HttpServer2.initializeWebServer(HttpServer2.java:393)
at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:344)
at org.apache.hadoop.http.HttpServer2.(HttpServer2.java:104)
at
org.apache.hadoop.http.HttpServer2$Builder.build(HttpServer2.java:292)
at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.startTimelineReaderWebApp(TimelineReaderServer.java:181)
at
org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderServer.serviceStart(TimelineReaderServer.java:124)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.yarn.server.timelineservice.reader.AbstractTimelineReaderHBaseTestBase.initialize(AbstractTimelineReaderHBaseTestBase.java:91)
at
org.apache.hadoop.yarn.server.timelineservice.reader.TestTimelineReaderWebServicesHBaseStorage.setupBeforeClass(TestTimelineReaderWebServicesHBaseStorage.java:79)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[ https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135376#comment-16135376 ] Jason Lowe commented on YARN-6930: -- I'm not hung up on whether we go full-blown enums, but I do think we should not do "semi-reflection" with the base class names. IMHO we should either do a full plugin interface with reflection or not use reflection at all. Maybe others feel differently. bq. When you say "list the ones we know about here", can you elaborate on what you mean? Sorry, I meant in the property description for yarn.nodemanager.runtime.linux.allowed-runtimes in yarn-default, which last I checked was used to automatically generate the property documentation in the Hadoop docs. If a user stumbles across this property, the description is not going to be sufficient to know what valid values are allowed. > Admins should be able to explicitly enable specific LinuxContainerRuntime in > the NodeManager > > > Key: YARN-6930 > URL: https://issues.apache.org/jira/browse/YARN-6930 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Shane Kumpf > Attachments: YARN-6930.001.patch, YARN-6930.002.patch, > YARN-6930.003.patch > > > Today, in the java land, all LinuxContainerRuntimes are always enabled when > using LinuxContainerExecutor and the user can simply invoke anything that > he/she wants - default, docker, java-sandbox. > We should have a way for admins to explicitly enable only specific runtimes > that he/she decides for the cluster. And by default, we should have > everything other than the default one disabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135349#comment-16135349
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r133308580
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/ResourceConfiguration.java
---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceType;
+
+import java.util.HashMap;
+
+/**
+ * A ResourceConfiguration represents an entity that is used to
configuration
+ * resources, such as maximum resources of a queue. It can be percentage of
+ * cluster resources or an absolute value.
+ */
+@Private
+@Unstable
+public class ResourceConfiguration {
+ private final boolean isPercentage;
+ private final Resource resource;
+ private final HashMap percentages;
+
+ public boolean isPercentage() {
--- End diff --
Why do we need this method?
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[ https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135340#comment-16135340 ] ASF GitHub Bot commented on YARN-2162: -- Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/261#discussion_r134268440 --- Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md --- @@ -85,9 +85,9 @@ The allocation file must be in XML format. The format contains five types of ele * **minResources**: minimum resources the queue is entitled to, in the form "X mb, Y vcores". For the single-resource fairness policy, the vcores value is ignored. If a queue's minimum share is not satisfied, it will be offered available resources before any other queue under the same parent. Under the single-resource fairness policy, a queue is considered unsatisfied if its memory usage is below its minimum memory share. Under dominant resource fairness, a queue is considered unsatisfied if its usage for its dominant resource with respect to the cluster capacity is below its minimum share for that resource. If multiple queues are unsatisfied in this situation, resources go to the queue with the smallest ratio between relevant resource usage and minimum. Note that it is possible that a queue that is below its minimum may not immediately get up to its minimum when it submits an application, because already-running jobs may be using those resources. -* **maxResources**: maximum resources a queue is allowed, in the form "X mb, Y vcores". A queue will never be assigned a container that would put its aggregate usage over this limit. +* **maxResources**: maximum resources a queue is allowed, in the form "X mb, Y vcores". A queue will never be assigned a container that would put its aggregate usage over this limit. Set it to a percentage of all cluster resources in form of "50%" or "50% memory, 60% vcore". -* **maxChildResources**: maximum resources an ad hoc child queue is allowed, in the form "X mb, Y vcores". Any ad hoc queue that is a direct child of a queue with this property set will have it's maxResources property set accordingly. +* **maxChildResources**: maximum resources an ad hoc child queue is allowed, in the form "X mb, Y vcores". Any ad hoc queue that is a direct child of a queue with this property set will have it's maxResources property set accordingly. Set it to a percentage of all cluster resources in form of "50%" or "50% memory, 60% vcore". --- End diff -- Similar to the above config? > add ability in Fair Scheduler to optionally configure maxResources in terms > of percentage > - > > Key: YARN-2162 > URL: https://issues.apache.org/jira/browse/YARN-2162 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler, scheduler >Reporter: Ashwin Shankar >Assignee: Yufei Gu > Labels: scheduler > Attachments: YARN-2162.001.patch, YARN-2162.002.patch > > > minResources and maxResources in fair scheduler configs are expressed in > terms of absolute numbers X mb, Y vcores. > As a result, when we expand or shrink our hadoop cluster, we need to > recalculate and change minResources/maxResources accordingly, which is pretty > inconvenient. > We can circumvent this problem if we can optionally configure these > properties in terms of percentage of cluster capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135352#comment-16135352
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134265745
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestAllocationFileLoaderService.java
---
@@ -235,21 +235,21 @@ public void testAllocationFileParsing() throws
Exception {
queueConf.getMinResources("root." +
YarnConfiguration.DEFAULT_QUEUE_NAME));
assertEquals(Resources.createResource(2048, 10),
-queueConf.getMaxResources("root.queueA"));
+queueConf.getMaxResources("root.queueA").getResource());
assertEquals(Resources.createResource(5120, 110),
-queueConf.getMaxResources("root.queueB"));
-assertEquals(Resources.createResource(5120, 0),
-queueConf.getMaxResources("root.queueC"));
+queueConf.getMaxResources("root.queueB").getResource());
+assertEquals(Resources.createResource(4096, 100),
--- End diff --
Why are we having to change this value here? Shouldn't the behavior be the
same?
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135346#comment-16135346
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r133308854
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/ResourceConfiguration.java
---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceType;
+
+import java.util.HashMap;
+
+/**
+ * A ResourceConfiguration represents an entity that is used to
configuration
+ * resources, such as maximum resources of a queue. It can be percentage of
+ * cluster resources or an absolute value.
+ */
+@Private
+@Unstable
+public class ResourceConfiguration {
+ private final boolean isPercentage;
+ private final Resource resource;
+ private final HashMap percentages;
+
+ public boolean isPercentage() {
+return isPercentage;
+ }
+
+ public ResourceConfiguration(HashMap percentages) {
+isPercentage = true;
+this.percentages = percentages;
+this.resource = null;
+ }
+
+ public ResourceConfiguration(Resource resource) {
+isPercentage = false;
+this.percentages = null;
+this.resource = resource;
+ }
+
+ public Resource getResource(Resource clusterResource) {
--- End diff --
I think there should be a single getResource method. I am divided between
explicitly passing clusterResource as an arg vs passing FSContext or some such
that you could get the cluster resource from lazily if required.
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135339#comment-16135339
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134264187
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
---
@@ -605,13 +606,15 @@ private void loadQueue(String parentName, Element
element,
queueAcls.put(queueName, acls);
resAcls.put(queueName, racls);
-if (maxQueueResources.containsKey(queueName) &&
-minQueueResources.containsKey(queueName)
-&& !Resources.fitsIn(minQueueResources.get(queueName),
-maxQueueResources.get(queueName))) {
+ResourceConfiguration maxResourceConf =
maxQueueResources.get(queueName);
+if (maxResourceConf != null &&
+!maxResourceConf.isPercentage() &&
--- End diff --
A warning is necessary even when the config is percentage-based. I would
argue the warning is more important in this case than the other.
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135345#comment-16135345
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134264441
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSLeafQueue.java
---
@@ -343,10 +343,10 @@ public void updateDemand() {
readLock.unlock();
}
// Cap demand to maxShare to limit allocation to maxShare
-demand = Resources.componentwiseMin(tmpDemand, maxShare);
+demand = Resources.componentwiseMin(tmpDemand, getMaxShare());
--- End diff --
Nice. Have we verified we don't access maxShare directly anywhere else? How
about other classes that have similar field?
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135338#comment-16135338
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r133308539
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/ResourceConfiguration.java
---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceType;
+
+import java.util.HashMap;
+
+/**
+ * A ResourceConfiguration represents an entity that is used to
configuration
+ * resources, such as maximum resources of a queue. It can be percentage of
+ * cluster resources or an absolute value.
+ */
+@Private
+@Unstable
+public class ResourceConfiguration {
+ private final boolean isPercentage;
--- End diff --
Do we need this boolean?
If both are set, maybe there should be a strict priority order? e.g. if
resource is set, use it. Otherwise, use percentages?
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135353#comment-16135353
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r133316267
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationFileLoaderService.java
---
@@ -495,17 +496,17 @@ private void loadQueue(String parentName, Element
element,
Element field = (Element) fieldNode;
if ("minResources".equals(field.getTagName())) {
String text = ((Text)field.getFirstChild()).getData().trim();
-Resource val =
+ResourceConfiguration val =
--- End diff --
Is it reasonable to use ResourceConfiguration for minResources? What are
the pros and cons?
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[ https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135351#comment-16135351 ] ASF GitHub Bot commented on YARN-2162: -- Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/261#discussion_r134268363 --- Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/FairScheduler.md --- @@ -85,9 +85,9 @@ The allocation file must be in XML format. The format contains five types of ele * **minResources**: minimum resources the queue is entitled to, in the form "X mb, Y vcores". For the single-resource fairness policy, the vcores value is ignored. If a queue's minimum share is not satisfied, it will be offered available resources before any other queue under the same parent. Under the single-resource fairness policy, a queue is considered unsatisfied if its memory usage is below its minimum memory share. Under dominant resource fairness, a queue is considered unsatisfied if its usage for its dominant resource with respect to the cluster capacity is below its minimum share for that resource. If multiple queues are unsatisfied in this situation, resources go to the queue with the smallest ratio between relevant resource usage and minimum. Note that it is possible that a queue that is below its minimum may not immediately get up to its minimum when it submits an application, because already-running jobs may be using those resources. -* **maxResources**: maximum resources a queue is allowed, in the form "X mb, Y vcores". A queue will never be assigned a container that would put its aggregate usage over this limit. +* **maxResources**: maximum resources a queue is allowed, in the form "X mb, Y vcores". A queue will never be assigned a container that would put its aggregate usage over this limit. Set it to a percentage of all cluster resources in form of "50%" or "50% memory, 60% vcore". --- End diff -- How about: **maxResources**: maximum resources a queue is allocated, expressed either in absolute values (X mb, Y vcores) or as a percentage of the cluster resources (X% memory, Y% cpu). A queue will not be assigned a container that would put its aggregate usage over this limit. > add ability in Fair Scheduler to optionally configure maxResources in terms > of percentage > - > > Key: YARN-2162 > URL: https://issues.apache.org/jira/browse/YARN-2162 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler, scheduler >Reporter: Ashwin Shankar >Assignee: Yufei Gu > Labels: scheduler > Attachments: YARN-2162.001.patch, YARN-2162.002.patch > > > minResources and maxResources in fair scheduler configs are expressed in > terms of absolute numbers X mb, Y vcores. > As a result, when we expand or shrink our hadoop cluster, we need to > recalculate and change minResources/maxResources accordingly, which is pretty > inconvenient. > We can circumvent this problem if we can optionally configure these > properties in terms of percentage of cluster capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135350#comment-16135350
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134269384
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java
---
@@ -287,27 +289,69 @@ public float getReservableNodes() {
*
* @throws AllocationConfigurationException
*/
- public static Resource parseResourceConfigValue(String val)
+ public static ResourceConfiguration parseResourceConfigValue(String val)
throws AllocationConfigurationException {
+ResourceConfiguration resourceConfiguration;
try {
val = StringUtils.toLowerCase(val);
- int memory = findResource(val, "mb");
- int vcores = findResource(val, "vcores");
- return BuilderUtils.newResource(memory, vcores);
+ if (val.contains("%")) {
+resourceConfiguration = new ResourceConfiguration(
+getResourcePercentage(val));
+ } else {
+int memory = findResource(val, "mb");
+int vcores = findResource(val, "vcores");
+resourceConfiguration = new ResourceConfiguration(
+BuilderUtils.newResource(memory, vcores));
+ }
} catch (AllocationConfigurationException ex) {
throw ex;
} catch (Exception ex) {
throw new AllocationConfigurationException(
"Error reading resource config", ex);
}
+return resourceConfiguration;
+ }
+
+ private static HashMap getResourcePercentage(
+ String val) throws AllocationConfigurationException {
+HashMap resourcePercentage = new HashMap<>();
+String[] strings = val.split(",");
+if (strings.length == 1) {
+ double percentage = findPercentage(strings[0], "");
+ for (ResourceType resourceType: ResourceType.values()) {
+resourcePercentage.put(resourceType, percentage/100);
+ }
+} else {
+ double memPercentage = findPercentage(val, "memory");
+ double vcorePercentage = findPercentage(val, "vcore");
--- End diff --
vcore is a unit of cpu. If we are using memory instead of mb, we should use
cpu instead of vcore.
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135344#comment-16135344
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r133314099
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java
---
@@ -321,8 +310,7 @@ Resource getMaxResources(String queue) {
* @param queue the target queue's name
* @return the max allocation on this queue or null if not set
*/
- @VisibleForTesting
- Resource getMaxChildResources(String queue) {
+ @VisibleForTesting ResourceConfiguration getMaxChildResources(String
queue) {
--- End diff --
@VisibleForTesting should be in the previous line.
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[ https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135348#comment-16135348 ] ASF GitHub Bot commented on YARN-2162: -- Github user kambatla commented on a diff in the pull request: https://github.com/apache/hadoop/pull/261#discussion_r133307097 --- Diff: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/ResourceConfiguration.java --- @@ -0,0 +1,71 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair; + +import org.apache.hadoop.classification.InterfaceAudience.Private; +import org.apache.hadoop.classification.InterfaceStability.Unstable; +import org.apache.hadoop.yarn.api.records.Resource; +import org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceType; + +import java.util.HashMap; + +/** + * A ResourceConfiguration represents an entity that is used to configuration --- End diff -- s/configuration/configure > add ability in Fair Scheduler to optionally configure maxResources in terms > of percentage > - > > Key: YARN-2162 > URL: https://issues.apache.org/jira/browse/YARN-2162 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler, scheduler >Reporter: Ashwin Shankar >Assignee: Yufei Gu > Labels: scheduler > Attachments: YARN-2162.001.patch, YARN-2162.002.patch > > > minResources and maxResources in fair scheduler configs are expressed in > terms of absolute numbers X mb, Y vcores. > As a result, when we expand or shrink our hadoop cluster, we need to > recalculate and change minResources/maxResources accordingly, which is pretty > inconvenient. > We can circumvent this problem if we can optionally configure these > properties in terms of percentage of cluster capacity. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135343#comment-16135343
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134266062
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFSParentQueue.java
---
@@ -42,6 +43,8 @@ public void setUp() throws Exception {
AllocationConfiguration allocConf = new AllocationConfiguration(conf);
when(scheduler.getAllocationConfiguration()).thenReturn(allocConf);
when(scheduler.getConf()).thenReturn(conf);
+when(scheduler.getResourceCalculator()).thenReturn(
--- End diff --
Why do we need to set the resource-calculator now and not before?
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135347#comment-16135347
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r133314026
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/AllocationConfiguration.java
---
@@ -293,26 +293,15 @@ Resource getMinResources(String queue) {
/**
* Get the maximum resource allocation for the given queue. If the max
in not
- * set, return the larger of the min and the default max.
+ * set, return the default max.
--- End diff --
Why are we changing this behavior?
With the new code, how do we handle the case where minResources for the
queue is higher than the default maxResources?
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135337#comment-16135337
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r133307006
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/ResourceConfiguration.java
---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceType;
+
+import java.util.HashMap;
+
+/**
+ * A ResourceConfiguration represents an entity that is used to
configuration
+ * resources, such as maximum resources of a queue. It can be percentage of
+ * cluster resources or an absolute value.
+ */
+@Private
+@Unstable
+public class ResourceConfiguration {
--- End diff --
I am not a fan of the name. That said, I haven't been able to come up with
a better name. Maybe, ConfigurableResource or something more descriptive like
AbsoluteOrPercentResource or ResourceAsAbsoluteOrPercent.
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135341#comment-16135341
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r134264696
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FSQueue.java
---
@@ -158,26 +158,33 @@ public Resource getMinShare() {
return minShare;
}
- public void setMaxShare(Resource maxShare){
+ public void setMaxShare(ResourceConfiguration maxShare){
this.maxShare = maxShare;
}
+ @Override
+ public Resource getMaxShare() {
+Resource maxResource =
maxShare.getResource(scheduler.getClusterResource());
+// Set max resource to min resource if min resource is greater than max
+// resource
+if(Resources.greaterThan(scheduler.getResourceCalculator(),
+scheduler.getClusterResource(), minShare, maxResource)) {
--- End diff --
Is the indentation off by one space?
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135342#comment-16135342
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r133307373
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/ResourceConfiguration.java
---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceType;
+
+import java.util.HashMap;
+
+/**
+ * A ResourceConfiguration represents an entity that is used to
configuration
+ * resources, such as maximum resources of a queue. It can be percentage of
+ * cluster resources or an absolute value.
+ */
+@Private
+@Unstable
+public class ResourceConfiguration {
+ private final boolean isPercentage;
+ private final Resource resource;
+ private final HashMap percentages;
--- End diff --
Is a map absolutely necessary? In an offline conversation, Daniel mentioned
the new resource types feature introduces ordering of resource types. Maybe, we
could use an array or vector?
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2162) add ability in Fair Scheduler to optionally configure maxResources in terms of percentage
[
https://issues.apache.org/jira/browse/YARN-2162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135336#comment-16135336
]
ASF GitHub Bot commented on YARN-2162:
--
Github user kambatla commented on a diff in the pull request:
https://github.com/apache/hadoop/pull/261#discussion_r133307075
--- Diff:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/ResourceConfiguration.java
---
@@ -0,0 +1,71 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.hadoop.yarn.api.records.Resource;
+import org.apache.hadoop.yarn.server.resourcemanager.resource.ResourceType;
+
+import java.util.HashMap;
+
+/**
+ * A ResourceConfiguration represents an entity that is used to
configuration
--- End diff --
Throw the class name in {@link }
> add ability in Fair Scheduler to optionally configure maxResources in terms
> of percentage
> -
>
> Key: YARN-2162
> URL: https://issues.apache.org/jira/browse/YARN-2162
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler, scheduler
>Reporter: Ashwin Shankar
>Assignee: Yufei Gu
> Labels: scheduler
> Attachments: YARN-2162.001.patch, YARN-2162.002.patch
>
>
> minResources and maxResources in fair scheduler configs are expressed in
> terms of absolute numbers X mb, Y vcores.
> As a result, when we expand or shrink our hadoop cluster, we need to
> recalculate and change minResources/maxResources accordingly, which is pretty
> inconvenient.
> We can circumvent this problem if we can optionally configure these
> properties in terms of percentage of cluster capacity.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6861) Reader API for sub application entities
[ https://issues.apache.org/jira/browse/YARN-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135317#comment-16135317 ] Vrushali C commented on YARN-6861: -- Checking the branch2 failure on my local machine > Reader API for sub application entities > --- > > Key: YARN-6861 > URL: https://issues.apache.org/jira/browse/YARN-6861 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: atsv2-subapp, yarn-5355-merge-blocker > Attachments: YARN-6861-YARN-5355.001.patch, > YARN-6861-YARN-5355.002.patch, YARN-6861-YARN-5355.003.patch, > YARN-6861-YARN-5355.004.patch, YARN-6861-YARN-5355.005.patch, > YARN-6861-YARN-5355.006.patch, YARN-6861-YARN-5355_branch2.001.patch > > > YARN-6733 and YARN-6734 writes data into sub application table. There should > be a way to read those entities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6861) Reader API for sub application entities
[ https://issues.apache.org/jira/browse/YARN-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135310#comment-16135310 ] Rohith Sharma K S commented on YARN-6861: - YARN-5355_branch2 test failure because of YARN-6820. Need to investigate it. > Reader API for sub application entities > --- > > Key: YARN-6861 > URL: https://issues.apache.org/jira/browse/YARN-6861 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: atsv2-subapp, yarn-5355-merge-blocker > Attachments: YARN-6861-YARN-5355.001.patch, > YARN-6861-YARN-5355.002.patch, YARN-6861-YARN-5355.003.patch, > YARN-6861-YARN-5355.004.patch, YARN-6861-YARN-5355.005.patch, > YARN-6861-YARN-5355.006.patch, YARN-6861-YARN-5355_branch2.001.patch > > > YARN-6733 and YARN-6734 writes data into sub application table. There should > be a way to read those entities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135306#comment-16135306
]
Eric Badger commented on YARN-6623:
---
[~vvasudev], I started reviewing the .003 patch, but I see that you've updated
a few times since then. So I'll post my comments so far and then continue my
review with the most recent patch.
Ran test-container-executor on both my mac and on rhel6 and it passed.
1. Should {{feature.docker.enabled}} be moved to be with the rest of the docker
configs?
2. For the Docker configs, I think it'd be more clear if they were prefixed
with {{docker.}}
3. Since we're going to remove the hard-coded string in YARN-6968, should we
leave the findbugs warning and take care of it there?
4.
{noformat}
+ /**
+ * Add command commandWithArguments - this method is only meant for use by
+ * sub-classes.
+ *
{noformat}
This is no longer just adding a command. Looks like it adds arguments to the
command and adds the command as well if it doesn't exist. So the comment should
be updated.
5.
{noformat}
+this.commandArguments = new TreeMap<>();
+commandArguments.put(dockerCommandKey, new ArrayList<>());
+commandArguments.get(dockerCommandKey).add(command);
{noformat}
I’m not sure I understand how this piece of code is supposed to work. In this
method we add a new list with the key “docker-command”. But then in
{{addCommandArguments()}} we add arguments via the {{key}}, which is always
called in the subclasses with the name of the command being run (e.g. inspect).
So the map for each {{DockerCommand}} will usually have 2 keys,
“docker-command” and the command that actually needs arguments “inspect”. Is
this how it’s intended to work or did I miss something here?
6.
{noformat}
+ protected final void addCommandArguments(String key, String value) {
+if (commandArguments.containsKey(key)) {
+ commandArguments.get(key).add(value);
+ return;
+}
{noformat}
No need to call {{contains()}} and then call {{get()}}. We can just call get
and then add the value if the return value isn't null.
7.
{noformat}
+ @Override
+ public String toString() {
+StringBuffer ret = new StringBuffer("");
+ret.append(this.command);
{noformat}
Any reason not to create the string with the command in the constructor?
Something like
{{StringBuffer ret = new StringBuffer(this.command);}}
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[
https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135299#comment-16135299
]
Shane Kumpf commented on YARN-6930:
---
Thanks for the feedback, [~jlowe]! I'm not opposed to moving this to enums and
simpler names if that is the preferred direction.
{quote}
If we are going the hardcoded route then arguably we should list the ones we
know about here so users know what they can do.
{quote}
When you say "list the ones we know about here", can you elaborate on what you
mean? List them in the documentation?
> Admins should be able to explicitly enable specific LinuxContainerRuntime in
> the NodeManager
>
>
> Key: YARN-6930
> URL: https://issues.apache.org/jira/browse/YARN-6930
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: nodemanager
>Reporter: Vinod Kumar Vavilapalli
>Assignee: Shane Kumpf
> Attachments: YARN-6930.001.patch, YARN-6930.002.patch,
> YARN-6930.003.patch
>
>
> Today, in the java land, all LinuxContainerRuntimes are always enabled when
> using LinuxContainerExecutor and the user can simply invoke anything that
> he/she wants - default, docker, java-sandbox.
> We should have a way for admins to explicitly enable only specific runtimes
> that he/she decides for the cluster. And by default, we should have
> everything other than the default one disabled.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6251) Add support for async Container release to prevent deadlock during container updates
[ https://issues.apache.org/jira/browse/YARN-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135254#comment-16135254 ] Arun Suresh commented on YARN-6251: --- Thanks for the review [~leftnoteasy]. I just updated the jira title. > Add support for async Container release to prevent deadlock during container > updates > > > Key: YARN-6251 > URL: https://issues.apache.org/jira/browse/YARN-6251 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6251.001.patch, YARN-6251.002.patch, > YARN-6251.003.patch, YARN-6251.004.patch, YARN-6251.005.patch, > YARN-6251.006.patch > > > Opening to track a locking issue that was uncovered when running a custom SLS > AMSimulator. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6251) Add support for async Container release to prevent deadlock during container updates
[ https://issues.apache.org/jira/browse/YARN-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated YARN-6251: -- Summary: Add support for async Container release to prevent deadlock during container updates (was: Fix Scheduler locking issue introduced by YARN-6216) > Add support for async Container release to prevent deadlock during container > updates > > > Key: YARN-6251 > URL: https://issues.apache.org/jira/browse/YARN-6251 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Arun Suresh >Assignee: Arun Suresh > Attachments: YARN-6251.001.patch, YARN-6251.002.patch, > YARN-6251.003.patch, YARN-6251.004.patch, YARN-6251.005.patch, > YARN-6251.006.patch > > > Opening to track a locking issue that was uncovered when running a custom SLS > AMSimulator. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135245#comment-16135245
]
Varun Vasudev commented on YARN-6623:
-
{quote}
#define EXECUTOR_PATH_MAX 131072
This is lots of allocation. The OS actually needs to zero all the allocated
memory before giving it to other processes after close, so this might add to
the overall CPU usage and memory bandwidth.
Given that yarn local dirs are bind mounted into containers, using 4096 has
proven problematic with even a small number of yarn local dirs/disks (3 in the
case I saw). We're going to need to make the size of this buffer configurable.
If it seems more appropriate to do so in a separate issue, I'm fine with that.
{quote}
This got fixed by YARN-6988.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[ https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Vasudev updated YARN-6623: Attachment: YARN-6623.005.patch New patch uploaded after rebasing to trunk. > Add support to turn off launching privileged containers in the > container-executor > - > > Key: YARN-6623 > URL: https://issues.apache.org/jira/browse/YARN-6623 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Reporter: Varun Vasudev >Assignee: Varun Vasudev >Priority: Blocker > Attachments: YARN-6623.001.patch, YARN-6623.002.patch, > YARN-6623.003.patch, YARN-6623.004.patch, YARN-6623.005.patch > > > Currently, launching privileged containers is controlled by the NM. We should > add a flag to the container-executor.cfg allowing admins to disable launching > privileged containers at the container-executor level. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6736) Consider writing to both ats v1 & v2 from RM for smoother upgrades
[ https://issues.apache.org/jira/browse/YARN-6736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135237#comment-16135237 ] Aaron Gresch commented on YARN-6736: Should this still be based off the YARN-5355 branch? > Consider writing to both ats v1 & v2 from RM for smoother upgrades > -- > > Key: YARN-6736 > URL: https://issues.apache.org/jira/browse/YARN-6736 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Aaron Gresch > Attachments: YARN-6736-YARN-5355.001.patch, > YARN-6736-YARN-5355.002.patch > > > When the cluster is being upgraded from atsv1 to v2, it may be good to have a > brief time period during which RM writes to both atsv1 and v2. This will help > frameworks like Tez migrate more smoothly. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135223#comment-16135223
]
Shane Kumpf commented on YARN-6623:
---
{quote}
#define EXECUTOR_PATH_MAX 131072
This is lots of allocation. The OS actually needs to zero all the allocated
memory before giving it to other processes after close, so this might add to
the overall CPU usage and memory bandwidth.
{quote}
Given that yarn local dirs are bind mounted into containers, using 4096 has
proven problematic with even a small number of yarn local dirs/disks (3 in the
case I saw). We're going to need to make the size of this buffer configurable.
If it seems more appropriate to do so in a separate issue, I'm fine with that.
{quote}
Image can be specified by a digest, which is more secure. I do not see that
supported by the regex. IMAGE[:TAG|@DIGEST]
{quote}
This is the same regex used on the java side, just duplicated in c-e to address
the concerns with direct invocation. We should fix this both places to support
the digest notation. Again, this seems like a separate issue from what this
patch is addressing.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7058) Add null check in AMRMClientImpl#getMatchingRequest
[ https://issues.apache.org/jira/browse/YARN-7058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated YARN-7058: - Attachment: YARN-7058-branch-2.001.patch > Add null check in AMRMClientImpl#getMatchingRequest > --- > > Key: YARN-7058 > URL: https://issues.apache.org/jira/browse/YARN-7058 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.9.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta > Attachments: YARN-7058-branch-2.001.patch > > > As of YARN-4889, the behavior of AMRMClientImpl#getMatchingRequests has > slightly changed. > After YARN-4889, the method will throw NPE if the method is called before > calling addContainerRequest. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-7058) Add null check in AMRMClientImpl#getMatchingRequest
[ https://issues.apache.org/jira/browse/YARN-7058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated YARN-7058: - Attachment: (was: YARN-7058-branch-2.001.patch) > Add null check in AMRMClientImpl#getMatchingRequest > --- > > Key: YARN-7058 > URL: https://issues.apache.org/jira/browse/YARN-7058 > Project: Hadoop YARN > Issue Type: Bug > Components: client >Affects Versions: 2.9.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta > Attachments: YARN-7058-branch-2.001.patch > > > As of YARN-4889, the behavior of AMRMClientImpl#getMatchingRequests has > slightly changed. > After YARN-4889, the method will throw NPE if the method is called before > calling addContainerRequest. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135193#comment-16135193
]
Hadoop QA commented on YARN-6623:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-6623 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6623 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882875/YARN-6623.004.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17031/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: nodemanager
>Reporter: Varun Vasudev
>Assignee: Varun Vasudev
>Priority: Blocker
> Attachments: YARN-6623.001.patch, YARN-6623.002.patch,
> YARN-6623.003.patch, YARN-6623.004.patch
>
>
> Currently, launching privileged containers is controlled by the NM. We should
> add a flag to the container-executor.cfg allowing admins to disable launching
> privileged containers at the container-executor level.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6623) Add support to turn off launching privileged containers in the container-executor
[
https://issues.apache.org/jira/browse/YARN-6623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Vasudev updated YARN-6623:
Attachment: YARN-6623.004.patch
Thanks for the review [~miklos.szeg...@cloudera.com]!
{quote}
/proc/mounts,/sys/fs/cgroup are always in the same place
This is actually not completely true. If you run in a container,/sys/fs/cgroup
can be anywhere
490.addReadOnlyMountLocation(CGROUPS_ROOT_DIRECTORY,
491CGROUPS_ROOT_DIRECTORY, false);
This is actually a security issue. As opposed to lxcfs, mounting cgroups will
expose information about all the other containers to each container. This
change is big enough but you might want to whitelist this in the future.
{quote}
This was already being set before this patch. Perhaps, it can be handled
cleanly as part of YARN-5534?
{quote}
SET(CMAKE_CXX_FLAGS "$\{CMAKE_CXX_FLAGS} -Wall -std=c++11")
Not sure, but this might not build on old OS like centos6
{quote}
Fixed.
{quote}
76 printWriter.println("\[docker-command-execution]");
77 for (Map.Entry entry :
cmd.getCommandArguments()
78 .entrySet()) \{
79printWriter.println(" " + entry.getKey() + "=" + StringUtils
80.join(",", entry.getValue()));
81 }
Probably it does worth to check, if entry for example contains “abc=\ndef” to
avoid injection attacks.
{quote}
Fixed.
{quote}
169 ret\[i + 1] = '\0';
I think it is safe to do a single ret\[I] = 0; after the loop
{quote}
Fixed.
{quote}
177void quote_and_append_arg(char **str, size_t *size, const char* param,
const char *arg) \{
178 char *tmp = escape_single_quote(arg);
179 strcat(*str, param);
180 strcat(*str, "'");
181 if(strlen(*str) + strlen(tmp) > *size) \{
182*str = (char *) realloc(*str, strlen(*str) + strlen(tmp) + 1024);
183if(*str == NULL) \{
184 exit(OUT_OF_MEMORY);
185}
186*size = strlen(*str) + strlen(tmp) + 1024;
187 }
188 strcat(*str, tmp);
189 strcat(*str, "' ");
190 free(tmp);
191}
What is 1024? How about setting * size before *str, so that there is no need
for duplication?
{quote}
Fixed.
{quote}
#define EXECUTOR_PATH_MAX 131072
This is lots of allocation. The OS actually needs to zero all the allocated
memory before giving it to other processes after close, so this might add to
the overall CPU usage and memory bandwidth.
{quote}
Fixed. Set back to 4096.
{quote}
35if (command != NULL) \{
36 free(command);
37}
This is not necessary, free always ignores NULL argument.
{quote}
Fixed.
{quote}
47 if (current_len + string_len < bufflen) \{
bufflen-1 to allocate space for the terminating 0
{quote}
Fixed.
{quote}
48strncpy(buff + current_len, string, bufflen - current_len);
strncpy does not add 0 terminator at the end of the target, if
strlen(string)==bufflen - current_len resulting in a read buffer overflow later.
{quote}
Fixed.
{quote}
165 if (0 == strncmp(container_name, CONTAINER_NAME_PREFIX,
strlen(CONTAINER_NAME_PREFIX))) \{
This is a double scan. You can just use strcmp with the same effect.
{quote}
{quote}
249 const char *regex_str =
"^((\[a-zA-Z0-9.-]+)(:\[0-9]+)?/)?(\[a-z0-9_./-]+)(:\[a-zA-Z0-9_.-]+)?$”;
Image can be specified by a digest, which is more secure. I do not see that
supported by the regex. IMAGE\[:TAG|@DIGEST]
{quote}
[~shaneku...@gmail.com] added this, I'm just moving it to a different file.
Shane can you please comment?
{quote}
# docker.binary=/bin/docker
115 docker_binary = strdup("/usr/bin/docker”);
We should choose one and use it everywhere.
{quote}
Fixed.
{quote}
477permitted_mount_len = strlen(permitted_mounts\[i]);
478if (permitted_mounts\[i]\[permitted_mount_len - 1] == '/') \{
Buffer underflow at permitted_mount_len == 0
{quote}
Fixed.
{quote}
488static char* normalize_mount(const char* mount) \{
It would be really nice to have some comments here.
{quote}
Fixed.
{quote}
503 size_t len = strlen(real_mount);
504 if (real_mount\[len - 1] != '/') {
505ret_ptr = (char *) alloc_and_clear_memory(len + 2, sizeof(char));
506strncpy(ret_ptr, real_mount, len);
507ret_ptr\[len] = '/';
508ret_ptr\[len + 1] = '\0';
Potential buffer underflow moreover most likely the character does not match
and we end with a normalized mount path of “/“. This happens, when
strlen(real_mount)==0
{quote}
Fixed
> Add support to turn off launching privileged containers in the
> container-executor
> -
>
> Key: YARN-6623
> URL: https://issues.apache.org/jira/browse/YARN-6623
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components:
[jira] [Commented] (YARN-6933) ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources() are duplicating work
[
https://issues.apache.org/jira/browse/YARN-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135165#comment-16135165
]
Hadoop QA commented on YARN-6933:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-6933 does not apply to YARN-3926. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6933 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882871/YARN-6933-YARN-3926.005.patch
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17030/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources()
> are duplicating work
> ---
>
> Key: YARN-6933
> URL: https://issues.apache.org/jira/browse/YARN-6933
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Manikandan R
> Labels: newbie++
> Attachments: YARN-6933-YARN-3926.001.patch,
> YARN-6933-YARN-3926.002.patch, YARN-6933-YARN-3926.003.patch,
> YARN-6933-YARN-3926.004.patch, YARN-6933-YARN-3926.005.patch
>
>
> Both are used to check that the mandatory resources were not redefined. Only
> one check is needed. I would recommend dropping {{DISALLOWED_RESOURCES}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6933) ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources() are duplicating work
[
https://issues.apache.org/jira/browse/YARN-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikandan R updated YARN-6933:
---
Attachment: YARN-6933-YARN-3926.005.patch
> ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources()
> are duplicating work
> ---
>
> Key: YARN-6933
> URL: https://issues.apache.org/jira/browse/YARN-6933
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Manikandan R
> Labels: newbie++
> Attachments: YARN-6933-YARN-3926.001.patch,
> YARN-6933-YARN-3926.002.patch, YARN-6933-YARN-3926.003.patch,
> YARN-6933-YARN-3926.004.patch, YARN-6933-YARN-3926.005.patch
>
>
> Both are used to check that the mandatory resources were not redefined. Only
> one check is needed. I would recommend dropping {{DISALLOWED_RESOURCES}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6933) ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources() are duplicating work
[
https://issues.apache.org/jira/browse/YARN-6933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135143#comment-16135143
]
Manikandan R commented on YARN-6933:
Modified the exception message. Also fixed checkstyle issues and attached the
new patch.
> ResourceUtils.DISALLOWED_NAMES and ResourceUtils.checkMandatoryResources()
> are duplicating work
> ---
>
> Key: YARN-6933
> URL: https://issues.apache.org/jira/browse/YARN-6933
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
>Affects Versions: YARN-3926
>Reporter: Daniel Templeton
>Assignee: Manikandan R
> Labels: newbie++
> Attachments: YARN-6933-YARN-3926.001.patch,
> YARN-6933-YARN-3926.002.patch, YARN-6933-YARN-3926.003.patch,
> YARN-6933-YARN-3926.004.patch, YARN-6933-YARN-3926.005.patch
>
>
> Both are used to check that the mandatory resources were not redefined. Only
> one check is needed. I would recommend dropping {{DISALLOWED_RESOURCES}}.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6861) Reader API for sub application entities
[
https://issues.apache.org/jira/browse/YARN-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135122#comment-16135122
]
Hadoop QA commented on YARN-6861:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
28s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} YARN-5355_branch2 Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 2m
25s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
27s{color} | {color:green} YARN-5355_branch2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
33s{color} | {color:green} YARN-5355_branch2 passed with JDK v1.8.0_144 {color}
|
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} YARN-5355_branch2 passed with JDK v1.7.0_131 {color}
|
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} YARN-5355_branch2 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
11s{color} | {color:green} YARN-5355_branch2 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase
in YARN-5355_branch2 has 1 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} YARN-5355_branch2 passed with JDK v1.8.0_144 {color}
|
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} YARN-5355_branch2 passed with JDK v1.7.0_131 {color}
|
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
32s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
57s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 31s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 6 new +
34 unchanged - 0 fixed = 40 total (was 34) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
34s{color} | {color:green} the patch passed with JDK v1.8.0_144 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
51s{color} | {color:green} hadoop-yarn-server-timelineservice in the patch
passed with JDK v1.7.0_131. {color} |
|
[jira] [Updated] (YARN-6861) Reader API for sub application entities
[ https://issues.apache.org/jira/browse/YARN-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rohith Sharma K S updated YARN-6861: Attachment: YARN-6861-YARN-5355_branch2.001.patch updated yarn-5355_branch2 patch. > Reader API for sub application entities > --- > > Key: YARN-6861 > URL: https://issues.apache.org/jira/browse/YARN-6861 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: atsv2-subapp, yarn-5355-merge-blocker > Attachments: YARN-6861-YARN-5355.001.patch, > YARN-6861-YARN-5355.002.patch, YARN-6861-YARN-5355.003.patch, > YARN-6861-YARN-5355.004.patch, YARN-6861-YARN-5355.005.patch, > YARN-6861-YARN-5355.006.patch, YARN-6861-YARN-5355_branch2.001.patch > > > YARN-6733 and YARN-6734 writes data into sub application table. There should > be a way to read those entities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5402) Fix NoSuchMethodError in ClusterMetricsInfo
[
https://issues.apache.org/jira/browse/YARN-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135062#comment-16135062
]
yuelongsun edited comment on YARN-5402 at 8/21/17 11:44 AM:
I got the same problem again using vesion 2.7.4. No one else meet this
problem? :(
was (Author: yuelong):
I got the same problem again using vesion 2.7.4. No one meet this problem? :(
> Fix NoSuchMethodError in ClusterMetricsInfo
> ---
>
> Key: YARN-5402
> URL: https://issues.apache.org/jira/browse/YARN-5402
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: webapp
>Affects Versions: YARN-3368
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-5402.YARN-3368.001.patch
>
>
> When trying out new UI on a cluster, the index page failed to load because of
> error {code}java.lang.NoSuchMethodError:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.getReservedMB()J{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5402) Fix NoSuchMethodError in ClusterMetricsInfo
[
https://issues.apache.org/jira/browse/YARN-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135062#comment-16135062
]
yuelongsun edited comment on YARN-5402 at 8/21/17 11:44 AM:
I got the same problem again using vesion 2.7.4. No one meet this problem? :(
was (Author: yuelong):
I got the same problem again using vesion 2.7.4. No other one meet this
problem? :(
> Fix NoSuchMethodError in ClusterMetricsInfo
> ---
>
> Key: YARN-5402
> URL: https://issues.apache.org/jira/browse/YARN-5402
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: webapp
>Affects Versions: YARN-3368
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-5402.YARN-3368.001.patch
>
>
> When trying out new UI on a cluster, the index page failed to load because of
> error {code}java.lang.NoSuchMethodError:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.getReservedMB()J{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-5402) Fix NoSuchMethodError in ClusterMetricsInfo
[
https://issues.apache.org/jira/browse/YARN-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135062#comment-16135062
]
yuelongsun edited comment on YARN-5402 at 8/21/17 11:43 AM:
I got the same problem again using vesion 2.7.4. No other one meet this
problem? :(
was (Author: yuelong):
I got the same problem again using vesion 2.7.4. nobody else meet this
problem? :(
> Fix NoSuchMethodError in ClusterMetricsInfo
> ---
>
> Key: YARN-5402
> URL: https://issues.apache.org/jira/browse/YARN-5402
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: webapp
>Affects Versions: YARN-3368
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-5402.YARN-3368.001.patch
>
>
> When trying out new UI on a cluster, the index page failed to load because of
> error {code}java.lang.NoSuchMethodError:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.getReservedMB()J{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5402) Fix NoSuchMethodError in ClusterMetricsInfo
[
https://issues.apache.org/jira/browse/YARN-5402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135062#comment-16135062
]
yuelongsun commented on YARN-5402:
--
I got the same problem again using vesion 2.7.4. nobody else meet this
problem? :(
> Fix NoSuchMethodError in ClusterMetricsInfo
> ---
>
> Key: YARN-5402
> URL: https://issues.apache.org/jira/browse/YARN-5402
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: webapp
>Affects Versions: YARN-3368
>Reporter: Weiwei Yang
>Assignee: Weiwei Yang
> Attachments: YARN-5402.YARN-3368.001.patch
>
>
> When trying out new UI on a cluster, the index page failed to load because of
> error {code}java.lang.NoSuchMethodError:
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics.getReservedMB()J{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6047) Documentation updates for TimelineService v2
[ https://issues.apache.org/jira/browse/YARN-6047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135057#comment-16135057 ] Rohith Sharma K S commented on YARN-6047: - Ping [~vrushalic] [~haibo.chen] [~jrottinghuis] > Documentation updates for TimelineService v2 > > > Key: YARN-6047 > URL: https://issues.apache.org/jira/browse/YARN-6047 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation, timelineserver >Reporter: Varun Saxena >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Fix For: YARN-5355 > > Attachments: TimelineServiceV2.html, YARN-6047-YARN-5355.001.patch, > YARN-6047-YARN-5355.002.patch, YARN-6047-YARN-5355.003.patch, > YARN-6047-YARN-5355.004.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-7036) [ATSv2] : NPE for recovered finished applications in NM during upgrade
[
https://issues.apache.org/jira/browse/YARN-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rohith Sharma K S resolved YARN-7036.
-
Resolution: Invalid
This issue happens because of TimelineClient fail to start when we configure
ssl-client.xml. It is mandatory that if we configure ssl-client.xml values i.e
the jks file must be present.
It is basically cluster set up issue which was causing this issue. Because of
this subsequent entity publishing also were failing.
Closing the issue as invalid.
> [ATSv2] : NPE for recovered finished applications in NM during upgrade
> --
>
> Key: YARN-7036
> URL: https://issues.apache.org/jira/browse/YARN-7036
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Rohith Sharma K S
>
> It is seen that NPE is logged in NM logs when finished applications are
> recovered during upgrade. But I do not see any impact for functionality of NM.
> Trace is
> {noformat}
> 017-08-17 10:22:46,137 WARN service.AbstractService
> (ServiceOperations.java:stopQuietly(82)) - When stopping the service
> org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl :
> java.lang.NullPointerException
> java.lang.NullPointerException
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl$TimelineEntityDispatcher.stop(TimelineV2ClientImpl.java:471)
> at
> org.apache.hadoop.yarn.client.api.impl.TimelineV2ClientImpl.serviceStop(TimelineV2ClientImpl.java:130)
> at
> org.apache.hadoop.service.AbstractService.stop(AbstractService.java:221)
> at
> org.apache.hadoop.service.ServiceOperations.stop(ServiceOperations.java:52)
> at
> org.apache.hadoop.service.ServiceOperations.stopQuietly(ServiceOperations.java:80)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:171)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$1.run(NMTimelinePublisher.java:415)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher$1.run(NMTimelinePublisher.java:410)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> at
> org.apache.hadoop.yarn.server.nodemanager.timelineservice.NMTimelinePublisher.createTimelineClient(NMTimelinePublisher.java:410)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.application.ApplicationImpl.(ApplicationImpl.java:125)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recoverApplication(ContainerManagerImpl.java:401)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.recover(ContainerManagerImpl.java:353)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:307)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:408)
> at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:790)
> at
> org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:851)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6861) Reader API for sub application entities
[ https://issues.apache.org/jira/browse/YARN-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134767#comment-16134767 ] Varun Saxena edited comment on YARN-6861 at 8/21/17 6:44 AM: - bq. AM will start with same user but they publish data with doAs user. The whole DAG execution will happens with one doAs user. Yeah I know doAs user would be used. I meant knowing the app id user would not be important in this case as there will be only one record returned per entity id. Anyways as you said FROM_ID would contain the app user id so should be good enough to differentiate between records. was (Author: varun_saxena): bq. AM will start with same user but they publish data with doAs user. The whole DAG execution will happens with one doAs user. Yeah I know doAs user would be used. I meant knowing the app id user would not be that important in this case. Anyways as you said FROM_ID would contain the app user id so should be good enough to differentiate between records. > Reader API for sub application entities > --- > > Key: YARN-6861 > URL: https://issues.apache.org/jira/browse/YARN-6861 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: atsv2-subapp, yarn-5355-merge-blocker > Attachments: YARN-6861-YARN-5355.001.patch, > YARN-6861-YARN-5355.002.patch, YARN-6861-YARN-5355.003.patch, > YARN-6861-YARN-5355.004.patch, YARN-6861-YARN-5355.005.patch, > YARN-6861-YARN-5355.006.patch > > > YARN-6733 and YARN-6734 writes data into sub application table. There should > be a way to read those entities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6861) Reader API for sub application entities
[ https://issues.apache.org/jira/browse/YARN-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134767#comment-16134767 ] Varun Saxena commented on YARN-6861: bq. AM will start with same user but they publish data with doAs user. The whole DAG execution will happens with one doAs user. Yeah I know doAs user would be used. I meant knowing the app id user would not be that important in this case. Anyways as you said FROM_ID would contain the app user id so should be good enough to differentiate between records. > Reader API for sub application entities > --- > > Key: YARN-6861 > URL: https://issues.apache.org/jira/browse/YARN-6861 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Labels: atsv2-subapp, yarn-5355-merge-blocker > Attachments: YARN-6861-YARN-5355.001.patch, > YARN-6861-YARN-5355.002.patch, YARN-6861-YARN-5355.003.patch, > YARN-6861-YARN-5355.004.patch, YARN-6861-YARN-5355.005.patch, > YARN-6861-YARN-5355.006.patch > > > YARN-6733 and YARN-6734 writes data into sub application table. There should > be a way to read those entities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7043) Cleanup ResourceProfileManager
[ https://issues.apache.org/jira/browse/YARN-7043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134764#comment-16134764 ] Sunil G commented on YARN-7043: --- [~leftnoteasy], jenkins seems fine now. Latest patch looks good for me > Cleanup ResourceProfileManager > -- > > Key: YARN-7043 > URL: https://issues.apache.org/jira/browse/YARN-7043 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-7043.YARN-3926.001.patch, > YARN-7043.YARN-3926.002.patch, YARN-7043.YARN-3926.003.patch, > YARN-7043.YARN-3926.004.patch > > > Several cleanups we can do for ResourceProfileManager: > 1) Move GetResourceTypesInfo from profile manager to ResourceUtils. > 2) Move logics to check profile enabled, etc. from ClientRMService to > ResourceUtils. > 3) Throw exception when resource profile is disabled and method accessed by > other modules. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6792) Incorrect XML convertion in NodeIDsInfo and LabelsToNodesInfo
[ https://issues.apache.org/jira/browse/YARN-6792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134765#comment-16134765 ] Bibin A Chundatt commented on YARN-6792: [~sunilg] This issue is valid for 2.8 too. So i think we should merge to 2.8 too > Incorrect XML convertion in NodeIDsInfo and LabelsToNodesInfo > - > > Key: YARN-6792 > URL: https://issues.apache.org/jira/browse/YARN-6792 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola >Priority: Blocker > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6792.v1.patch, YARN-6792.v2.patch > > > NodeIDsInfo contains a typo and there is a missing constructor in > LabelsToNodesInfo. These bugs does not allow a correct conversation in XML of > LabelsToNodesInfo. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6384) Add configuratin to set max cpu usage when strict-resource-usage is false with cgroups
[
https://issues.apache.org/jira/browse/YARN-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16134743#comment-16134743
]
Hadoop QA commented on YARN-6384:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s{color}
| {color:red} YARN-6384 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-6384 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12882819/YARN-6384-0.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/17028/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add configuratin to set max cpu usage when strict-resource-usage is false
> with cgroups
> --
>
> Key: YARN-6384
> URL: https://issues.apache.org/jira/browse/YARN-6384
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: dengkai
> Attachments: YARN-6384-0.patch
>
>
> When using cgroups on yarn, if
> yarn.nodemanager.linux-container-executor.cgroups.strict-resource-usage is
> false, user may get very more cpu time than expected based on the vcores.
> There should be a upper limit even resource-usage is not strict, just like a
> percentage which user can get more than promised by vcores. I think it's
> important in a shared cluster.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
