[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[ https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736819#comment-16736819 ] Hudson commented on YARN-9037: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15743 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15743/]) YARN-9037. [CSI] Ignore volume resource in resource calculators based on (wwei: rev 0921b706f7f80c40e061d2c0f8c8b2e4910071e5) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/TempQueuePerPartition.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResourceUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/monitor/capacity/AbstractPreemptableResourceCalculator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/ClusterNodeTracker.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/volume/csi/TestVolumeProcessor.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/resources/resource-types/resource-types-6.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerConfiguration.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/policies/DominantResourceFairnessPolicy.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/ConfigurableResource.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/volume/csi/ContainerVolumePublisher.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/util/resource/ResourceUtils.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java > [CSI] Ignore volume resource in resource calculators based on tags > -- > > Key: YARN-9037 > URL: https://issues.apache.org/jira/browse/YARN-9037 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Sunil Govindan >Priority: Major > Labels: CSI > Fix For: 3.3.0 > > Attachments: YARN-9037-002.patch, YARN-9037.001.patch, > YARN-9037.003.patch, YARN-9037.004.patch > > > The pre-provisioned volume is specified as a resource, but such resource is > different comparing to what is managed now in YARN, e.g memory, vcores. They > are constrained by 3rd party storage systems, so it looks more like an > unmanaged resource. In such case, we need to ignore the resource calculation > over them in the resource calculators. This can be done based on the resource > tags. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[ https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736808#comment-16736808 ] Weiwei Yang commented on YARN-9037: --- I have committed this to trunk, thanks [~sunilg]. > [CSI] Ignore volume resource in resource calculators based on tags > -- > > Key: YARN-9037 > URL: https://issues.apache.org/jira/browse/YARN-9037 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Sunil Govindan >Priority: Major > Labels: CSI > Fix For: 3.3.0 > > Attachments: YARN-9037-002.patch, YARN-9037.001.patch, > YARN-9037.003.patch, YARN-9037.004.patch > > > The pre-provisioned volume is specified as a resource, but such resource is > different comparing to what is managed now in YARN, e.g memory, vcores. They > are constrained by 3rd party storage systems, so it looks more like an > unmanaged resource. In such case, we need to ignore the resource calculation > over them in the resource calculators. This can be done based on the resource > tags. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[ https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736792#comment-16736792 ] Weiwei Yang commented on YARN-9037: --- Thanks [~sunilg], I will commit this shortly today. > [CSI] Ignore volume resource in resource calculators based on tags > -- > > Key: YARN-9037 > URL: https://issues.apache.org/jira/browse/YARN-9037 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Sunil Govindan >Priority: Major > Labels: CSI > Attachments: YARN-9037-002.patch, YARN-9037.001.patch, > YARN-9037.003.patch, YARN-9037.004.patch > > > The pre-provisioned volume is specified as a resource, but such resource is > different comparing to what is managed now in YARN, e.g memory, vcores. They > are constrained by 3rd party storage systems, so it looks more like an > unmanaged resource. In such case, we need to ignore the resource calculation > over them in the resource calculators. This can be done based on the resource > tags. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[ https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736781#comment-16736781 ] Sunil Govindan commented on YARN-9037: -- Yes. [~cheersyang]. Let's get this in. Please commit this if no issues. > [CSI] Ignore volume resource in resource calculators based on tags > -- > > Key: YARN-9037 > URL: https://issues.apache.org/jira/browse/YARN-9037 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Sunil Govindan >Priority: Major > Labels: CSI > Attachments: YARN-9037-002.patch, YARN-9037.001.patch, > YARN-9037.003.patch, YARN-9037.004.patch > > > The pre-provisioned volume is specified as a resource, but such resource is > different comparing to what is managed now in YARN, e.g memory, vcores. They > are constrained by 3rd party storage systems, so it looks more like an > unmanaged resource. In such case, we need to ignore the resource calculation > over them in the resource calculators. This can be done based on the resource > tags. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[
https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735479#comment-16735479
]
Hadoop QA commented on YARN-9037:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
26s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
34s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
15s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 32s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 98 unchanged - 0 fixed = 101 total (was 98) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 2s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
48s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
42s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m
58s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 92m 25s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
38s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}214m 5s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.security.TestAMRMTokens |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9037 |
| JIRA Patch URL |
[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[
https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735384#comment-16735384
]
Weiwei Yang commented on YARN-9037:
---
Hi [~sunilg]
I think the patch works as designed, thanks.
I've done some changes based on your patch, as following
# Added another end to end UT in \{{TestVolumeProcessor}} to verify the
container allocation with volumes
# Renamed to EXTERNAL_VOLUME_RESOURCE_TAG to represent tag name for csi
volumes in \{{ResourceUtils}}
# Modified the value of volume resource in
\{{testMultipleOpsForResourcesWithTags}} to avoid the possibility of
verification scenario slipping away
Please take a look, thanks
> [CSI] Ignore volume resource in resource calculators based on tags
> --
>
> Key: YARN-9037
> URL: https://issues.apache.org/jira/browse/YARN-9037
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Weiwei Yang
>Assignee: Sunil Govindan
>Priority: Major
> Labels: CSI
> Attachments: YARN-9037-002.patch, YARN-9037.001.patch,
> YARN-9037.003.patch, YARN-9037.004.patch
>
>
> The pre-provisioned volume is specified as a resource, but such resource is
> different comparing to what is managed now in YARN, e.g memory, vcores. They
> are constrained by 3rd party storage systems, so it looks more like an
> unmanaged resource. In such case, we need to ignore the resource calculation
> over them in the resource calculators. This can be done based on the resource
> tags.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[ https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714914#comment-16714914 ] Weiwei Yang commented on YARN-9037: --- Hi [~sunilg] Latest patch looks good to me, I'll do some testing on a cluster to verify. Thanks a lot. > [CSI] Ignore volume resource in resource calculators based on tags > -- > > Key: YARN-9037 > URL: https://issues.apache.org/jira/browse/YARN-9037 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Sunil Govindan >Priority: Major > Labels: CSI > Attachments: YARN-9037-002.patch, YARN-9037.001.patch, > YARN-9037.003.patch > > > The pre-provisioned volume is specified as a resource, but such resource is > different comparing to what is managed now in YARN, e.g memory, vcores. They > are constrained by 3rd party storage systems, so it looks more like an > unmanaged resource. In such case, we need to ignore the resource calculation > over them in the resource calculators. This can be done based on the resource > tags. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[
https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714652#comment-16714652
]
Hadoop QA commented on YARN-9037:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
21s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
13s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 29s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 98 unchanged - 0 fixed = 101 total (was 98) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
50s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
48s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}108m 1s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}203m 43s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9037 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12951166/YARN-9037.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 0c0eae9c8869 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64
[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[ https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714449#comment-16714449 ] Sunil Govindan commented on YARN-9037: -- Ideally we let all creation of resources to have any types of resources added in the cluster. Hence getResourceTypesArray usage is fine. We just focus on the ops method where we ensure that resource types with tags named "system:csi-volume" is not considered to be a resource to be considered as countable. Added one test case to verify the change is fine in simpler level. > [CSI] Ignore volume resource in resource calculators based on tags > -- > > Key: YARN-9037 > URL: https://issues.apache.org/jira/browse/YARN-9037 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Sunil Govindan >Priority: Major > Labels: CSI > Attachments: YARN-9037-002.patch, YARN-9037.001.patch, > YARN-9037.003.patch > > > The pre-provisioned volume is specified as a resource, but such resource is > different comparing to what is managed now in YARN, e.g memory, vcores. They > are constrained by 3rd party storage systems, so it looks more like an > unmanaged resource. In such case, we need to ignore the resource calculation > over them in the resource calculators. This can be done based on the resource > tags. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[ https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714450#comment-16714450 ] Sunil Govindan commented on YARN-9037: -- [~cheersyang] cud u pls check > [CSI] Ignore volume resource in resource calculators based on tags > -- > > Key: YARN-9037 > URL: https://issues.apache.org/jira/browse/YARN-9037 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Sunil Govindan >Priority: Major > Labels: CSI > Attachments: YARN-9037-002.patch, YARN-9037.001.patch, > YARN-9037.003.patch > > > The pre-provisioned volume is specified as a resource, but such resource is > different comparing to what is managed now in YARN, e.g memory, vcores. They > are constrained by 3rd party storage systems, so it looks more like an > unmanaged resource. In such case, we need to ignore the resource calculation > over them in the resource calculators. This can be done based on the resource > tags. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[ https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16700031#comment-16700031 ] Weiwei Yang commented on YARN-9037: --- Hi [~sunilg] The patch overall looks good. Something I am not quite sure about ResourceUtils has some other public methods, e.g getResourceTypesArray(), this is used in DomainResourceFairnessPolicy to compute shares, would that cause issue when the array has some non-countable resource? Apart from that, can we add some UT for verification? E.g to create some resource requests with such tag, verify the get methods as well as the allocation result. Thanks > [CSI] Ignore volume resource in resource calculators based on tags > -- > > Key: YARN-9037 > URL: https://issues.apache.org/jira/browse/YARN-9037 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-9037-002.patch, YARN-9037.001.patch > > > The pre-provisioned volume is specified as a resource, but such resource is > different comparing to what is managed now in YARN, e.g memory, vcores. They > are constrained by 3rd party storage systems, so it looks more like an > unmanaged resource. In such case, we need to ignore the resource calculation > over them in the resource calculators. This can be done based on the resource > tags. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[
https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16699978#comment-16699978
]
Hadoop QA commented on YARN-9037:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
17s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
1m 13s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 3 new + 88 unchanged - 0 fixed = 91 total (was 88) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 15s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
46s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
33s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}101m 22s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}185m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | YARN-9037 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12949600/YARN-9037-002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 6a920ba324fb 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool |
[jira] [Commented] (YARN-9037) [CSI] Ignore volume resource in resource calculators based on tags
[ https://issues.apache.org/jira/browse/YARN-9037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693067#comment-16693067 ] Sunil Govindan commented on YARN-9037: -- cc [~cheersyang] Pls review. > [CSI] Ignore volume resource in resource calculators based on tags > -- > > Key: YARN-9037 > URL: https://issues.apache.org/jira/browse/YARN-9037 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Weiwei Yang >Assignee: Sunil Govindan >Priority: Major > Attachments: YARN-9037.001.patch > > > The pre-provisioned volume is specified as a resource, but such resource is > different comparing to what is managed now in YARN, e.g memory, vcores. They > are constrained by 3rd party storage systems, so it looks more like an > unmanaged resource. In such case, we need to ignore the resource calculation > over them in the resource calculators. This can be done based on the resource > tags. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
