[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16501854#comment-16501854 ] Szilard Nemeth commented on YARN-7451: -- Hi [~templedf]! Tested with latest upstream, resourceRequests shows custom resources and the scheduler endpoint also exposes custom resources for queues. Looks like this has been fixed with https://issues.apache.org/jira/browse/YARN-7817 Do you think it's worth to keep unit tests and open another Jira task to add them as 7817 haven't add any tests? Thanks in advance! > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-7451.001.patch, YARN-7451.002.patch, > YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch, > YARN-7451.006.patch, YARN-7451.007.patch, YARN-7451.008.patch, > YARN-7451.009.patch, YARN-7451.010.patch, YARN-7451.011.patch, > YARN-7451.012.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 > } > } > ] > {noformat} > However, the web UI shows resource types usage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490739#comment-16490739 ] Gergo Repas commented on YARN-7451: --- [~snemeth] Thanks for the updated patch. A couple of comments: * {{ResourceInfoWithCustomResourceTypes.customResources}} does not have the unit-information * {{ResourceInfoWithCustomResourceTypes.customResources()}} ** its javadoc is empty ** the memory and CPU resources are filtered out based on names. The rest of the codebase is already relying on having memory/CPU in the first two slots of the array, I think we could do the same in this method (iterating over the resourceInformations array starting with index=2) * There are checkstyle warnings > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-7451.001.patch, YARN-7451.002.patch, > YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch, > YARN-7451.006.patch, YARN-7451.007.patch, YARN-7451.008.patch, > YARN-7451.009.patch, YARN-7451.010.patch, YARN-7451.011.patch, > YARN-7451.012.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 > } > } > ] >
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490451#comment-16490451 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 32s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 20 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 6s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 34s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 13 new + 128 unchanged - 10 fixed = 141 total (was 138) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 24s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 68m 47s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}126m 29s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd | | JIRA Issue | YARN-7451 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12925105/YARN-7451.012.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 263ae18d6945 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 86bc642 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_162 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/20866/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit |
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16490334#comment-16490334 ] Szilard Nemeth commented on YARN-7451: -- Hi [~templedf], [~grepas]! I'm ready with a new patch. Since EclipseLink serialization and all of its dependencies and other test framework improvements were removed based on our offline discussion with Daniel, this patch is significantly smaller and different in many aspects than the previous patches, so I'm assuming it's worth for Gergo to have a look before Daniel checks it again. What do you think? About the checkstyle issues: * hides a field errors: I don't think it is needed to bother with those as renaming method parameters just for making this check happy would be strange. * Missing package-info: I don't think it is needed for test packages, currently most of our test packages don't have this. > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-7451.001.patch, YARN-7451.002.patch, > YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch, > YARN-7451.006.patch, YARN-7451.007.patch, YARN-7451.008.patch, > YARN-7451.009.patch, YARN-7451.010.patch, YARN-7451.011.patch, > YARN-7451.012.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, >
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16489985#comment-16489985 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 38s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} pathlen {color} | {color:red} 0m 0s{color} | {color:red} The patch appears to contain 1 files with names longer than 240 {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 21 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 28s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 34s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 15 new + 128 unchanged - 10 fixed = 143 total (was 138) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 43s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 68m 16s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 27s{color} | {color:red} The patch generated 1 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}127m 19s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd | | JIRA Issue | YARN-7451 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12925023/YARN-7451.011.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 088c0a22ee3f 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9 14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / d9852eb | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_162 | | pathlen | https://builds.apache.org/job/PreCommit-YARN-Build/20859/artifact/out/pathlen.txt | | findbugs | v3.1.0-RC1 | | checkstyle |
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16329010#comment-16329010 ] genericqa commented on YARN-7451: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 1s{color} | {color:green} The patch appears to include 35 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 19s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 17s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 51s{color} | {color:orange} root: The patch generated 24 new + 156 unchanged - 19 fixed = 180 total (was 175) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 38s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 37s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m 2s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s{color} | {color:green} hadoop-client-minicluster in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s{color} | {color:green} hadoop-client-check-test-invariants in the patch passed. {color} | | {color:green}+1{color} | {color:green}
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328966#comment-16328966 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m 52s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 35 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 36s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 9s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 40s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 7s{color} | {color:orange} root: The patch generated 46 new + 156 unchanged - 19 fixed = 202 total (was 175) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 33s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 32s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 61m 24s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s{color} | {color:green} hadoop-client-minicluster in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s{color} |
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328835#comment-16328835 ] Szilard Nemeth commented on YARN-7451: -- uploaded patch 010 with javadoc and checkstyle fixes > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-7451.001.patch, YARN-7451.002.patch, > YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch, > YARN-7451.006.patch, YARN-7451.007.patch, YARN-7451.008.patch, > YARN-7451.009.patch, YARN-7451.010.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 > } > } > ] > {noformat} > However, the web UI shows resource types usage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328804#comment-16328804 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 35 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 14s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 17s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 22s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 46s{color} | {color:orange} root: The patch generated 46 new + 156 unchanged - 19 fixed = 202 total (was 175) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 38s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 28s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m 9s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s{color} | {color:green} hadoop-client-minicluster in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s{color} |
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327132#comment-16327132 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 11m 4s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-7451 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12906199/YARN-7451.009.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19257/console | | Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-7451.001.patch, YARN-7451.002.patch, > YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch, > YARN-7451.006.patch, YARN-7451.007.patch, YARN-7451.008.patch, > YARN-7451.009.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 >
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16327103#comment-16327103 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 5m 36s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-7451 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12906199/YARN-7451.009.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19256/console | | Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-7451.001.patch, YARN-7451.002.patch, > YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch, > YARN-7451.006.patch, YARN-7451.007.patch, YARN-7451.008.patch, > YARN-7451.009.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 >
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326419#comment-16326419 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 33 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 35s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 44s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 14s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 45s{color} | {color:orange} root: The patch generated 153 new + 167 unchanged - 11 fixed = 320 total (was 178) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 9s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 12s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 28s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 16s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 54s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 28s{color} | {color:green} hadoop-client-minicluster in the patch
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326411#comment-16326411 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 33s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 33 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 56s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 17s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 15m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 53s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 39s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 32s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 14m 18s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 1s{color} | {color:orange} root: The patch generated 153 new + 167 unchanged - 11 fixed = 320 total (was 178) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 30s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 23s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 32s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m 9s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s{color} | {color:green} hadoop-client-minicluster in the patch
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326301#comment-16326301 ] Szilard Nemeth commented on YARN-7451: -- Thanks [~grepas] for your comments. Please see my updated patch (v8) for the fixes for your comments. # Good idea, I implemented a test class for this. # fixed # I added javadoc for most of the classes that were reasonable. > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-7451.001.patch, YARN-7451.002.patch, > YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch, > YARN-7451.006.patch, YARN-7451.007.patch, YARN-7451.008.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 > } > } > ] > {noformat} > However, the web UI shows resource types usage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16326088#comment-16326088 ] Gergo Repas commented on YARN-7451: --- Thanks [~snemeth] for the updated patch. The lack of extensibility of Jersey's LazyClassIterator is unfortunate, but I think using reflection in this case is okay. Couple of comments: # I think some explicit test cases for CustomServiceIteratorProvider would be useful: I understand this code is executed during the RM webservices tests, but would be good to test explicitly if jersey implementation details that we rely on is still as expected # CustomServiceIteratorProvider: in the pattern matching it's better to use Pattern.quote() around the string # There are empty javadoc-s in many of the classes you added > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth >Priority: Major > Attachments: YARN-7451.001.patch, YARN-7451.002.patch, > YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch, > YARN-7451.006.patch, YARN-7451.007.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 > } > } > ] > {noformat} > However, the web UI shows resource
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325733#comment-16325733 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 46s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 34 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 38s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 38s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 30s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 11m 36s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 57s{color} | {color:orange} root: The patch generated 31 new + 156 unchanged - 19 fixed = 187 total (was 175) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 6s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 12s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 17s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 32s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 2 new + 4 unchanged - 0 fixed = 6 total (was 4) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 16s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 78m 35s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s{color} | {color:green} hadoop-client-minicluster in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 16s{color} | {color:green}
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16325211#comment-16325211 ] Szilard Nemeth commented on YARN-7451: -- Sure, I will check that. I agree this seems the last option we have. Thanks! > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth > Attachments: YARN-7451.001.patch, YARN-7451.002.patch, > YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch, > YARN-7451.006.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 > } > } > ] > {noformat} > However, the web UI shows resource types usage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16324185#comment-16324185 ] Szilard Nemeth commented on YARN-7451: -- Unfortunately, I got an unfavorable response for the legal jira, see: https://issues.apache.org/jira/browse/LEGAL-359, so I cannot include the ServiceFinder as a source file, apparently. Any ideas how to resolve this? thanks! > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth > Attachments: YARN-7451.001.patch, YARN-7451.002.patch, > YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch, > YARN-7451.006.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 > } > } > ] > {noformat} > However, the web UI shows resource types usage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323947#comment-16323947 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 7m 8s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-7451 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12905862/YARN-7451.006.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19220/console | | Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth > Attachments: YARN-7451.001.patch, YARN-7451.002.patch, > YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch, > YARN-7451.006.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 > } > } >
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323938#comment-16323938 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 0m 9s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-7451 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12905862/YARN-7451.006.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19219/console | | Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth > Attachments: YARN-7451.001.patch, YARN-7451.002.patch, > YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch, > YARN-7451.006.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 > } > } >
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323865#comment-16323865 ] Gergo Repas commented on YARN-7451: --- Thanks [~snemeth] for the patch. A couple of comments: # You documented the motivation behind the serialization-related changes at some locations (etc. in ResourceInfoWithCustomResourceTypes), but would be useful to add some explanation to other locations. For example: ## ResourceManager.startWepApp() when setting the IteratorProvider ## JAXBContextResolver.ECLIPSELINK_SERIALIZED_TYPES - some comment would be useful why these classes need special handling (maybe even a short example). ## in the resourcemanager's pom.xml - where you're adding the new dependency # SchedulerInfo.getMaxClusterLevelAppPriority() - please add a comment what's the reason for adding @XmlTransient # In some classes imports are changed without any logic change, etc: RMWebApp, TestRMWebServicesConfigurationMutation # ArrayElementPathSegment.extractIndex() - when input does not match pattern - this case is not handled # There are outstanding checkstyle, license and javadoc issues > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth > Attachments: YARN-7451.001.patch, YARN-7451.002.patch, > YARN-7451.003.patch, YARN-7451.004.patch, YARN-7451.005.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { >
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16323216#comment-16323216 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 33 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 16m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 41s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 13m 0s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 15s{color} | {color:orange} root: The patch generated 154 new + 166 unchanged - 12 fixed = 320 total (was 178) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 25s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 33s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 19s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m 35s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s{color} | {color:green} hadoop-client-minicluster in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s{color} |
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16322779#comment-16322779 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 33 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 49s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 51s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 7m 43s{color} | {color:red} root in trunk failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 29s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 50s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 11m 50s{color} | {color:red} root generated 181 new + 1059 unchanged - 0 fixed = 1240 total (was 1059) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 55s{color} | {color:orange} root: The patch generated 154 new + 166 unchanged - 12 fixed = 320 total (was 178) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 29s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 31s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m 17s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s{color} | {color:green} hadoop-client-minicluster in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16321047#comment-16321047 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 33 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 10s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 39s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 11s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 22s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 46s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 51s{color} | {color:orange} root: The patch generated 154 new + 166 unchanged - 12 fixed = 320 total (was 178) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 49s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 13s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 27s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 65m 19s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s{color} | {color:green} hadoop-client-minicluster in the
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320195#comment-16320195 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 33 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 18s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 55s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 21s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 12m 17s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 4s{color} | {color:orange} root: The patch generated 153 new + 167 unchanged - 11 fixed = 320 total (was 178) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 0s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 23s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 32s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 1 new + 4 unchanged - 0 fixed = 5 total (was 4) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 64m 18s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 24s{color} | {color:green} hadoop-client-minicluster in the patch
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16320015#comment-16320015 ] Szilard Nemeth commented on YARN-7451: -- Uploaded 002 patch, fixed the testcase NPE error > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth > Attachments: YARN-7451.001.patch, YARN-7451.002.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 > } > } > ] > {noformat} > However, the web UI shows resource types usage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319350#comment-16319350 ] Szilard Nemeth commented on YARN-7451: -- Yep it compiled without any issues locally for me. I don' t have a clue how hadoop-yarn-services-api could be related to my change though. I will take a look tomorrow morning. > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth > Attachments: YARN-7451.001.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 > } > } > ] > {noformat} > However, the web UI shows resource types usage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319326#comment-16319326 ] Yufei Gu commented on YARN-7451: {code} [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project hadoop-yarn-services-api: Compilation failure -> [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn -rf :hadoop-yarn-services-api {code} The patch can't compile. Can you compile it locally? [~snemeth]. > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth > Attachments: YARN-7451.001.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 > } > } > ] > {noformat} > However, the web UI shows resource types usage. -- This
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16319110#comment-16319110 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 33 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 30s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 31m 36s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 15m 37s{color} | {color:red} root in trunk failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 23s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 20m 58s{color} | {color:red} branch has errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 50s{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 30s{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed. {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 31s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 29s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m 6s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 14m 6s{color} | {color:red} root generated 181 new + 1059 unchanged - 0 fixed = 1240 total (was 1059) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 11s{color} | {color:orange} root: The patch generated 153 new + 167 unchanged - 11 fixed = 320 total (was 178) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 54s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 61 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s{color} | {color:red} The patch 768 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 59s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 21s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 31s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 18s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red}
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318442#comment-16318442 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} docker {color} | {color:red} 11m 34s{color} | {color:red} Docker failed to build yetus/hadoop:5b98639. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-7451 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12905282/YARN-7451.001.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/19151/console | | Powered by | Apache Yetus 0.7.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Szilard Nemeth > Attachments: YARN-7451.001.patch, > YARN-7451__Expose_custom_resource_types_on_RM_scheduler_API_as_flattened_map01_02.patch > > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. > Additionally, when calling the RM scheduler API it returns: > {noformat} > "childQueues": { > "queue": [ > { > "allocatedContainers": 101, > "amMaxResources": { > "memory": 320390, > "vCores": 192 > }, > "amUsedResources": { > "memory": 1024, > "vCores": 1 > }, > "clusterResources": { > "memory": 640779, > "vCores": 384 > }, > "demandResources": { > "memory": 103424, > "vCores": 101 > }, > "fairResources": { > "memory": 640779, > "vCores": 384 > }, > "maxApps": 2147483647, > "maxResources": { > "memory": 640779, > "vCores": 384 > }, > "minResources": { > "memory": 0, > "vCores": 0 > }, > "numActiveApps": 1, > "numPendingApps": 0, > "preemptable": true, > "queueName": "root.users.systest", > "reservedContainers": 0, > "reservedResources": { > "memory": 0, > "vCores": 0 > }, > "schedulingPolicy": "fair", > "steadyFairResources": { > "memory": 320390, > "vCores": 192 > }, > "type": "fairSchedulerLeafQueueInfo", > "usedResources": { > "memory": 103424, > "vCores": 101 > } > } > ] > {noformat} > However, the web UI shows resource types usage. -- This message was sent by
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16318315#comment-16318315 ] genericqa commented on YARN-7451: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 33 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 29s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 32m 19s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 3m 0s{color} | {color:red} root in trunk failed. {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 13s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 22m 2s{color} | {color:red} branch has errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 49s{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 28s{color} | {color:red} hadoop-yarn-server-resourcemanager in trunk failed. {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 37s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 29s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 17m 23s{color} | {color:red} root in the patch failed. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 17m 23s{color} | {color:red} root in the patch failed. {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 2m 2s{color} | {color:orange} root: The patch generated 153 new + 167 unchanged - 11 fixed = 320 total (was 178) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 55 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s{color} | {color:red} The patch 768 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 4s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 13s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-project hadoop-client-modules/hadoop-client-minicluster hadoop-client-modules/hadoop-client-check-test-invariants {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 27s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s{color} | {color:green} hadoop-project in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 63m 22s{color} | {color:red}
[jira] [Commented] (YARN-7451) Resources Types should be visible in the Cluster Apps API "resourceRequests" section
[ https://issues.apache.org/jira/browse/YARN-7451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16261256#comment-16261256 ] Grant Sohn commented on YARN-7451: -- Based on discussions with [~templedf], this should be part of the capability map. > Resources Types should be visible in the Cluster Apps API "resourceRequests" > section > > > Key: YARN-7451 > URL: https://issues.apache.org/jira/browse/YARN-7451 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager, restapi >Affects Versions: 3.0.0 >Reporter: Grant Sohn >Assignee: Yufei Gu > > When running jobs that request resource types the RM Cluster Apps API should > include this in the "resourceRequests" object. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org