[jira] [Commented] (YARN-6741) Deleting all children of a Parent Queue on refresh throws exception
[ https://issues.apache.org/jira/browse/YARN-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122885#comment-16122885 ] Bibin A Chundatt commented on YARN-6741: [~naganarasimha...@apache.org] Could you recheck failed testcases. > Deleting all children of a Parent Queue on refresh throws exception > --- > > Key: YARN-6741 > URL: https://issues.apache.org/jira/browse/YARN-6741 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.0.0-alpha3 >Reporter: Naganarasimha G R >Assignee: Naganarasimha G R > Attachments: YARN-6741.001.patch, YARN-6741.002.patch, > YARN-6741.003.patch, YARN-6741.004.patch, YARN-6741.005.patch > > > If we configure CS such that all children of a parent queue are deleted and > made as a leaf queue, then {{refreshQueue}} operation fails when > re-initializing the parent Queue > {code} >// Sanity check > if (!(newlyParsedQueue instanceof ParentQueue) || !newlyParsedQueue > .getQueuePath().equals(getQueuePath())) { > throw new IOException( > "Trying to reinitialize " + getQueuePath() + " from " > + newlyParsedQueue.getQueuePath()); > } > {code} > *Expected Behavior:* > Converting a Parent Queue to leafQueue on refreshQueue needs to be supported. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-65) Reduce RM app memory footprint once app has completed
[ https://issues.apache.org/jira/browse/YARN-65?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122872#comment-16122872 ] Bibin A Chundatt commented on YARN-65: -- {quote} I think we can improve it setting AMContainerSpec to null rather than setting individual fields. {quote} We will loose ApplicationACL in that case. {{RMAppManager#createAndPopulateNewRMApp}} {code} // Inform the ACLs Manager this.applicationACLsManager.addApplication(applicationId, submissionContext.getAMContainerSpec().getApplicationACLs()); {code} Any idea why the applicationACL was set to AM container Launch Context?? > Reduce RM app memory footprint once app has completed > - > > Key: YARN-65 > URL: https://issues.apache.org/jira/browse/YARN-65 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 0.23.3 >Reporter: Jason Lowe >Assignee: Manikandan R > Attachments: YARN-65.001.patch, YARN-65.002.patch, YARN-65.003.patch, > YARN-65.004.patch, YARN-65.005.patch, YARN-65.006.patch, YARN-65.007.patch > > > The ResourceManager holds onto a configurable number of completed > applications (yarn.resource.max-completed-applications, defaults to 1), > and the memory footprint of these completed applications can be significant. > For example, the {{submissionContext}} in RMAppImpl contains references to > protocolbuffer objects and other items that probably aren't necessary to keep > around once the application has completed. We could significantly reduce the > memory footprint of the RM by releasing objects that are no longer necessary > once an application completes. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6727) Improve getQueueUserAcls API to query for specific queue and user
[ https://issues.apache.org/jira/browse/YARN-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bibin A Chundatt updated YARN-6727: --- Attachment: YARN-6727.003.patch Attaching updated patch > Improve getQueueUserAcls API to query for specific queue and user > -- > > Key: YARN-6727 > URL: https://issues.apache.org/jira/browse/YARN-6727 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: YARN-6727.001.patch, YARN-6727.002.patch, > YARN-6727.003.patch, YARN-6727.WIP.patch > > > Currently {{ApplicationClientProtocol#getQueueUserAcls}} return data for all > the queues available in scheduler for user. > User wants to know whether he has rights of a particular queue only. For > systems with 5K queues returning all queues list is not efficient. > Suggested change: support additional parameters *userName and queueName* as > optional. Admin user should be able to query other users ACL for a particular > queueName. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6903) Yarn-native-service framework core rewrite
[ https://issues.apache.org/jira/browse/YARN-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122846#comment-16122846 ] Hadoop QA commented on YARN-6903: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 114 new or modified test files. {color} | || || || || {color:brown} yarn-native-services Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 59s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 5s{color} | {color:green} yarn-native-services passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 32s{color} | {color:green} yarn-native-services passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 40s{color} | {color:green} yarn-native-services passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m 2s{color} | {color:green} yarn-native-services passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 50s{color} | {color:green} yarn-native-services passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 8s{color} | {color:green} yarn-native-services passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 44s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 5m 44s{color} | {color:red} hadoop-yarn-project_hadoop-yarn generated 3 new + 133 unchanged - 5 fixed = 136 total (was 138) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 35s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 581 new + 1569 unchanged - 405 fixed = 2150 total (was 1974) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 6m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 24s{color} | {color:green} There were no new shellcheck issues. {color} | | {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m 12s{color} | {color:green} There were no new shelldocs issues. {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 26 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s{color} | {color:red} The patch 1 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 5s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 4s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 35s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 19s{color} | {color:red} hadoop-yarn-slider in
[jira] [Commented] (YARN-6882) AllocationFileLoaderService.reloadAllocations() should use the diamond operator
[ https://issues.apache.org/jira/browse/YARN-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122827#comment-16122827 ] Larry Lo commented on YARN-6882: Hi, [~templedf]. It looks like not all unit tests passed. I have no idea why the minor changes cause this problem. Could you give me some suggestions to fix it ? > AllocationFileLoaderService.reloadAllocations() should use the diamond > operator > --- > > Key: YARN-6882 > URL: https://issues.apache.org/jira/browse/YARN-6882 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: Larry Lo >Priority: Trivial > Labels: newbie > Attachments: YARN-6882.001.patch > > > Here:{code}for (FSQueueType queueType : FSQueueType.values()) { > configuredQueues.put(queueType, new HashSet()); > }{code} and here:{code}List queueElements = new > ArrayList();{code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5328) InMemoryPlan enhancements required to support recurring reservations in the YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122812#comment-16122812 ] Hadoop QA commented on YARN-5328: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 47s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 14s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 16s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 53s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 9s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 28 new + 290 unchanged - 58 fixed = 318 total (was 348) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 3s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 32s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 37s{color} | {color:red} hadoop-yarn-api in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 23s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}106m 44s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields | | Timed out junit tests | org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-5328 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881377/YARN-5328-v7.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux d2023250e83f 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a32e013 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/16847/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt | | javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/16847/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hado
[jira] [Commented] (YARN-6969) Remove method getMinShareMemoryFraction and getPendingContainers in class FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122774#comment-16122774 ] Larry Lo commented on YARN-6969: Hi, [~yufeigu]. the findbugs show there are warnings about URF_UNREAD_FIELD. Could I just skip or have to consider to remove theses two unread fields ? > Remove method getMinShareMemoryFraction and getPendingContainers in class > FairSchedulerQueueInfo > > > Key: YARN-6969 > URL: https://issues.apache.org/jira/browse/YARN-6969 > Project: Hadoop YARN > Issue Type: Task > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Larry Lo >Priority: Trivial > Labels: newbie++ > Attachments: YARN-6969.001.patch > > > They are not used anymore. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6257) CapacityScheduler REST API produces incorrect JSON - JSON object operationsInfo contains deplicate key
[ https://issues.apache.org/jira/browse/YARN-6257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122755#comment-16122755 ] Tao Yang commented on YARN-6257: This problem was imported by YARN-3293 (2.8.0+). The operationsInfo can't be correctly used before as it's not follow JSON format. [~vvasudev], Please help to review this issue. > CapacityScheduler REST API produces incorrect JSON - JSON object > operationsInfo contains deplicate key > -- > > Key: YARN-6257 > URL: https://issues.apache.org/jira/browse/YARN-6257 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.8.1 >Reporter: Tao Yang >Assignee: Tao Yang >Priority: Minor > Attachments: YARN-6257.001.patch > > > In response string of CapacityScheduler REST API, > scheduler/schedulerInfo/health/operationsInfo have duplicate key 'entry' as a > JSON object : > {code} > "operationsInfo":{ > > "entry":{"key":"last-preemption","value":{"nodeId":"N/A","containerId":"N/A","queue":"N/A"}}, > > "entry":{"key":"last-reservation","value":{"nodeId":"N/A","containerId":"N/A","queue":"N/A"}}, > > "entry":{"key":"last-allocation","value":{"nodeId":"N/A","containerId":"N/A","queue":"N/A"}}, > > "entry":{"key":"last-release","value":{"nodeId":"N/A","containerId":"N/A","queue":"N/A"}} > } > {code} > To solve this problem, I suppose the type of operationsInfo field in > CapacitySchedulerHealthInfo class should be converted from Map to List. > After convert to List, The operationsInfo string will be: > {code} > "operationInfos":[ > > {"operation":"last-allocation","nodeId":"N/A","containerId":"N/A","queue":"N/A"}, > > {"operation":"last-release","nodeId":"N/A","containerId":"N/A","queue":"N/A"}, > > {"operation":"last-preemption","nodeId":"N/A","containerId":"N/A","queue":"N/A"}, > > {"operation":"last-reservation","nodeId":"N/A","containerId":"N/A","queue":"N/A"} > ] > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6885) AllocationFileLoaderService.loadQueue() should use a switch statement in the main tag parsing loop instead of the if/else-if/...
[ https://issues.apache.org/jira/browse/YARN-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122737#comment-16122737 ] Yu-Tang Lin commented on YARN-6885: --- Hi [#Daniel Templeton], thanks for the comment! I put the text and val object outside the switch case due to I can't reuse the same variable name just like the previous code in each case section, but just like you mentioned, it'll cost more to cast the variable. About the comment 4 and 5, I tried the treat these two cases just like other cases in previous patch, but the finding-bugs plugin showed that the boxing issues might be occurred, so I added the brace and tried to use primitive variables. Could you please give me some suggestions? And I didn't get the comment 6, do you mean the code below should be put in the default case section? {code:java} if (isLeaf && !"parent".equals(element.getAttribute("type"))) { configuredQueues.get(FSQueueType.LEAF).add(queueName); } else { if (isReservable) { throw new AllocationConfigurationException("The configuration settings" + " for " + queueName + " are invalid. A queue element that " + "contains child queue elements or that has the type='parent' " + "attribute cannot also include a reservation element."); } configuredQueues.get(FSQueueType.PARENT).add(queueName); } {code} > AllocationFileLoaderService.loadQueue() should use a switch statement in the > main tag parsing loop instead of the if/else-if/... > > > Key: YARN-6885 > URL: https://issues.apache.org/jira/browse/YARN-6885 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: Yu-Tang Lin >Priority: Minor > Labels: newbie > Fix For: 3.0.0-alpha4 > > Attachments: YARN-6885.005.patch > > > {code} if ("minResources".equals(field.getTagName())) { > String text = ((Text)field.getFirstChild()).getData().trim(); > Resource val = > FairSchedulerConfiguration.parseResourceConfigValue(text); > minQueueResources.put(queueName, val); > } else if ("maxResources".equals(field.getTagName())) { > ...{code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5328) InMemoryPlan enhancements required to support recurring reservations in the YARN ReservationSystem
[ https://issues.apache.org/jira/browse/YARN-5328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Subru Krishnan updated YARN-5328: - Attachment: YARN-5328-v7.patch Thanks [~curino] for the fixes for the wrap-around reservations. I have uploaded a rebased and cleaned up patch. > InMemoryPlan enhancements required to support recurring reservations in the > YARN ReservationSystem > -- > > Key: YARN-5328 > URL: https://issues.apache.org/jira/browse/YARN-5328 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Subru Krishnan >Assignee: Subru Krishnan > Attachments: YARN-5328-v1.patch, YARN-5328-v2.patch, > YARN-5328-v3.patch, YARN-5328-v4.patch, YARN-5328-v5.patch, > YARN-5328-v6.patch, YARN-5328-v7.patch > > > YARN-5326 proposes adding native support for recurring reservations in the > YARN ReservationSystem. This JIRA is a sub-task to track the changes required > in InMemoryPlan to accomplish it. Please refer to the design doc in the > parent JIRA for details. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6967) Limit application attempt's diagnostic message size thoroughly
[ https://issues.apache.org/jira/browse/YARN-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122652#comment-16122652 ] Chengbing Liu commented on YARN-6967: - [~templedf] [~andras.piros] Could you review the patch again? Thanks! > Limit application attempt's diagnostic message size thoroughly > -- > > Key: YARN-6967 > URL: https://issues.apache.org/jira/browse/YARN-6967 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 2.8.1 >Reporter: Chengbing Liu >Assignee: Chengbing Liu > Attachments: YARN-6967.01.patch > > > YARN-6125 implemented {{BoundedAppender}} and applied to the field > {{diagnostics}} to limit the diagnostic message's size. > However, some code bypasses this limit. In > {{RMAppAttemptImpl.rememberTargetTransitionsAndStoreState(...)}}, a local > variable {{diags}} will finally be written into ZooKeeper if ZKRMStateStore > is used. > A simple fix is to also use {{BoundedAppender}} for the local variable. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6980) The YARN_TIMELINE_HEAPSIZE does not work
[ https://issues.apache.org/jira/browse/YARN-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinjiang Ling updated YARN-6980: Attachment: YARN-6980-1.patch Thanks for your advice, [~aw]. Now I set both the Xms and Xmx in YARN_TIMELINESERVER_OPTS and it works, thank you. I update the patch and changed it in yarn-env.sh back to YARN_TIMELINESERVER_HEAPSIZE, because I think it would cause some misleading. > The YARN_TIMELINE_HEAPSIZE does not work > > > Key: YARN-6980 > URL: https://issues.apache.org/jira/browse/YARN-6980 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Jinjiang Ling >Assignee: Jinjiang Ling >Priority: Minor > Attachments: YARN-6980-1.patch, YARN-6980.patch > > > As I want to set the java heap size of timeline server, I find there is a > variable named YARN_TIMELINE_HEAPSIZE in the comments of yarn-env.sh, then I > set it but does not work. > Then I find the variable is changed in > -[HADOOP-10950|https://issues.apache.org/jira/browse/HADOOP-10950]-, > but it is just changed in the comments of yarn-env.sh and don't in "yarn" and > "yarn.cmd" command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6896) Federation: routing REST invocations transparently to multiple RMs (part 1 - basic execution)
[ https://issues.apache.org/jira/browse/YARN-6896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122632#comment-16122632 ] Hadoop QA commented on YARN-6896: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 6 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 45s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 0s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 51s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 214 unchanged - 0 fixed = 215 total (was 214) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 7s{color} | {color:green} hadoop-yarn-server-router in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 27s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 49m 20s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6896 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881355/YARN-6896.v3.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux e541370e29e1 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a32e013 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/16845/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/16845/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16845/console | | Powered by | A
[jira] [Commented] (YARN-6958) Moving logging APIs over to slf4j in hadoop-yarn-server-timelineservice
[ https://issues.apache.org/jira/browse/YARN-6958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122620#comment-16122620 ] Yeliang Cang commented on YARN-6958: Thanks for the review, [~ajisakaa]! > Moving logging APIs over to slf4j in hadoop-yarn-server-timelineservice > --- > > Key: YARN-6958 > URL: https://issues.apache.org/jira/browse/YARN-6958 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yeliang Cang >Assignee: Yeliang Cang > Fix For: 2.9.0, 3.0.0-beta1 > > Attachments: YARN-6958.001.patch, YARN-6958.002.patch, > YARN-6958-branch-2.001.patch > > > This jira moves logging APIS over to slf4j in the following modules: > {code} > hadoop-yarn-server-timeline-pluginstorage > hadoop-yarn-server-timelineservice > hadoop-yarn-server-timelineservice-hbase > hadoop-yarn-server-timelineservice-hbase-tests > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6903) Yarn-native-service framework core rewrite
[ https://issues.apache.org/jira/browse/YARN-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-6903: -- Attachment: YARN-6903.yarn-native-services.07.patch Not sure why the TestYarnNativeService always timed out on jenkins. Uploaded a new patch which commented out the old tests and only run the new ones. > Yarn-native-service framework core rewrite > -- > > Key: YARN-6903 > URL: https://issues.apache.org/jira/browse/YARN-6903 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-6903.yarn-native-services.01.patch, > YARN-6903.yarn-native-services.02.patch, > YARN-6903.yarn-native-services.03.patch, > YARN-6903.yarn-native-services.04.patch, > YARN-6903.yarn-native-services.05.patch, > YARN-6903.yarn-native-services.06.patch, > YARN-6903.yarn-native-services.07.patch > > > There are some new features like rich placement scheduling, container auto > restart, container upgrade in YARN core that can be taken advantage by the > native-service framework. Besides, there are quite a lot legacy code which > are no longer required. > So we decide to rewrite the core part to have a leaner codebase and make use > of various advanced features in YARN. > And the new code design will be in align with what we have designed for the > service API YARN-4793 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6905) Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat
[ https://issues.apache.org/jira/browse/YARN-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122571#comment-16122571 ] Vrushali C commented on YARN-6905: -- thanks [~haibo.chen], patch 001 looks good to me. There is a minor checkstyle line length issue, which can be fixed during commit. I will wait for a day or two to give others a chance to review before committing it. > Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat > --- > > Key: YARN-6905 > URL: https://issues.apache.org/jira/browse/YARN-6905 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 3.0.0-beta1 > Environment: Ubuntu 14.04 > x86, ppc64le > $ java -version > openjdk version "1.8.0_111" > OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14) > OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode) >Reporter: Sonia Garudi >Assignee: Haibo Chen > Attachments: YARN-6905.00.patch, YARN-6905.01.patch > > > There are multiple test failing in Hadoop YARN Timeline Service HBase tests > project with the following error : > {code} > java.lang.NoClassDefFoundError: org/apache/hadoop/util/FastNumberFormat > at > org.apache.hadoop.yarn.api.records.ApplicationId.toString(ApplicationId.java:104) > {code} > Below are the failing tests : > {code} > TestHBaseTimelineStorageApps.testWriteApplicationToHBase > TestHBaseTimelineStorageApps.testEvents > TestHBaseTimelineStorageEntities.testEventsEscapeTs > TestHBaseTimelineStorageEntities.testWriteEntityToHBase > TestHBaseTimelineStorageEntities.testEventsWithEmptyInfo > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6896) Federation: routing REST invocations transparently to multiple RMs (part 1 - basic execution)
[ https://issues.apache.org/jira/browse/YARN-6896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122558#comment-16122558 ] Giovanni Matteo Fumarola edited comment on YARN-6896 at 8/10/17 11:54 PM: -- Attached V3 that addressed your feedback. I fixed in a different way from the one suggested. Each {{MockDefaultFederationInterceptorREST}} has its own {{SubClusterId}}. * CreateNewApplication returns an applicationId that depends on the {{SubClusterId}}. In my tests I validate the ApplicationId timestamp belongs to one of the subclusters in my test. * SubmitApplication returns a response with its own {{SubClusterId}}. In my tests I validate the response has the same subclusterId from the one saved in the StateStore. During the submission I added the application submitted in a list inside the mock class. * GetApplication and KillApplication have a check if the application was submitted in that subcluster. If not, the mock class throws an exception. In my test I validate I do not get any exception and I get a positive responses. I validate the patch in a small test cluster. was (Author: giovanni.fumarola): Attached V3 that addressed your feedback. I fixed in a different way from the one suggested. Each {{MockDefaultFederationInterceptorREST}} has its own {{SubClusterId}}. * CreateNewApplication returns an applicationId that depends on the {{SubClusterId}}. In my tests I validate the ApplicationId timestamp belongs to one of the subclusters in my test. * SubmitApplication returns a response with its own {{SubClusterId}}. In my tests I validate the response has the same subclusterId from the one saved in the StateStore. During the submission I added the application submitted in a list inside the mock class. * GetApplication and KillApplication have a check if the application was submitted in that subcluster. If not, the mock class throws an exception. In my test I validate I do not get any exception and I get a positive responses. > Federation: routing REST invocations transparently to multiple RMs (part 1 - > basic execution) > - > > Key: YARN-6896 > URL: https://issues.apache.org/jira/browse/YARN-6896 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-6896.proto.patch, YARN-6896.v1.patch, > YARN-6896.v2.patch, YARN-6896.v3.patch > > > This JIRA tracks the design/implementation of the layer for routing > RMWebServicesProtocol requests to the appropriate RM(s) in a federated YARN > cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6896) Federation: routing REST invocations transparently to multiple RMs (part 1 - basic execution)
[ https://issues.apache.org/jira/browse/YARN-6896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122558#comment-16122558 ] Giovanni Matteo Fumarola commented on YARN-6896: Attached V3 that addressed your feedback. I fixed in a different way from the one suggested. Each {{MockDefaultFederationInterceptorREST}} has its own {{SubClusterId}}. * CreateNewApplication returns an applicationId that depends on the {{SubClusterId}}. In my tests I validate the ApplicationId timestamp belongs to one of the subclusters in my test. * SubmitApplication returns a response with its own {{SubClusterId}}. In my tests I validate the response has the same subclusterId from the one saved in the StateStore. During the submission I added the application submitted in a list inside the mock class. * GetApplication and KillApplication have a check if the application was submitted in that subcluster. If not, the mock class throws an exception. In my test I validate I do not get any exception and I get a positive responses. > Federation: routing REST invocations transparently to multiple RMs (part 1 - > basic execution) > - > > Key: YARN-6896 > URL: https://issues.apache.org/jira/browse/YARN-6896 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-6896.proto.patch, YARN-6896.v1.patch, > YARN-6896.v2.patch, YARN-6896.v3.patch > > > This JIRA tracks the design/implementation of the layer for routing > RMWebServicesProtocol requests to the appropriate RM(s) in a federated YARN > cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6896) Federation: routing REST invocations transparently to multiple RMs (part 1 - basic execution)
[ https://issues.apache.org/jira/browse/YARN-6896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-6896: --- Attachment: YARN-6896.v3.patch > Federation: routing REST invocations transparently to multiple RMs (part 1 - > basic execution) > - > > Key: YARN-6896 > URL: https://issues.apache.org/jira/browse/YARN-6896 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-6896.proto.patch, YARN-6896.v1.patch, > YARN-6896.v2.patch, YARN-6896.v3.patch > > > This JIRA tracks the design/implementation of the layer for routing > RMWebServicesProtocol requests to the appropriate RM(s) in a federated YARN > cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6990) AmIpFilter:findRedirectUrl use HAServiceProtocol to getServiceStatus
[ https://issues.apache.org/jira/browse/YARN-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yunjiong zhao resolved YARN-6990. - Resolution: Duplicate Assignee: yunjiong zhao > AmIpFilter:findRedirectUrl use HAServiceProtocol to getServiceStatus > > > Key: YARN-6990 > URL: https://issues.apache.org/jira/browse/YARN-6990 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > > Due to we have multiple IP in ResourceManager, when try to access proxy URL > like https://*:50030/proxy/application_1502349494018_10877/, it will failed > due to it use HAServiceProtocol to find out which one is active RM. > {code} > 2017-08-10 10:51:42,344 WARN [971256592@qtp-666312528-0] > org.apache.hadoop.ipc.Client: Exception encountered while connecting to the > server : > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[KERBEROS] > at > org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172) > at > org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396) > at > org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:563) > at > org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:378) > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:732) > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:728) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:727) > at > org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:378) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1492) > at org.apache.hadoop.ipc.Client.call(Client.java:1402) > at org.apache.hadoop.ipc.Client.call(Client.java:1363) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) > at com.sun.proxy.$Proxy84.getServiceStatus(Unknown Source) > at > org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:122) > at org.apache.hadoop.yarn.util.RMHAUtils.getHAState(RMHAUtils.java:68) > at > org.apache.hadoop.yarn.util.RMHAUtils.findActiveRMHAId(RMHAUtils.java:44) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:174) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:138) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1243) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > at > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > {code} > This only can happen on RM have multiple IPs, related code is inside > AmIpFilter.java doFilter function: > {code} > if (!getProxyAddresses().contains(httpReq.getRemoteAddr(
[jira] [Commented] (YARN-6990) AmIpFilter:findRedirectUrl use HAServiceProtocol to getServiceStatus
[ https://issues.apache.org/jira/browse/YARN-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122538#comment-16122538 ] yunjiong zhao commented on YARN-6990: - I just find YARN-6625 fixed the issue. We use 2.7. > AmIpFilter:findRedirectUrl use HAServiceProtocol to getServiceStatus > > > Key: YARN-6990 > URL: https://issues.apache.org/jira/browse/YARN-6990 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: yunjiong zhao >Assignee: yunjiong zhao > > Due to we have multiple IP in ResourceManager, when try to access proxy URL > like https://*:50030/proxy/application_1502349494018_10877/, it will failed > due to it use HAServiceProtocol to find out which one is active RM. > {code} > 2017-08-10 10:51:42,344 WARN [971256592@qtp-666312528-0] > org.apache.hadoop.ipc.Client: Exception encountered while connecting to the > server : > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[KERBEROS] > at > org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172) > at > org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396) > at > org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:563) > at > org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:378) > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:732) > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:728) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:727) > at > org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:378) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1492) > at org.apache.hadoop.ipc.Client.call(Client.java:1402) > at org.apache.hadoop.ipc.Client.call(Client.java:1363) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) > at com.sun.proxy.$Proxy84.getServiceStatus(Unknown Source) > at > org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:122) > at org.apache.hadoop.yarn.util.RMHAUtils.getHAState(RMHAUtils.java:68) > at > org.apache.hadoop.yarn.util.RMHAUtils.findActiveRMHAId(RMHAUtils.java:44) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:174) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:138) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1243) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > at > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > {code} > This only can happen on RM have multiple IPs, related code is inside > AmIpFilter.java doFilter function: > {code} > if (!
[jira] [Updated] (YARN-6990) AmIpFilter:findRedirectUrl use HAServiceProtocol to getServiceStatus
[ https://issues.apache.org/jira/browse/YARN-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yunjiong zhao updated YARN-6990: Affects Version/s: 2.7.0 > AmIpFilter:findRedirectUrl use HAServiceProtocol to getServiceStatus > > > Key: YARN-6990 > URL: https://issues.apache.org/jira/browse/YARN-6990 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: yunjiong zhao > > Due to we have multiple IP in ResourceManager, when try to access proxy URL > like https://*:50030/proxy/application_1502349494018_10877/, it will failed > due to it use HAServiceProtocol to find out which one is active RM. > {code} > 2017-08-10 10:51:42,344 WARN [971256592@qtp-666312528-0] > org.apache.hadoop.ipc.Client: Exception encountered while connecting to the > server : > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[KERBEROS] > at > org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172) > at > org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396) > at > org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:563) > at > org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:378) > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:732) > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:728) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:727) > at > org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:378) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1492) > at org.apache.hadoop.ipc.Client.call(Client.java:1402) > at org.apache.hadoop.ipc.Client.call(Client.java:1363) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) > at com.sun.proxy.$Proxy84.getServiceStatus(Unknown Source) > at > org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:122) > at org.apache.hadoop.yarn.util.RMHAUtils.getHAState(RMHAUtils.java:68) > at > org.apache.hadoop.yarn.util.RMHAUtils.findActiveRMHAId(RMHAUtils.java:44) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:174) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:138) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1243) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > at > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > {code} > This only can happen on RM have multiple IPs, related code is inside > AmIpFilter.java doFilter function: > {code} > if (!getProxyAddresses().contains(httpReq.getRemoteAddr())) { > String redirectUrl = findRedirectUrl(); > Stri
[jira] [Created] (YARN-6992) "Kill application" button is present even if the application is FINISHED in RM UI
Sumana Sathish created YARN-6992: Summary: "Kill application" button is present even if the application is FINISHED in RM UI Key: YARN-6992 URL: https://issues.apache.org/jira/browse/YARN-6992 Project: Hadoop YARN Issue Type: Bug Reporter: Sumana Sathish Assignee: Suma Shivaprasad -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6990) AmIpFilter:findRedirectUrl use HAServiceProtocol to getServiceStatus
[ https://issues.apache.org/jira/browse/YARN-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122534#comment-16122534 ] Yufei Gu commented on YARN-6990: Which version do you use? After YARN-6625, AmIpFilter doesn't use HAServiceProtocol to find the active RM. > AmIpFilter:findRedirectUrl use HAServiceProtocol to getServiceStatus > > > Key: YARN-6990 > URL: https://issues.apache.org/jira/browse/YARN-6990 > Project: Hadoop YARN > Issue Type: Bug >Reporter: yunjiong zhao > > Due to we have multiple IP in ResourceManager, when try to access proxy URL > like https://*:50030/proxy/application_1502349494018_10877/, it will failed > due to it use HAServiceProtocol to find out which one is active RM. > {code} > 2017-08-10 10:51:42,344 WARN [971256592@qtp-666312528-0] > org.apache.hadoop.ipc.Client: Exception encountered while connecting to the > server : > org.apache.hadoop.security.AccessControlException: Client cannot authenticate > via:[KERBEROS] > at > org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172) > at > org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396) > at > org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:563) > at > org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:378) > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:732) > at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:728) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:422) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) > at > org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:727) > at > org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:378) > at org.apache.hadoop.ipc.Client.getConnection(Client.java:1492) > at org.apache.hadoop.ipc.Client.call(Client.java:1402) > at org.apache.hadoop.ipc.Client.call(Client.java:1363) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) > at com.sun.proxy.$Proxy84.getServiceStatus(Unknown Source) > at > org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:122) > at org.apache.hadoop.yarn.util.RMHAUtils.getHAState(RMHAUtils.java:68) > at > org.apache.hadoop.yarn.util.RMHAUtils.findActiveRMHAId(RMHAUtils.java:44) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:174) > at > org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:138) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1243) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) > at > org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) > at > org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) > at > org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) > at > org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) > at > org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) > at > org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) > at > org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) > at > org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) > at org.mortbay.jetty.Server.handle(Server.java:326) > at > org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) > at > org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) > at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) > at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) > at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) > at > org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) > at > org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) > {code} > This only can happen on RM have multiple IPs, related code is inside > AmIpFilter.java doFilter function: > {code} > if (!getProxyAddresses()
[jira] [Created] (YARN-6991) "Kill application" button does not show error if other user tries to kill the application for secure cluster
Sumana Sathish created YARN-6991: Summary: "Kill application" button does not show error if other user tries to kill the application for secure cluster Key: YARN-6991 URL: https://issues.apache.org/jira/browse/YARN-6991 Project: Hadoop YARN Issue Type: Bug Reporter: Sumana Sathish Assignee: Suma Shivaprasad 1. Submit an application by user 1 2. log into RM UI as user 2 3. Kill the application submitted by user 1 4. Even though application does not get killed, there is no error/info dialog box being shown to let the user that "user doesnot have permissions to kill application of other user" -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6990) AmIpFilter:findRedirectUrl use HAServiceProtocol to getServiceStatus
yunjiong zhao created YARN-6990: --- Summary: AmIpFilter:findRedirectUrl use HAServiceProtocol to getServiceStatus Key: YARN-6990 URL: https://issues.apache.org/jira/browse/YARN-6990 Project: Hadoop YARN Issue Type: Bug Reporter: yunjiong zhao Due to we have multiple IP in ResourceManager, when try to access proxy URL like https://*:50030/proxy/application_1502349494018_10877/, it will failed due to it use HAServiceProtocol to find out which one is active RM. {code} 2017-08-10 10:51:42,344 WARN [971256592@qtp-666312528-0] org.apache.hadoop.ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[KERBEROS] at org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172) at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396) at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:563) at org.apache.hadoop.ipc.Client$Connection.access$1900(Client.java:378) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:732) at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:728) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:727) at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:378) at org.apache.hadoop.ipc.Client.getConnection(Client.java:1492) at org.apache.hadoop.ipc.Client.call(Client.java:1402) at org.apache.hadoop.ipc.Client.call(Client.java:1363) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) at com.sun.proxy.$Proxy84.getServiceStatus(Unknown Source) at org.apache.hadoop.ha.protocolPB.HAServiceProtocolClientSideTranslatorPB.getServiceStatus(HAServiceProtocolClientSideTranslatorPB.java:122) at org.apache.hadoop.yarn.util.RMHAUtils.getHAState(RMHAUtils.java:68) at org.apache.hadoop.yarn.util.RMHAUtils.findActiveRMHAId(RMHAUtils.java:44) at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.findRedirectUrl(AmIpFilter.java:174) at org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter.doFilter(AmIpFilter.java:138) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1243) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) {code} This only can happen on RM have multiple IPs, related code is inside AmIpFilter.java doFilter function: {code} if (!getProxyAddresses().contains(httpReq.getRemoteAddr())) { String redirectUrl = findRedirectUrl(); String target = redirectUrl + httpReq.getRequestURI(); ProxyUtils.sendRedirect(httpReq, httpResp, target); return; } {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6903) Yarn-native-service framework core rewrite
[ https://issues.apache.org/jira/browse/YARN-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122494#comment-16122494 ] Hadoop QA commented on YARN-6903: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 62 new or modified test files. {color} | || || || || {color:brown} yarn-native-services Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 2s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 57s{color} | {color:green} yarn-native-services passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m 38s{color} | {color:green} yarn-native-services passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 46s{color} | {color:green} yarn-native-services passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m 33s{color} | {color:green} yarn-native-services passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 51s{color} | {color:green} yarn-native-services passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 16s{color} | {color:green} yarn-native-services passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 7m 0s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 7m 0s{color} | {color:red} hadoop-yarn-project_hadoop-yarn generated 3 new + 133 unchanged - 5 fixed = 136 total (was 138) {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 44s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 518 new + 1523 unchanged - 405 fixed = 2041 total (was 1928) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 8m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m 26s{color} | {color:green} There were no new shellcheck issues. {color} | | {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m 13s{color} | {color:green} There were no new shelldocs issues. {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s{color} | {color:red} The patch has 26 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 2s{color} | {color:red} The patch 1 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 6s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 23s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-slider/hadoop-yarn-slider-core generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 40s{color} | {color:red} hadoop-yarn in the patch failed. {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 23s{color} | {color:red} hadoop-yarn-slider in t
[jira] [Commented] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122474#comment-16122474 ] Hadoop QA commented on YARN-6668: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color} | {color:red} YARN-6668 does not apply to YARN-1011. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-6668 | | GITHUB PR | https://github.com/apache/hadoop/pull/241 | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16844/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > Attachments: YARN-6668.000.patch, YARN-6668.001.patch, > YARN-6668.002.patch, YARN-6668.003.patch, YARN-6668.004.patch, > YARN-6668.005.patch, YARN-6668.006.patch, YARN-6668.007.patch, > YARN-6668.008.patch > > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6905) Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat
[ https://issues.apache.org/jira/browse/YARN-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122462#comment-16122462 ] Hadoop QA commented on YARN-6905: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 18m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 57s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 59s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 48s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1 new + 6 unchanged - 0 fixed = 7 total (was 6) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m 14s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 43m 9s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6905 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881338/YARN-6905.01.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 50db3c34a5fd 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | |
[jira] [Updated] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-6668: - Attachment: YARN-6668.008.patch > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > Attachments: YARN-6668.000.patch, YARN-6668.001.patch, > YARN-6668.002.patch, YARN-6668.003.patch, YARN-6668.004.patch, > YARN-6668.005.patch, YARN-6668.006.patch, YARN-6668.007.patch, > YARN-6668.008.patch > > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-6668: - Attachment: (was: YARN-6668.008.patch) > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > Attachments: YARN-6668.000.patch, YARN-6668.001.patch, > YARN-6668.002.patch, YARN-6668.003.patch, YARN-6668.004.patch, > YARN-6668.005.patch, YARN-6668.006.patch, YARN-6668.007.patch > > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-6668: - Attachment: YARN-6668.008.patch > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > Attachments: YARN-6668.000.patch, YARN-6668.001.patch, > YARN-6668.002.patch, YARN-6668.003.patch, YARN-6668.004.patch, > YARN-6668.005.patch, YARN-6668.006.patch, YARN-6668.007.patch, > YARN-6668.008.patch > > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122437#comment-16122437 ] Hadoop QA commented on YARN-6668: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color} | {color:red} YARN-6668 does not apply to YARN-1011. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-6668 | | GITHUB PR | https://github.com/apache/hadoop/pull/241 | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16843/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > Attachments: YARN-6668.000.patch, YARN-6668.001.patch, > YARN-6668.002.patch, YARN-6668.003.patch, YARN-6668.004.patch, > YARN-6668.005.patch, YARN-6668.006.patch, YARN-6668.007.patch > > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6989) Ensure timeline service v2 codebase gets UGI from HttpServletRequest in a consistent way
Vrushali C created YARN-6989: Summary: Ensure timeline service v2 codebase gets UGI from HttpServletRequest in a consistent way Key: YARN-6989 URL: https://issues.apache.org/jira/browse/YARN-6989 Project: Hadoop YARN Issue Type: Sub-task Reporter: Vrushali C As noticed during discussions in YARN-6820, the webservices in timeline service v2 get the UGI created from the user obtained by invoking getRemoteUser on the HttpServletRequest . It will be good to use getUserPrincipal instead of invoking getRemoteUser on the HttpServletRequest. Filing jira to update the code. Per Java EE documentations for 6 and 7, the behavior around getRemoteUser and getUserPrincipal is listed at: http://docs.oracle.com/javaee/6/tutorial/doc/gjiie.html#bncba https://docs.oracle.com/javaee/7/tutorial/security-webtier003.htm {code} getRemoteUser, which determines the user name with which the client authenticated. The getRemoteUser method returns the name of the remote user (the caller) associated by the container with the request. If no user has been authenticated, this method returns null. getUserPrincipal, which determines the principal name of the current user and returns a java.security.Principal object. If no user has been authenticated, this method returns null. Calling the getName method on the Principal returned by getUserPrincipal returns the name of the remote user. {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-6668: - Attachment: (was: YARN-6668.008.patch) > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > Attachments: YARN-6668.000.patch, YARN-6668.001.patch, > YARN-6668.002.patch, YARN-6668.003.patch, YARN-6668.004.patch, > YARN-6668.005.patch, YARN-6668.006.patch, YARN-6668.007.patch > > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6820) Restrict read access to timelineservice v2 data
[ https://issues.apache.org/jira/browse/YARN-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122427#comment-16122427 ] Vrushali C commented on YARN-6820: -- Thanks [~jlowe] , filed YARN-6989 > Restrict read access to timelineservice v2 data > > > Key: YARN-6820 > URL: https://issues.apache.org/jira/browse/YARN-6820 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: yarn-5355-merge-blocker > Attachments: YARN-6820-YARN-5355.0001.patch, > YARN-6820-YARN-5355.002.patch, YARN-6820-YARN-5355.003.patch, > YARN-6820-YARN-5355.004.patch, YARN-6820-YARN-5355.005.patch > > > Need to provide a way to restrict read access in ATSv2. Not all users should > be able to read all entities. On the flip side, some folks may not need any > read restrictions, so we need to provide a way to disable this access > restriction as well. > Initially this access restriction could be done in a simple way via a > whitelist of users allowed to read data. That set of users can read all data, > no other user can read any data. Can be turned off for all users to read all > data. > Could be stored in a "domain" table in hbase perhaps. Or a configuration > setting for the cluster. Or something else that's simple enough. ATSv1 has a > concept of domain for isolating users for reading. Would be good to keep that > in consideration. > In ATSv1, domain offers a namespace for Timeline server allowing users to > host multiple entities, isolating them from other users and applications. A > “Domain” in ATSV1 primarily stores owner info, read and& write ACL > information, created and modified time stamp information. Each Domain is > identified by an ID which must be unique across all users in the YARN cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6987) Log app attempt during InvalidStateTransition
[ https://issues.apache.org/jira/browse/YARN-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122425#comment-16122425 ] Jonathan Eagles commented on YARN-6987: --- Test failures are unrelated to this patch. > Log app attempt during InvalidStateTransition > - > > Key: YARN-6987 > URL: https://issues.apache.org/jira/browse/YARN-6987 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: YARN-6987.001.patch > > > Found this InvalidStateTransition logged in the resource manager log file > with no way to determine exactly which app attempt this was associated with. > {noformat} > 2017-08-04 17:22:29,895 [AsyncDispatcher event handler] ERROR > attempt.RMAppAttemptImpl: Can't handle this event at current state > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > UNREGISTERED at LAUNCHED > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:802) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:108) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:803) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:784) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6668) Use cgroup to get container resource utilization
[ https://issues.apache.org/jira/browse/YARN-6668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Miklos Szegedi updated YARN-6668: - Attachment: YARN-6668.008.patch > Use cgroup to get container resource utilization > > > Key: YARN-6668 > URL: https://issues.apache.org/jira/browse/YARN-6668 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager >Affects Versions: 3.0.0-alpha3 >Reporter: Haibo Chen >Assignee: Miklos Szegedi > Attachments: YARN-6668.000.patch, YARN-6668.001.patch, > YARN-6668.002.patch, YARN-6668.003.patch, YARN-6668.004.patch, > YARN-6668.005.patch, YARN-6668.006.patch, YARN-6668.007.patch, > YARN-6668.008.patch > > > Container Monitor relies on proc file system to get container resource > utilization, which is not as efficient as reading cgroup accounting. We > should in NM, when cgroup is enabled, read cgroup stats instead. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6834) A container request with only racks specified and relax locality set to false is never honoured
[ https://issues.apache.org/jira/browse/YARN-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122403#comment-16122403 ] Hadoop QA commented on YARN-6834: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 57s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 36s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 40s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 55s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 44 unchanged - 0 fixed = 48 total (was 44) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 43m 6s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 30s{color} | {color:green} hadoop-yarn-server-tests in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m 36s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}115m 55s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue | | | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6834 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/128813
[jira] [Commented] (YARN-6851) Capacity Scheduler: document configs for controlling # containers allowed to be allocated per node heartbeat
[ https://issues.apache.org/jira/browse/YARN-6851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122389#comment-16122389 ] Hadoop QA commented on YARN-6851: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m 52s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 41m 10s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6851 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881274/YARN-6851.001.patch | | Optional Tests | asflicense mvnsite | | uname | Linux 4410b1f7f4e9 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16841/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Capacity Scheduler: document configs for controlling # containers allowed to > be allocated per node heartbeat > > > Key: YARN-6851 > URL: https://issues.apache.org/jira/browse/YARN-6851 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Minor > Attachments: YARN-6851.001.patch > > > YARN-4161 introduces new configs for controlling how many containers allowed > to be allocated in each node heartbeat. And we also have offswitchCount > config before. Would be better to document these configurations in CS section. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122382#comment-16122382 ] Hadoop QA commented on YARN-6852: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 19m 0s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 39s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 25 new + 0 unchanged - 0 fixed = 25 total (was 0) {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 6s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 49m 48s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6852 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881319/YARN-6852.007.patch | | Optional Tests | asflicense compile cc mvnsite javac unit | | uname | Linux e62e4e5b9086 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | Default Java | 1.8.0_144 | | cc | https://builds.apache.org/job/PreCommit-YARN-Build/16840/artifact/patchprocess/diff-compile-cc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/16840/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16840/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > [YARN-6223] Native code changes to support isolate GPU devices by using > CGroups > --- > > Key: YARN-6852 > URL: https://issues.apache.org/jira/browse/YARN-6852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6852.001.patch, YARN-6852.002.patch, > YARN-6852.003.patch, YARN-6852.004.patch, YARN-6852.005.patch, > YARN-6852.006.patch, YARN-6852.007.patch > > > This JIRA plan to add support of: > 1) Isolation in CGroups. (native side). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6905) Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat
[ https://issues.apache.org/jira/browse/YARN-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haibo Chen updated YARN-6905: - Attachment: YARN-6905.01.patch Thanks for the review, [~vrushalic]. I have updated the patch accordingly > Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat > --- > > Key: YARN-6905 > URL: https://issues.apache.org/jira/browse/YARN-6905 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 3.0.0-beta1 > Environment: Ubuntu 14.04 > x86, ppc64le > $ java -version > openjdk version "1.8.0_111" > OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14) > OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode) >Reporter: Sonia Garudi >Assignee: Haibo Chen > Attachments: YARN-6905.00.patch, YARN-6905.01.patch > > > There are multiple test failing in Hadoop YARN Timeline Service HBase tests > project with the following error : > {code} > java.lang.NoClassDefFoundError: org/apache/hadoop/util/FastNumberFormat > at > org.apache.hadoop.yarn.api.records.ApplicationId.toString(ApplicationId.java:104) > {code} > Below are the failing tests : > {code} > TestHBaseTimelineStorageApps.testWriteApplicationToHBase > TestHBaseTimelineStorageApps.testEvents > TestHBaseTimelineStorageEntities.testEventsEscapeTs > TestHBaseTimelineStorageEntities.testWriteEntityToHBase > TestHBaseTimelineStorageEntities.testEventsWithEmptyInfo > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6044) Resource bar of Capacity Scheduler UI does not show correctly
[ https://issues.apache.org/jira/browse/YARN-6044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122376#comment-16122376 ] Junping Du commented on YARN-6044: -- Is this related to YARN-4484? cc [~sunilg]. > Resource bar of Capacity Scheduler UI does not show correctly > - > > Key: YARN-6044 > URL: https://issues.apache.org/jira/browse/YARN-6044 > Project: Hadoop YARN > Issue Type: Bug > Components: capacityscheduler >Affects Versions: 2.8.0 >Reporter: Tao Yang >Priority: Minor > > Test Environment: > 1. NodeLable > yarn rmadmin -addToClusterNodeLabels "label1(exclusive=false)" > 2. capacity-scheduler.xml > yarn.scheduler.capacity.root.queues=a,b > yarn.scheduler.capacity.root.a.capacity=60 > yarn.scheduler.capacity.root.b.capacity=40 > yarn.scheduler.capacity.root.a.accessible-node-labels=label1 > yarn.scheduler.capacity.root.accessible-node-labels.label1.capacity=100 > yarn.scheduler.capacity.root.a.accessible-node-labels.label1.capacity=100 > In this test case, for queue(root.b) in partition(label1), the resource > bar(represents absolute-max-capacity) should be 100%(default). The scheduler > UI shows correctly after RM started, but when I started an app in > queue(root.b) and partition(label1) , the resource bar of this queue is > changed from 100% to 0%. > For corrent queue(root.a), the queueCapacities of partition(label1) was > inited in ParentQueue/LeafQueue constructor and > max-capacity/absolute-max-capacity were setted with correct value, due to > yarn.scheduler.capacity.root.a.accessible-node-labels is defined in > capacity-scheduler.xml > For incorrent queue(root.b), the queueCapacities of partition(label1) didn't > exist at first, the max-capacity and absolute-max-capacity were setted with > default value(100%) in PartitionQueueCapacitiesInfo so that Scheduler UI > could show correctly. When this queue was allocating resource for > partition(label1), the queueCapacities of partition(label1) was created and > only used-capacity and absolute-used-capacity were setted in > AbstractCSQueue#allocateResource. max-capacity and absolute-max-capacity have > to use float default value 0 which are defined in QueueCapacities$Capacities. > Whether max-capacity and absolute-max-capacity should have default > value(100%) in Capacities constructor to avoid losing default value if > somewhere called not given? > Please feel free to give your suggestions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6896) Federation: routing REST invocations transparently to multiple RMs (part 1 - basic execution)
[ https://issues.apache.org/jira/browse/YARN-6896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122362#comment-16122362 ] Carlo Curino commented on YARN-6896: Thanks [~giovanni.fumarola] your answers make sense, and the patch looks good to me, with only a couple of things on testing/validation: # In {{TestFederationInterceptorREST.testSubmitApplication}} (and similar methods) you don't check that the request was actually sent to the RM, but only that it was book-keeped in the {{FederationStateStore}}, i.e., we only check half of the functionality. Since you have your own mock class {{MockDefaultFederationInterceptorREST}}, you can have a global static variable in the class that you for example increment for each invocation, or you can use Mockito {{spy()}} and make sure that one-and-only-one of the {{MockDefaultFederationInterceptorREST}} is invoked in normal cases, and that for retry scenarios that the right set of invocations is performed (e.g., counting separately good/bad SC invocations?). Something along this line would strengthen the tests. # Did you run this in a live cluster? Not strictly needed, but since the tests are "local" to this interceptor a bit of integration validation could help us be confident about it. > Federation: routing REST invocations transparently to multiple RMs (part 1 - > basic execution) > - > > Key: YARN-6896 > URL: https://issues.apache.org/jira/browse/YARN-6896 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-6896.proto.patch, YARN-6896.v1.patch, > YARN-6896.v2.patch > > > This JIRA tracks the design/implementation of the layer for routing > RMWebServicesProtocol requests to the appropriate RM(s) in a federated YARN > cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6988) container-executor fails for docker when command length > 4096 B
[ https://issues.apache.org/jira/browse/YARN-6988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122348#comment-16122348 ] Eric Badger commented on YARN-6988: --- We should definitely increase the limit, but don't want to malloc the entire max amount of memory that the system allows, since that would be a waste. A compromise could be to set the arg length to the min of the system value and 128K. > container-executor fails for docker when command length > 4096 B > > > Key: YARN-6988 > URL: https://issues.apache.org/jira/browse/YARN-6988 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Eric Badger >Assignee: Eric Badger > > {{run_docker}} and {{launch_docker_container_as_user}} allocate their command > arrays using EXECUTOR_PATH_MAX, which is hardcoded to 4096 in > configuration.h. Because of this, the full docker command can only be 4096 > characters. If it is longer, it will be truncated and the command will fail > with a parsing error. Because of the bind-mounting of volumes, the arguments > to the docker command can quickly get large. For example, I passed the 4096 > limit with an 11 disk node. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6988) container-executor fails for docker when command length > 4096 B
Eric Badger created YARN-6988: - Summary: container-executor fails for docker when command length > 4096 B Key: YARN-6988 URL: https://issues.apache.org/jira/browse/YARN-6988 Project: Hadoop YARN Issue Type: Sub-task Reporter: Eric Badger Assignee: Eric Badger {{run_docker}} and {{launch_docker_container_as_user}} allocate their command arrays using EXECUTOR_PATH_MAX, which is hardcoded to 4096 in configuration.h. Because of this, the full docker command can only be 4096 characters. If it is longer, it will be truncated and the command will fail with a parsing error. Because of the bind-mounting of volumes, the arguments to the docker command can quickly get large. For example, I passed the 4096 limit with an 11 disk node. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6820) Restrict read access to timelineservice v2 data
[ https://issues.apache.org/jira/browse/YARN-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122324#comment-16122324 ] Jason Lowe commented on YARN-6820: -- Sure, let's file a separate JIRA to get consistent about how we're getting the remote user. It certainly doesn't make sense to have some parts of YARN getting the remote user from the request while others are getting the name of the principal. With that issue out of the way, I'm +1 on the latest patch. I'll commit this tomorrow to the YARN-5355 branch if there are no objections. > Restrict read access to timelineservice v2 data > > > Key: YARN-6820 > URL: https://issues.apache.org/jira/browse/YARN-6820 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: yarn-5355-merge-blocker > Attachments: YARN-6820-YARN-5355.0001.patch, > YARN-6820-YARN-5355.002.patch, YARN-6820-YARN-5355.003.patch, > YARN-6820-YARN-5355.004.patch, YARN-6820-YARN-5355.005.patch > > > Need to provide a way to restrict read access in ATSv2. Not all users should > be able to read all entities. On the flip side, some folks may not need any > read restrictions, so we need to provide a way to disable this access > restriction as well. > Initially this access restriction could be done in a simple way via a > whitelist of users allowed to read data. That set of users can read all data, > no other user can read any data. Can be turned off for all users to read all > data. > Could be stored in a "domain" table in hbase perhaps. Or a configuration > setting for the cluster. Or something else that's simple enough. ATSv1 has a > concept of domain for isolating users for reading. Would be good to keep that > in consideration. > In ATSv1, domain offers a namespace for Timeline server allowing users to > host multiple entities, isolating them from other users and applications. A > “Domain” in ATSV1 primarily stores owner info, read and& write ACL > information, created and modified time stamp information. Each Domain is > identified by an ID which must be unique across all users in the YARN cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6987) Log app attempt during InvalidStateTransition
[ https://issues.apache.org/jira/browse/YARN-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122318#comment-16122318 ] Hadoop QA commented on YARN-6987: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 36s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 50m 34s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 82m 19s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing | | Timed out junit tests | org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6987 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881308/YARN-6987.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 8a17f2a23aac 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/16837/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/16837/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16837/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Log app
[jira] [Commented] (YARN-5536) Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout
[ https://issues.apache.org/jira/browse/YARN-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122304#comment-16122304 ] Junping Du commented on YARN-5536: -- My current priority on related effort is still on YARN-5464. [~mingma], do you have bandwidth on this? > Multiple format support (JSON, etc.) for exclude node file in NM graceful > decommission with timeout > --- > > Key: YARN-5536 > URL: https://issues.apache.org/jira/browse/YARN-5536 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Junping Du >Priority: Blocker > > Per discussion in YARN-4676, we agree that multiple format (other than xml) > should be supported to decommission nodes with timeout values. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6972) Adding RM ClusterId in AppInfo
[ https://issues.apache.org/jira/browse/YARN-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122302#comment-16122302 ] Hadoop QA commented on YARN-6972: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 21s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 5 new + 92 unchanged - 2 fixed = 97 total (was 94) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 12s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 64m 43s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor | | | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification | | | hadoop.yarn.server.resourcemanager.TestRMHA | | | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer | | | hadoop.yarn.server.resourcemanager.webapp.TestRMWebAppFairScheduler | | | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6972 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881309/YARN-6972.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 2fbad27afbc6 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/16838/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/16838/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | htt
[jira] [Commented] (YARN-6620) [YARN-6223] NM Java side code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122286#comment-16122286 ] Hadoop QA commented on YARN-6620: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 43s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 11m 15s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 54s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 56s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 40s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 7s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 22 new + 457 unchanged - 0 fixed = 479 total (was 457) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 9s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 32s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 1 new + 105 unchanged - 0 fixed = 106 total (was 105) {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 42s{color} | {color:red} hadoop-yarn-api in the patch failed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 15m 34s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 36s{color} | {color:red} The patch generated 2 ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 74m 47s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | | Boxing/unboxing to parse a primitive org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.GpuResourceAllocator.recoverAssignedGpus(ContainerId) At GpuResourceAllocator.java:org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.resources.GpuResourceAllocator.recoverAssignedGpus(ContainerId) At GpuResourceAllocator.java:[line 92] | | | Boxing/unboxing to parse a primitive org.apache.hadoop.yarn.server.nodemanager.containerman
[jira] [Updated] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-6852: - Attachment: YARN-6852.007.patch Attached 007 patch, fixed more cc warnings, will try to fix all cc warnings in the next patch. > [YARN-6223] Native code changes to support isolate GPU devices by using > CGroups > --- > > Key: YARN-6852 > URL: https://issues.apache.org/jira/browse/YARN-6852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6852.001.patch, YARN-6852.002.patch, > YARN-6852.003.patch, YARN-6852.004.patch, YARN-6852.005.patch, > YARN-6852.006.patch, YARN-6852.007.patch > > > This JIRA plan to add support of: > 1) Isolation in CGroups. (native side). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6980) The YARN_TIMELINE_HEAPSIZE does not work
[ https://issues.apache.org/jira/browse/YARN-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122265#comment-16122265 ] Allen Wittenauer edited comment on YARN-6980 at 8/10/17 8:27 PM: - This is backward compatibility bits, so the variable should be YARN_TIMELINESERVER_HEAPSIZE as it was in branch-2. Just looks like a mistake in yarn-env.sh. It's worth pointing out that the *preferred* way to set memory for specific daemons in 3.x is to put the -Xmx in _OPTS was (Author: aw): This is backward compatibility bits, so the variable should be YARN_TIMELINESERVER_HEAPSIZE as it was in branch-2. Just looks like a mistake in yarn-env.sh. > The YARN_TIMELINE_HEAPSIZE does not work > > > Key: YARN-6980 > URL: https://issues.apache.org/jira/browse/YARN-6980 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Jinjiang Ling >Assignee: Jinjiang Ling >Priority: Minor > Attachments: YARN-6980.patch > > > As I want to set the java heap size of timeline server, I find there is a > variable named YARN_TIMELINE_HEAPSIZE in the comments of yarn-env.sh, then I > set it but does not work. > Then I find the variable is changed in > -[HADOOP-10950|https://issues.apache.org/jira/browse/HADOOP-10950]-, > but it is just changed in the comments of yarn-env.sh and don't in "yarn" and > "yarn.cmd" command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6980) The YARN_TIMELINE_HEAPSIZE does not work
[ https://issues.apache.org/jira/browse/YARN-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122265#comment-16122265 ] Allen Wittenauer commented on YARN-6980: This is backward compatibility bits, so the variable should be YARN_TIMELINESERVER_HEAPSIZE as it was in branch-2. Just looks like a mistake in yarn-env.sh. > The YARN_TIMELINE_HEAPSIZE does not work > > > Key: YARN-6980 > URL: https://issues.apache.org/jira/browse/YARN-6980 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.0.0-alpha4 >Reporter: Jinjiang Ling >Assignee: Jinjiang Ling >Priority: Minor > Attachments: YARN-6980.patch > > > As I want to set the java heap size of timeline server, I find there is a > variable named YARN_TIMELINE_HEAPSIZE in the comments of yarn-env.sh, then I > set it but does not work. > Then I find the variable is changed in > -[HADOOP-10950|https://issues.apache.org/jira/browse/HADOOP-10950]-, > but it is just changed in the comments of yarn-env.sh and don't in "yarn" and > "yarn.cmd" command. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6969) Remove method getMinShareMemoryFraction and getPendingContainers in class FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122263#comment-16122263 ] Hadoop QA commented on YARN-6969: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 12s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager generated 0 new + 351 unchanged - 1 fixed = 351 total (was 352) {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 48m 47s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 72m 51s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | | Unread field:FairSchedulerQueueInfo.java:[line 112] | | | Unread field:FairSchedulerQueueInfo.java:[line 117] | | Failed junit tests | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer | | | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation | | Timed out junit tests | org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA | | | org.apache.hadoop.yarn.server.resourcemanager.TestRMHAForNodeLabels | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6969 | | GITHUB PR | https://github.com/apache/hadoop/pull/260 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 5805f86403bd 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/16834/artifact/patchprocess/new-fi
[jira] [Commented] (YARN-6905) Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat
[ https://issues.apache.org/jira/browse/YARN-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122239#comment-16122239 ] Vrushali C commented on YARN-6905: -- Thanks [~haibo.chen] for the patch. - I think the javadoc errors are related. - Also, if possible, could you please add a test case that invokes ApplicationId.fromString on convertApplicationIdToString and checks if the returned app id object and the input app id object are equal? - I would prefer not to have "Gross hack!" in the function documentation too. Perhaps reword it to remove that, keep the rest of the description as you already have on the function documentation and add in "This is a work-around implementation as discussed in YARN-6905" or something after it? > Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat > --- > > Key: YARN-6905 > URL: https://issues.apache.org/jira/browse/YARN-6905 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Affects Versions: 3.0.0-beta1 > Environment: Ubuntu 14.04 > x86, ppc64le > $ java -version > openjdk version "1.8.0_111" > OpenJDK Runtime Environment (build 1.8.0_111-8u111-b14-3~14.04.1-b14) > OpenJDK 64-Bit Server VM (build 25.111-b14, mixed mode) >Reporter: Sonia Garudi >Assignee: Haibo Chen > Attachments: YARN-6905.00.patch > > > There are multiple test failing in Hadoop YARN Timeline Service HBase tests > project with the following error : > {code} > java.lang.NoClassDefFoundError: org/apache/hadoop/util/FastNumberFormat > at > org.apache.hadoop.yarn.api.records.ApplicationId.toString(ApplicationId.java:104) > {code} > Below are the failing tests : > {code} > TestHBaseTimelineStorageApps.testWriteApplicationToHBase > TestHBaseTimelineStorageApps.testEvents > TestHBaseTimelineStorageEntities.testEventsEscapeTs > TestHBaseTimelineStorageEntities.testWriteEntityToHBase > TestHBaseTimelineStorageEntities.testEventsWithEmptyInfo > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6130) [ATSv2 Security] Generate a delegation token for AM when app collector is created and pass it to AM via NM and RM
[ https://issues.apache.org/jira/browse/YARN-6130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122192#comment-16122192 ] Vrushali C commented on YARN-6130: -- We could consider creating a separate jira for branch-2 to add in patches from this jira and YARN-6133. > [ATSv2 Security] Generate a delegation token for AM when app collector is > created and pass it to AM via NM and RM > - > > Key: YARN-6130 > URL: https://issues.apache.org/jira/browse/YARN-6130 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Varun Saxena >Assignee: Varun Saxena > Labels: yarn-5355-merge-blocker > Attachments: YARN-6130-YARN-5355.01.patch, > YARN-6130-YARN-5355.02.patch, YARN-6130-YARN-5355.03.patch, > YARN-6130-YARN-5355.04.patch, YARN-6130-YARN-5355.05.patch, > YARN-6130-YARN-5355.06.patch, YARN-6130-YARN-5355_branch2.01.patch, > YARN-6130-YARN-5355_branch2.02.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[ https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122190#comment-16122190 ] Hadoop QA commented on YARN-6930: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 43s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 4s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 49s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 32s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 23s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 35s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 54s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 214 unchanged - 0 fixed = 215 total (was 214) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 104 unchanged - 1 fixed = 104 total (was 105) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 33s{color} | {color:green} hadoop-yarn-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 42s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 13m 59s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:green}+1{color} | {
[jira] [Commented] (YARN-6047) Documentation updates for TimelineService v2
[ https://issues.apache.org/jira/browse/YARN-6047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122191#comment-16122191 ] Vrushali C commented on YARN-6047: -- Note: Need to update documentation with changes in YARN-6820 (the whitelisting of users allowed to read atsv2 data). > Documentation updates for TimelineService v2 > > > Key: YARN-6047 > URL: https://issues.apache.org/jira/browse/YARN-6047 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation, timelineserver >Reporter: Varun Saxena >Assignee: Rohith Sharma K S > Labels: yarn-5355-merge-blocker > Attachments: YARN-6047-YARN-5355.001.patch, > YARN-6047-YARN-5355.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6820) Restrict read access to timelineservice v2 data
[ https://issues.apache.org/jira/browse/YARN-6820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122188#comment-16122188 ] Vrushali C commented on YARN-6820: -- Thanks [~jlowe] bq. don't see why we would sometimes want to get the user via the principal and sometimes not Yes, I was initially using getRemoteUser since the webservices in timeline service are invoking that. But the RM webservices seem to be using the getUserPrincipal. I looked up in the Java EE documentations for 6 and 7 regarding the behavior around getRemoteUser and getUserPrincipal. http://docs.oracle.com/javaee/6/tutorial/doc/gjiie.html#bncba https://docs.oracle.com/javaee/7/tutorial/security-webtier003.htm Both 6 and 7 seem to say the same thing: {code} getRemoteUser, which determines the user name with which the client authenticated. The getRemoteUser method returns the name of the remote user (the caller) associated by the container with the request. If no user has been authenticated, this method returns null. getUserPrincipal, which determines the principal name of the current user and returns a java.security.Principal object. If no user has been authenticated, this method returns null. Calling the getName method on the Principal returned by getUserPrincipal returns the name of the remote user. {code} It looks like getUserPrincipal.getName will get the name of the remote user, so I think both calls that timeline service is making, will return the same thing. That said, I can still file a jira to ensure the webservices code (unrelated to this patch) will invoke the getUserPrincipal so that we have consistency across the codebase. What do you think? > Restrict read access to timelineservice v2 data > > > Key: YARN-6820 > URL: https://issues.apache.org/jira/browse/YARN-6820 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Vrushali C >Assignee: Vrushali C > Labels: yarn-5355-merge-blocker > Attachments: YARN-6820-YARN-5355.0001.patch, > YARN-6820-YARN-5355.002.patch, YARN-6820-YARN-5355.003.patch, > YARN-6820-YARN-5355.004.patch, YARN-6820-YARN-5355.005.patch > > > Need to provide a way to restrict read access in ATSv2. Not all users should > be able to read all entities. On the flip side, some folks may not need any > read restrictions, so we need to provide a way to disable this access > restriction as well. > Initially this access restriction could be done in a simple way via a > whitelist of users allowed to read data. That set of users can read all data, > no other user can read any data. Can be turned off for all users to read all > data. > Could be stored in a "domain" table in hbase perhaps. Or a configuration > setting for the cluster. Or something else that's simple enough. ATSv1 has a > concept of domain for isolating users for reading. Would be good to keep that > in consideration. > In ATSv1, domain offers a namespace for Timeline server allowing users to > host multiple entities, isolating them from other users and applications. A > “Domain” in ATSV1 primarily stores owner info, read and& write ACL > information, created and modified time stamp information. Each Domain is > identified by an ID which must be unique across all users in the YARN cluster. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6834) A container request with only racks specified and relax locality set to false is never honoured
[ https://issues.apache.org/jira/browse/YARN-6834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Muhammad Samir Khan updated YARN-6834: -- Attachment: YARN-6834.001.patch Not sure if the attached patch is the best way to solve the issue but putting it up for comments. > A container request with only racks specified and relax locality set to false > is never honoured > --- > > Key: YARN-6834 > URL: https://issues.apache.org/jira/browse/YARN-6834 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Reporter: Muhammad Samir Khan > Attachments: YARN-6834.001.patch, yarn-6834-unittest.patch > > > A patch for a unit test is attached to reproduce the issue. It creates a > container request with only racks specified (nodes=null) and relax locality > set to false. With the node-locality-delay conf set appropriately, we wait > indefinitely for a container allocation and the test will timeout. > My understanding of what causes this issue is as follows. The > RegularContainerAllocator delays a rack local allocation based on the > node-locality-delay parameter. This delay is based on missed opportunities. > However, the corresponding off-switch request is skipped but does not count > towards a missed opportunity (because relax locality is set to false). So the > allocator waits indefinitely. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6972) Adding RM ClusterId in AppInfo
[ https://issues.apache.org/jira/browse/YARN-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122180#comment-16122180 ] Tanuj Nayak edited comment on YARN-6972 at 8/10/17 7:38 PM: Attached v3, didn't see your comments I'll fix that in v4. was (Author: tanujnay): Attached v3. > Adding RM ClusterId in AppInfo > -- > > Key: YARN-6972 > URL: https://issues.apache.org/jira/browse/YARN-6972 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Tanuj Nayak > Attachments: YARN-6972.001.patch, YARN-6972.002.patch, > YARN-6972.003.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6972) Adding RM ClusterId in AppInfo
[ https://issues.apache.org/jira/browse/YARN-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanuj Nayak updated YARN-6972: -- Attachment: YARN-6972.003.patch Attached v3. > Adding RM ClusterId in AppInfo > -- > > Key: YARN-6972 > URL: https://issues.apache.org/jira/browse/YARN-6972 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Tanuj Nayak > Attachments: YARN-6972.001.patch, YARN-6972.002.patch, > YARN-6972.003.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6923) Metrics for Federation Router
[ https://issues.apache.org/jira/browse/YARN-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122182#comment-16122182 ] Hadoop QA commented on YARN-6923: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 12s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 46s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 12s{color} | {color:red} hadoop-yarn-server-router in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 20s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 26m 52s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.router.webapp.TestRouterWebServicesREST | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6923 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881302/YARN-6923.v1.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux b30b72c40d8e 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/16835/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/16835/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/16835/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16835/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message
[jira] [Commented] (YARN-6882) AllocationFileLoaderService.reloadAllocations() should use the diamond operator
[ https://issues.apache.org/jira/browse/YARN-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122177#comment-16122177 ] Hadoop QA commented on YARN-6882: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 44m 21s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 12s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 65m 34s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.monitor.TestSchedulingMonitor | | | hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation | | | hadoop.yarn.server.resourcemanager.TestRMEmbeddedElector | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6882 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881288/YARN-6882.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 719829fd1018 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/16831/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/16831/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16831/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automaticall
[jira] [Comment Edited] (YARN-6861) Reader API for sub application entities
[ https://issues.apache.org/jira/browse/YARN-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122159#comment-16122159 ] Vrushali C edited comment on YARN-6861 at 8/10/17 7:23 PM: --- [~rohithsharma] [~varun_saxena] and I discussed this in our weekly call today. The discussion was about the naming of these apis. The consensus was, as of now, we will proceed with this patch. We discussed if we should call it something other than /users//entities// to indicate that these are entities that are being queried for without knowledge of the yarn application id. At present, these apis will return sub-application entities. For example, a query that an user "userA" runs on a Tez setup. This user is different from the user, say user "userYARN" who is running the Tez AM. Note 1: Entities from only such queries will go to two places in the backend: - in the entity table within the context of an application: {code} userYARN / cluster/ flow / flowrun id / appid / entity {code} - in the sub application table outside the context of an application: {code} userA / cluster / entity {code} Note 2: In this same example, the Tez AM itself writes some lifecycle events and metrics of it's containers. These will go only to entity table for user "userYARN". The reader APIs in this patch are going to return data that belongs to the context of entities stored outside of an application, that is, from the sub application table. The reader APIs like {code} GET /ws/v2/timeline/clusters/{cluster name}/apps/{app id}/entities/{entity type} or GET /ws/v2/timeline/apps/{app id}/entities/{entity type} {code} will return all entities, that is, entities written in "Note 1" as well as written in "Note 2". The reader APIs in this patch will return a subset of entities, those written in "Note 1". The point we discussed was that when we move on to having user level (and queue level) aggregations, we would need reader APIs to return that data. For example, an API that returns say megabytemillis (or all MR counters) for a user within a time range, say like last week. These APIs help understand usage of a user or queue on the cluster. This data is aggregated data and those APIs could like have similar API format /users//entities perhaps. In this case, we could call the API /usersummary//entities. As of now, we will proceed with this patch. was (Author: vrushalic): [~rohithsharma] [~varun_saxena] and I discussed this in our weekly call today. The discussion was about the naming of these apis. The consensus was, as of now, we will proceed with this patch. We discussed if we should call it something other than /users//entities// to indicate that these are entities that are being queried for without knowledge of the yarn application id. At present, these apis will return sub-application entities. For example, a query that an user "userA" runs on a Tez setup. This user is different from the user, say user "userYARN" who is running the Tez AM. Note 1: Entities from only such queries will go to two places in the backend: - in the entity table within the context of an application: {code} userYARN / cluster/ flow / flowrun id / appid / entity {code} - in the sub application table outside the context of an application: {code} userA / cluster / entity {code} Note 2: In this same example, the Tez AM itself writes some lifecycle events and metrics of it's containers. These will go only to entity table for user "userYARN". The reader APIs in this patch are going to return data that belongs to the context of entities stored outside of an application, that is, from the sub application table. The reader APIs like GET /ws/v2/timeline/clusters/{cluster name}/apps/{app id}/entities/{entity type} or GET /ws/v2/timeline/apps/{app id}/entities/{entity type} will return all entities, that is, entities written in "Note 1" as well as written in "Note 2". The reader APIs in this patch will return a subset of entities, those written in "Note 1". The point we discussed was that when we move on to having user level (and queue level) aggregations, we would need reader APIs to return that data. For example, an API that returns say megabytemillis (or all MR counters) for a user within a time range, say like last week. These APIs help understand usage of a user or queue on the cluster. This data is aggregated data and those APIs could like have similar API format /users//entities perhaps. In this case, we could call the API /usersummary//entities. As of now, we will proceed with this patch. > Reader API for sub application entities > --- > > Key: YARN-6861 > URL: https://issues.apache.org/jira/browse/YARN-6861 > Project: Hadoop YARN > Issue Type: Sub-task > Components: ti
[jira] [Commented] (YARN-6861) Reader API for sub application entities
[ https://issues.apache.org/jira/browse/YARN-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122159#comment-16122159 ] Vrushali C commented on YARN-6861: -- [~rohithsharma] [~varun_saxena] and I discussed this in our weekly call today. The discussion was about the naming of these apis. The consensus was, as of now, we will proceed with this patch. We discussed if we should call it something other than /users//entities// to indicate that these are entities that are being queried for without knowledge of the yarn application id. At present, these apis will return sub-application entities. For example, a query that an user "userA" runs on a Tez setup. This user is different from the user, say user "userYARN" who is running the Tez AM. Note 1: Entities from only such queries will go to two places in the backend: - in the entity table within the context of an application: {code} userYARN / cluster/ flow / flowrun id / appid / entity {code} - in the sub application table outside the context of an application: {code} sub app userA / cluster / entity {code} Note 2: In this same example, the Tez AM itself writes some lifecycle events and metrics of it's containers. These will go only to entity table for user "userYARN". The reader APIs in this patch are going to return data that belongs to the context of entities stored outside of an application, that is, from the sub application table. The reader APIs like GET /ws/v2/timeline/clusters/{cluster name}/apps/{app id}/entities/{entity type} or GET /ws/v2/timeline/apps/{app id}/entities/{entity type} will return all entities, that is, entities written in "Note 1" as well as written in "Note 2". The reader APIs in this patch will return a subset of entities, those written in "Note 1". The point we discussed was that when we move on to having user level (and queue level) aggregations, we would need reader APIs to return that data. For example, an API that returns say megabytemillis (or all MR counters) for a user within a time range, say like last week. These APIs help understand usage of a user or queue on the cluster. This data is aggregated data and those APIs could like have similar API format /users//entities perhaps. In this case, we could call the API /usersummary//entities. As of now, we will proceed with this patch. > Reader API for sub application entities > --- > > Key: YARN-6861 > URL: https://issues.apache.org/jira/browse/YARN-6861 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelinereader >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: YARN-6861-YARN-5355.001.patch, > YARN-6861-YARN-5355.002.patch > > > YARN-6733 and YARN-6734 writes data into sub application table. There should > be a way to read those entities. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-6861) Reader API for sub application entities
[ https://issues.apache.org/jira/browse/YARN-6861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122159#comment-16122159 ] Vrushali C edited comment on YARN-6861 at 8/10/17 7:22 PM: --- [~rohithsharma] [~varun_saxena] and I discussed this in our weekly call today. The discussion was about the naming of these apis. The consensus was, as of now, we will proceed with this patch. We discussed if we should call it something other than /users//entities// to indicate that these are entities that are being queried for without knowledge of the yarn application id. At present, these apis will return sub-application entities. For example, a query that an user "userA" runs on a Tez setup. This user is different from the user, say user "userYARN" who is running the Tez AM. Note 1: Entities from only such queries will go to two places in the backend: - in the entity table within the context of an application: {code} userYARN / cluster/ flow / flowrun id / appid / entity {code} - in the sub application table outside the context of an application: {code} userA / cluster / entity {code} Note 2: In this same example, the Tez AM itself writes some lifecycle events and metrics of it's containers. These will go only to entity table for user "userYARN". The reader APIs in this patch are going to return data that belongs to the context of entities stored outside of an application, that is, from the sub application table. The reader APIs like GET /ws/v2/timeline/clusters/{cluster name}/apps/{app id}/entities/{entity type} or GET /ws/v2/timeline/apps/{app id}/entities/{entity type} will return all entities, that is, entities written in "Note 1" as well as written in "Note 2". The reader APIs in this patch will return a subset of entities, those written in "Note 1". The point we discussed was that when we move on to having user level (and queue level) aggregations, we would need reader APIs to return that data. For example, an API that returns say megabytemillis (or all MR counters) for a user within a time range, say like last week. These APIs help understand usage of a user or queue on the cluster. This data is aggregated data and those APIs could like have similar API format /users//entities perhaps. In this case, we could call the API /usersummary//entities. As of now, we will proceed with this patch. was (Author: vrushalic): [~rohithsharma] [~varun_saxena] and I discussed this in our weekly call today. The discussion was about the naming of these apis. The consensus was, as of now, we will proceed with this patch. We discussed if we should call it something other than /users//entities// to indicate that these are entities that are being queried for without knowledge of the yarn application id. At present, these apis will return sub-application entities. For example, a query that an user "userA" runs on a Tez setup. This user is different from the user, say user "userYARN" who is running the Tez AM. Note 1: Entities from only such queries will go to two places in the backend: - in the entity table within the context of an application: {code} userYARN / cluster/ flow / flowrun id / appid / entity {code} - in the sub application table outside the context of an application: {code} sub app userA / cluster / entity {code} Note 2: In this same example, the Tez AM itself writes some lifecycle events and metrics of it's containers. These will go only to entity table for user "userYARN". The reader APIs in this patch are going to return data that belongs to the context of entities stored outside of an application, that is, from the sub application table. The reader APIs like GET /ws/v2/timeline/clusters/{cluster name}/apps/{app id}/entities/{entity type} or GET /ws/v2/timeline/apps/{app id}/entities/{entity type} will return all entities, that is, entities written in "Note 1" as well as written in "Note 2". The reader APIs in this patch will return a subset of entities, those written in "Note 1". The point we discussed was that when we move on to having user level (and queue level) aggregations, we would need reader APIs to return that data. For example, an API that returns say megabytemillis (or all MR counters) for a user within a time range, say like last week. These APIs help understand usage of a user or queue on the cluster. This data is aggregated data and those APIs could like have similar API format /users//entities perhaps. In this case, we could call the API /usersummary//entities. As of now, we will proceed with this patch. > Reader API for sub application entities > --- > > Key: YARN-6861 > URL: https://issues.apache.org/jira/browse/YARN-6861 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timeliner
[jira] [Commented] (YARN-6878) TestCapacityScheduler.testDefaultNodeLabelExpressionQueueConfig() has the args to assertEqual() in the wrong order
[ https://issues.apache.org/jira/browse/YARN-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122155#comment-16122155 ] Larry Lo commented on YARN-6878: Thanks Daniel! I'll try to fix this issue. > TestCapacityScheduler.testDefaultNodeLabelExpressionQueueConfig() has the > args to assertEqual() in the wrong order > -- > > Key: YARN-6878 > URL: https://issues.apache.org/jira/browse/YARN-6878 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler, test >Affects Versions: 3.0.0-alpha4 >Reporter: Daniel Templeton >Priority: Trivial > Labels: newbie > > The expected value should come before the actual value. It would be nice to > add some assert messages as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6987) Log app attempt during InvalidStateTransition
[ https://issues.apache.org/jira/browse/YARN-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated YARN-6987: -- Attachment: YARN-6987.001.patch > Log app attempt during InvalidStateTransition > - > > Key: YARN-6987 > URL: https://issues.apache.org/jira/browse/YARN-6987 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles > Attachments: YARN-6987.001.patch > > > Found this InvalidStateTransition logged in the resource manager log file > with no way to determine exactly which app attempt this was associated with. > {noformat} > 2017-08-04 17:22:29,895 [AsyncDispatcher event handler] ERROR > attempt.RMAppAttemptImpl: Can't handle this event at current state > org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: > UNREGISTERED at LAUNCHED > at > org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) > at > org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) > at > org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:802) > at > org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:108) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:803) > at > org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:784) > at > org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184) > at > org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110) > at java.lang.Thread.run(Thread.java:745) > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-6987) Log app attempt during InvalidStateTransition
Jonathan Eagles created YARN-6987: - Summary: Log app attempt during InvalidStateTransition Key: YARN-6987 URL: https://issues.apache.org/jira/browse/YARN-6987 Project: Hadoop YARN Issue Type: Bug Reporter: Jonathan Eagles Assignee: Jonathan Eagles Found this InvalidStateTransition logged in the resource manager log file with no way to determine exactly which app attempt this was associated with. {noformat} 2017-08-04 17:22:29,895 [AsyncDispatcher event handler] ERROR attempt.RMAppAttemptImpl: Can't handle this event at current state org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: UNREGISTERED at LAUNCHED at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305) at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46) at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:802) at org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl.handle(RMAppAttemptImpl.java:108) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:803) at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$ApplicationAttemptEventDispatcher.handle(ResourceManager.java:784) at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:184) at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:110) at java.lang.Thread.run(Thread.java:745) {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6413) Decouple Yarn Registry API from ZK
[ https://issues.apache.org/jira/browse/YARN-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122143#comment-16122143 ] Jian He commented on YARN-6413: --- So the registry path is currently like this: {code} /users/{username}/{serviceclass}/{instancename}/components/{componentname} {code}, see [document|https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/registry/yarn-registry.html] For native-service, ApplicationServiceRecordKey#appId will need to be appName and ContainerServiceRecordKey#containerId will need to be containerName, probably rename it to be containerKey or appKey ? And the new method RegistryUtil#getPathForServiceRecordKey, the generated path is inconsistent with the document - which will beak the callers of registry API. Could you make it in consistent with the document ? > Decouple Yarn Registry API from ZK > -- > > Key: YARN-6413 > URL: https://issues.apache.org/jira/browse/YARN-6413 > Project: Hadoop YARN > Issue Type: Improvement > Components: amrmproxy, api, resourcemanager >Reporter: Ellen Hui >Assignee: Ellen Hui > Attachments: 0001-Registry-API-v2.patch, 0002-Registry-API-v2.patch, > 0003-Registry-API-api-only.patch, 0004-Registry-API-api-stubbed.patch > > > Right now the Yarn Registry API (defined in the RegistryOperations interface) > is a very thin layer over Zookeeper. This jira proposes changing the > interface to abstract away the implementation details so that we can write a > FS-based implementation of the registry service, which will be used to > support AMRMProxy HA. > The new interface will use register/delete/resolve APIs instead of > Zookeeper-specific operations like mknode. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122138#comment-16122138 ] Hadoop QA commented on YARN-6852: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 7 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 38s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 41 new + 0 unchanged - 0 fixed = 41 total (was 0) {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 19s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 13s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 30m 36s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6852 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881293/YARN-6852.006.patch | | Optional Tests | asflicense compile cc mvnsite javac unit | | uname | Linux 3ce5e6484bec 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | Default Java | 1.8.0_131 | | cc | https://builds.apache.org/job/PreCommit-YARN-Build/16832/artifact/patchprocess/diff-compile-cc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/16832/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16832/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > [YARN-6223] Native code changes to support isolate GPU devices by using > CGroups > --- > > Key: YARN-6852 > URL: https://issues.apache.org/jira/browse/YARN-6852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6852.001.patch, YARN-6852.002.patch, > YARN-6852.003.patch, YARN-6852.004.patch, YARN-6852.005.patch, > YARN-6852.006.patch > > > This JIRA plan to add support of: > 1) Isolation in CGroups. (native side). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6620) [YARN-6223] NM Java side code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-6620: - Attachment: YARN-6620.002.patch Rebased to latest trunk (002) > [YARN-6223] NM Java side code changes to support isolate GPU devices by using > CGroups > - > > Key: YARN-6620 > URL: https://issues.apache.org/jira/browse/YARN-6620 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6620.001.patch, YARN-6620.002.patch > > > This JIRA plan to add support of: > 1) GPU configuration for NodeManagers > 2) Isolation in CGroups. (Java side). > 3) NM restart and recovery allocated GPU devices -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6923) Metrics for Federation Router
[ https://issues.apache.org/jira/browse/YARN-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Giovanni Matteo Fumarola updated YARN-6923: --- Attachment: YARN-6923.v1.patch > Metrics for Federation Router > - > > Key: YARN-6923 > URL: https://issues.apache.org/jira/browse/YARN-6923 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Giovanni Matteo Fumarola > Attachments: YARN-6923.v1.patch > > > This JIRA proposes addition of metrics for Federation StateStore -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6972) Adding RM ClusterId in AppInfo
[ https://issues.apache.org/jira/browse/YARN-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122115#comment-16122115 ] Hadoop QA commented on YARN-6972: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 36s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 23s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 65 unchanged - 0 fixed = 66 total (was 65) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 48m 19s{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 72m 47s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification | | | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServiceAppsNodelabel | | | hadoop.yarn.server.resourcemanager.TestRMHA | | | hadoop.yarn.server.resourcemanager.webapp.TestRMWebAppFairScheduler | | | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer | | Timed out junit tests | org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands | | | org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6972 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881191/YARN-6972.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 1151407af2a9 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 312e57b | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/16827/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/16827/artifact/patchprocess/patch-unit-hadoop
[jira] [Commented] (YARN-6972) Adding RM ClusterId in AppInfo
[ https://issues.apache.org/jira/browse/YARN-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122112#comment-16122112 ] Giovanni Matteo Fumarola commented on YARN-6972: Hi [~tanujnay], try to pull the new code in trunk and rebase your patch. However, now that we renamed {{clusterId}} into {{clusterTimestamp}}, you should rename {{subclusterId}} in {{clusterId}}. > Adding RM ClusterId in AppInfo > -- > > Key: YARN-6972 > URL: https://issues.apache.org/jira/browse/YARN-6972 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Tanuj Nayak > Attachments: YARN-6972.001.patch, YARN-6972.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6969) Remove method getMinShareMemoryFraction and getPendingContainers in class FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry Lo updated YARN-6969: --- Attachment: YARN-6969.001.patch Sure, please help review the patch. > Remove method getMinShareMemoryFraction and getPendingContainers in class > FairSchedulerQueueInfo > > > Key: YARN-6969 > URL: https://issues.apache.org/jira/browse/YARN-6969 > Project: Hadoop YARN > Issue Type: Task > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Larry Lo >Priority: Trivial > Labels: newbie++ > Attachments: YARN-6969.001.patch > > > They are not used anymore. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5736) YARN container executor config does not handle white space
[ https://issues.apache.org/jira/browse/YARN-5736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122091#comment-16122091 ] Miklos Szegedi commented on YARN-5736: -- Sure, please include the addendum as well. > YARN container executor config does not handle white space > -- > > Key: YARN-5736 > URL: https://issues.apache.org/jira/browse/YARN-5736 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Miklos Szegedi >Assignee: Miklos Szegedi >Priority: Trivial > Labels: oct16-medium > Fix For: 3.0.0-alpha2 > > Attachments: YARN_5736.000.patch, YARN-5736.001.patch, > YARN-5736.002.patch, YARN-5736.addendum.000.patch > > > The container executor configuration reader does not handle white spaces or > malformed key value pairs in the config file correctly or gracefully > as an example the following key value line which is part of the configuration > (note the << is used as a marker to show the extra trailing space): > yarn.nodemanager.linux-container-executor.group=yarn << > is a valid line but when you run the check over the file: > [root@test]#./container-executor --checksetup > Can't get group information for yarn - Success. > [root@test]# > It fails to find the yarn group but it really tries to find the "yarn " group > which fails. There is no trimming anywhere while processing the lines. If a > space would be added in before or after the = sign a failure would also occur. > Minor nit is the fact that a failure still is logged as a Success -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6883) AllocationFileLoaderService.reloadAllocations() should use a switch statement in the main tag parsing loop instead of the if/else-if/...
[ https://issues.apache.org/jira/browse/YARN-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122088#comment-16122088 ] Daniel Templeton commented on YARN-6883: Sweet. I'll close this one when YARN-6885 is committed. > AllocationFileLoaderService.reloadAllocations() should use a switch statement > in the main tag parsing loop instead of the if/else-if/... > > > Key: YARN-6883 > URL: https://issues.apache.org/jira/browse/YARN-6883 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: Larry Lo >Priority: Minor > Labels: newbie > > {code}if ("queue".equals(element.getTagName()) || > "pool".equals(element.getTagName())) { > queueElements.add(element); > } else if ("user".equals(element.getTagName())) { > ...{code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6969) Remove method getMinShareMemoryFraction and getPendingContainers in class FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122083#comment-16122083 ] Yufei Gu commented on YARN-6969: Thanks [~rkanter]. Nice! I add some one as a contributor now. [~LarryLo], the pull request looks good to me. To make Hadoop QA Jenkins kick off, can you upload a patch file here? > Remove method getMinShareMemoryFraction and getPendingContainers in class > FairSchedulerQueueInfo > > > Key: YARN-6969 > URL: https://issues.apache.org/jira/browse/YARN-6969 > Project: Hadoop YARN > Issue Type: Task > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Larry Lo >Priority: Trivial > Labels: newbie++ > > They are not used anymore. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6883) AllocationFileLoaderService.reloadAllocations() should use a switch statement in the main tag parsing loop instead of the if/else-if/...
[ https://issues.apache.org/jira/browse/YARN-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122082#comment-16122082 ] Larry Lo commented on YARN-6883: Sure! Please close this issue and I will find other JIRAs. > AllocationFileLoaderService.reloadAllocations() should use a switch statement > in the main tag parsing loop instead of the if/else-if/... > > > Key: YARN-6883 > URL: https://issues.apache.org/jira/browse/YARN-6883 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: Larry Lo >Priority: Minor > Labels: newbie > > {code}if ("queue".equals(element.getTagName()) || > "pool".equals(element.getTagName())) { > queueElements.add(element); > } else if ("user".equals(element.getTagName())) { > ...{code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6972) Adding RM ClusterId in AppInfo
[ https://issues.apache.org/jira/browse/YARN-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122073#comment-16122073 ] Hadoop QA commented on YARN-6972: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 7s{color} | {color:red} YARN-6972 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-6972 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881291/YARN-6972.002.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16833/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Adding RM ClusterId in AppInfo > -- > > Key: YARN-6972 > URL: https://issues.apache.org/jira/browse/YARN-6972 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Tanuj Nayak > Attachments: YARN-6972.001.patch, YARN-6972.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6883) AllocationFileLoaderService.reloadAllocations() should use a switch statement in the main tag parsing loop instead of the if/else-if/...
[ https://issues.apache.org/jira/browse/YARN-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122068#comment-16122068 ] Daniel Templeton commented on YARN-6883: If it's alright with you I'll close this JIRA as handled by YARN-6885. There are lots of other open JIRAs to tackle! :) > AllocationFileLoaderService.reloadAllocations() should use a switch statement > in the main tag parsing loop instead of the if/else-if/... > > > Key: YARN-6883 > URL: https://issues.apache.org/jira/browse/YARN-6883 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: Larry Lo >Priority: Minor > Labels: newbie > > {code}if ("queue".equals(element.getTagName()) || > "pool".equals(element.getTagName())) { > queueElements.add(element); > } else if ("user".equals(element.getTagName())) { > ...{code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6882) AllocationFileLoaderService.reloadAllocations() should use the diamond operator
[ https://issues.apache.org/jira/browse/YARN-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122072#comment-16122072 ] Daniel Templeton commented on YARN-6882: LGTM +1 > AllocationFileLoaderService.reloadAllocations() should use the diamond > operator > --- > > Key: YARN-6882 > URL: https://issues.apache.org/jira/browse/YARN-6882 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: Larry Lo >Priority: Trivial > Labels: newbie > Attachments: YARN-6882.001.patch > > > Here:{code}for (FSQueueType queueType : FSQueueType.values()) { > configuredQueues.put(queueType, new HashSet()); > }{code} and here:{code}List queueElements = new > ArrayList();{code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6883) AllocationFileLoaderService.reloadAllocations() should use a switch statement in the main tag parsing loop instead of the if/else-if/...
[ https://issues.apache.org/jira/browse/YARN-6883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122063#comment-16122063 ] Larry Lo commented on YARN-6883: It looks like the same issue to change from if/else=if to switch statement in the same Class but different methods. Should I keep to fix this issue or something else? > AllocationFileLoaderService.reloadAllocations() should use a switch statement > in the main tag parsing loop instead of the if/else-if/... > > > Key: YARN-6883 > URL: https://issues.apache.org/jira/browse/YARN-6883 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: Larry Lo >Priority: Minor > Labels: newbie > > {code}if ("queue".equals(element.getTagName()) || > "pool".equals(element.getTagName())) { > queueElements.add(element); > } else if ("user".equals(element.getTagName())) { > ...{code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6852) [YARN-6223] Native code changes to support isolate GPU devices by using CGroups
[ https://issues.apache.org/jira/browse/YARN-6852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wangda Tan updated YARN-6852: - Attachment: YARN-6852.006.patch Fixed compile issues and added additional tests (006) > [YARN-6223] Native code changes to support isolate GPU devices by using > CGroups > --- > > Key: YARN-6852 > URL: https://issues.apache.org/jira/browse/YARN-6852 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-6852.001.patch, YARN-6852.002.patch, > YARN-6852.003.patch, YARN-6852.004.patch, YARN-6852.005.patch, > YARN-6852.006.patch > > > This JIRA plan to add support of: > 1) Isolation in CGroups. (native side). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6972) Adding RM ClusterId in AppInfo
[ https://issues.apache.org/jira/browse/YARN-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tanuj Nayak updated YARN-6972: -- Attachment: YARN-6972.002.patch Attached v2 with [~giovanni.fumarola]'s suggestions along with the refactoring of {code:java}clusterId{code} to {code:java}clusterTimestamp{code}. > Adding RM ClusterId in AppInfo > -- > > Key: YARN-6972 > URL: https://issues.apache.org/jira/browse/YARN-6972 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Tanuj Nayak > Attachments: YARN-6972.001.patch, YARN-6972.002.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-6969) Remove method getMinShareMemoryFraction and getPendingContainers in class FairSchedulerQueueInfo
[ https://issues.apache.org/jira/browse/YARN-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry Lo reassigned YARN-6969: -- Assignee: Larry Lo > Remove method getMinShareMemoryFraction and getPendingContainers in class > FairSchedulerQueueInfo > > > Key: YARN-6969 > URL: https://issues.apache.org/jira/browse/YARN-6969 > Project: Hadoop YARN > Issue Type: Task > Components: fairscheduler >Reporter: Yufei Gu >Assignee: Larry Lo >Priority: Trivial > Labels: newbie++ > > They are not used anymore. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6930) Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager
[ https://issues.apache.org/jira/browse/YARN-6930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shane Kumpf updated YARN-6930: -- Attachment: YARN-6930.002.patch > Admins should be able to explicitly enable specific LinuxContainerRuntime in > the NodeManager > > > Key: YARN-6930 > URL: https://issues.apache.org/jira/browse/YARN-6930 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Vinod Kumar Vavilapalli >Assignee: Shane Kumpf > Attachments: YARN-6930.001.patch, YARN-6930.002.patch > > > Today, in the java land, all LinuxContainerRuntimes are always enabled when > using LinuxContainerExecutor and the user can simply invoke anything that > he/she wants - default, docker, java-sandbox. > We should have a way for admins to explicitly enable only specific runtimes > that he/she decides for the cluster. And by default, we should have > everything other than the default one disabled. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[ https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122038#comment-16122038 ] Hadoop QA commented on YARN-6610: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} YARN-3926 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 54s{color} | {color:green} YARN-3926 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} YARN-3926 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s{color} | {color:green} YARN-3926 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} YARN-3926 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 55s{color} | {color:green} YARN-3926 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s{color} | {color:green} YARN-3926 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 24s{color} | {color:green} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common generated 0 new + 4569 unchanged - 5 fixed = 4569 total (was 4574) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 31s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 23m 0s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6610 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881282/YARN-6610.YARN-3926.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux ea7a45e04bb9 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | YARN-3926 / 1b586d7 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/16828/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16828/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > DominantResourceCalculator.getResourceAsValue() dominant param is no longer > appropriate > --- > > Key: YARN-6610 > URL: https://issues.apache.org/jira/browse/YARN-6610 > Project: Hadoop YARN > Issue Type: Sub-task > Componen
[jira] [Updated] (YARN-5536) Multiple format support (JSON, etc.) for exclude node file in NM graceful decommission with timeout
[ https://issues.apache.org/jira/browse/YARN-5536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated YARN-5536: -- Priority: Blocker (was: Critical) Target Version/s: 2.9.0, 3.0.0-beta1 (was: 2.9.0) Sounds more like a blocker if we need to get it done before release. [~mingma], [~djp], are we able to get this done by mid-September for beta1? > Multiple format support (JSON, etc.) for exclude node file in NM graceful > decommission with timeout > --- > > Key: YARN-5536 > URL: https://issues.apache.org/jira/browse/YARN-5536 > Project: Hadoop YARN > Issue Type: Sub-task > Components: graceful >Reporter: Junping Du >Priority: Blocker > > Per discussion in YARN-4676, we agree that multiple format (other than xml) > should be supported to decommission nodes with timeout values. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6882) AllocationFileLoaderService.reloadAllocations() should use the diamond operator
[ https://issues.apache.org/jira/browse/YARN-6882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Larry Lo updated YARN-6882: --- Attachment: YARN-6882.001.patch Please help to review this patch, thanks. > AllocationFileLoaderService.reloadAllocations() should use the diamond > operator > --- > > Key: YARN-6882 > URL: https://issues.apache.org/jira/browse/YARN-6882 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.0.0-alpha4 >Reporter: Daniel Templeton >Assignee: Larry Lo >Priority: Trivial > Labels: newbie > Attachments: YARN-6882.001.patch > > > Here:{code}for (FSQueueType queueType : FSQueueType.values()) { > configuredQueues.put(queueType, new HashSet()); > }{code} and here:{code}List queueElements = new > ArrayList();{code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6903) Yarn-native-service framework core rewrite
[ https://issues.apache.org/jira/browse/YARN-6903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jian He updated YARN-6903: -- Attachment: YARN-6903.yarn-native-services.06.patch > Yarn-native-service framework core rewrite > -- > > Key: YARN-6903 > URL: https://issues.apache.org/jira/browse/YARN-6903 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jian He >Assignee: Jian He > Attachments: YARN-6903.yarn-native-services.01.patch, > YARN-6903.yarn-native-services.02.patch, > YARN-6903.yarn-native-services.03.patch, > YARN-6903.yarn-native-services.04.patch, > YARN-6903.yarn-native-services.05.patch, > YARN-6903.yarn-native-services.06.patch > > > There are some new features like rich placement scheduling, container auto > restart, container upgrade in YARN core that can be taken advantage by the > native-service framework. Besides, there are quite a lot legacy code which > are no longer required. > So we decide to rewrite the core part to have a leaner codebase and make use > of various advanced features in YARN. > And the new code design will be in align with what we have designed for the > service API YARN-4793 -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6885) AllocationFileLoaderService.loadQueue() should use a switch statement in the main tag parsing loop instead of the if/else-if/...
[ https://issues.apache.org/jira/browse/YARN-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122026#comment-16122026 ] Hadoop QA commented on YARN-6885: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 32s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 23s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 3 new + 17 unchanged - 1 fixed = 20 total (was 18) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 43m 51s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 66m 40s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6885 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881272/YARN-6885.005.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux cbf0b6c5053a 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a8b7546 | | Default Java | 1.8.0_131 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/16825/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/16825/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/16825/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > AllocationFileLoaderService.loadQueue() should use a switch statement in t
[jira] [Commented] (YARN-6905) Multiple HBaseTimelineStorage test failures due to missing FastNumberFormat
[ https://issues.apache.org/jira/browse/YARN-6905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122009#comment-16122009 ] Hadoop QA commented on YARN-6905: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 30s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 44s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase-tests {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 14s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-hbase generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 4m 8s{color} | {color:green} hadoop-yarn-server-timelineservice-hbase-tests in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 36m 0s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:14b5c93 | | JIRA Issue | YARN-6905 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12881277/YARN-6905.00.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 7c7629cb4d69 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | tru
[jira] [Commented] (YARN-6972) Adding RM ClusterId in AppInfo
[ https://issues.apache.org/jira/browse/YARN-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122001#comment-16122001 ] Giovanni Matteo Fumarola commented on YARN-6972: I checked {{AppInfo.java}} and I found that it has {{clusterId}}. That name is misleading. We should rename it {{clusterTimestamp}}. [~sunilg]/[~subru]? > Adding RM ClusterId in AppInfo > -- > > Key: YARN-6972 > URL: https://issues.apache.org/jira/browse/YARN-6972 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Tanuj Nayak > Attachments: YARN-6972.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6789) new api to get all supported resources from RM
[ https://issues.apache.org/jira/browse/YARN-6789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121998#comment-16121998 ] Wangda Tan commented on YARN-6789: -- [~dan...@cloudera.com], thanks for sharing your thoughts. Please see my response below: bq. If I understand correctly, what you're proposing is making the units a global dictate. The RM says that memory has the unit "Mi", therefore all NMs and clients must use that unit. This is true, unit of values for different resource types are fixed. To be more clear, I propose to remove the "unit" from ResourceInformation object. "Unit" for each resource types are unwritten rules which each daemon/clients/am should know that. "Unit" can be set for ResourceTypeInformation (by setting resource-types.xml), however it is more like documentation instead of affecting internal logics. bq. What happens if an admin changes the units for a resource type? This is not allowed in my proposal, I doubt that in what scenario we need to change global units. For example, MB as unit for memory works so well today, I don't think it make sense to upgrade it to GB/TB since value is a 64 bits integer. This is majorly to avoid dealing with backward compatibility and inconsistency between daemons and clients. As I stated above, what if an old client talk to RM asks 4096 memory without specifying "MB" as unit? If we make it to be globally fixed and unwritten rule, we don't need to worry about inconsistency any more. And all rolling upgrade problems are gone for the unit. Thoughts? > new api to get all supported resources from RM > -- > > Key: YARN-6789 > URL: https://issues.apache.org/jira/browse/YARN-6789 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-6789-YARN-3926.001.patch > > > It will be better to provide an api to get all supported resource types from > RM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6972) Adding RM ClusterId in AppInfo
[ https://issues.apache.org/jira/browse/YARN-6972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16121993#comment-16121993 ] Giovanni Matteo Fumarola commented on YARN-6972: Thanks [~tanujnay] for the patch. Good job. Few feedback: 1) Move {{String subclusterId}} after {{AppTimeoutsInfo timeouts}}. Doing the same things for the tests and the get method. This will avoid future conflicts. 2) In the test, instead of setting {{YarnConfiguration.DEFAULT_RM_CLUSTER_ID}} please set a random word. e.g. "SubCluster1". In this way, you will validate if your code gets the correct one and not always the default. > Adding RM ClusterId in AppInfo > -- > > Key: YARN-6972 > URL: https://issues.apache.org/jira/browse/YARN-6972 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Tanuj Nayak > Attachments: YARN-6972.001.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6610) DominantResourceCalculator.getResourceAsValue() dominant param is no longer appropriate
[ https://issues.apache.org/jira/browse/YARN-6610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated YARN-6610: --- Attachment: YARN-6610.YARN-3926.004.patch Refactored a little to make it cleaner. > DominantResourceCalculator.getResourceAsValue() dominant param is no longer > appropriate > --- > > Key: YARN-6610 > URL: https://issues.apache.org/jira/browse/YARN-6610 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: YARN-3926 >Reporter: Daniel Templeton >Assignee: Daniel Templeton >Priority: Critical > Attachments: YARN-6610.001.patch, YARN-6610.YARN-3926.002.patch, > YARN-6610.YARN-3926.003.patch, YARN-6610.YARN-3926.004.patch > > > The {{dominant}} param assumes there are only two resources, i.e. true means > to compare the dominant, and false means to compare the subordinate. Now > that there are _n_ resources, this parameter no longer makes sense. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org