[jira] [Updated] (YARN-10809) testWithHbaseConfAtHdfsFileSystem consistently failing
[ https://issues.apache.org/jira/browse/YARN-10809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-10809: -- Labels: pull-request-available (was: ) > testWithHbaseConfAtHdfsFileSystem consistently failing > -- > > Key: YARN-10809 > URL: https://issues.apache.org/jira/browse/YARN-10809 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Viraj Jasani >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > TestHBaseTimelineStorageUtils#testWithHbaseConfAtHdfsFileSystem is > consistently failing with > {code:java} > java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer > at > org.apache.hadoop.hdfs.MiniDFSCluster.isNameNodeUp(MiniDFSCluster.java:2618) > at > org.apache.hadoop.hdfs.MiniDFSCluster.isClusterUp(MiniDFSCluster.java:2632) > at > org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1498) > at > org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:977) > at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:576) > at > org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:518) > at > org.apache.hadoop.yarn.server.timelineservice.storage.common.TestHBaseTimelineStorageUtils.testWithHbaseConfAtHdfsFileSystem(TestHBaseTimelineStorageUtils.java:86) > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10809) testWithHbaseConfAtHdfsFileSystem consistently failing
Viraj Jasani created YARN-10809: --- Summary: testWithHbaseConfAtHdfsFileSystem consistently failing Key: YARN-10809 URL: https://issues.apache.org/jira/browse/YARN-10809 Project: Hadoop YARN Issue Type: Bug Reporter: Viraj Jasani Assignee: Viraj Jasani TestHBaseTimelineStorageUtils#testWithHbaseConfAtHdfsFileSystem is consistently failing with {code:java} java.lang.NoClassDefFoundError: org/mockito/stubbing/Answer at org.apache.hadoop.hdfs.MiniDFSCluster.isNameNodeUp(MiniDFSCluster.java:2618) at org.apache.hadoop.hdfs.MiniDFSCluster.isClusterUp(MiniDFSCluster.java:2632) at org.apache.hadoop.hdfs.MiniDFSCluster.waitClusterUp(MiniDFSCluster.java:1498) at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:977) at org.apache.hadoop.hdfs.MiniDFSCluster.(MiniDFSCluster.java:576) at org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:518) at org.apache.hadoop.yarn.server.timelineservice.storage.common.TestHBaseTimelineStorageUtils.testWithHbaseConfAtHdfsFileSystem(TestHBaseTimelineStorageUtils.java:86) {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10807) Parents node labels are incorrectly added to child queues in weight mode
[ https://issues.apache.org/jira/browse/YARN-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358980#comment-17358980 ] Qi Zhu commented on YARN-10807: --- Thanks [~bteke] for update. The patch LGTM. > Parents node labels are incorrectly added to child queues in weight mode > - > > Key: YARN-10807 > URL: https://issues.apache.org/jira/browse/YARN-10807 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10807.001.patch, YARN-10807.002.patch > > > In ParentQueue.updateClusterResource when calculating the normalized weights > CS will iterate through the parent's nodelabels. If the parent has a node > label that a specific child doesn't it will incorrectly added to the child's > node label list through the queueCapacities.setNormalizedWeights(label, > weight) call: > {code:java} > // Normalize weight of children > if (getCapacityConfigurationTypeForQueues(childQueues) > == QueueCapacityType.WEIGHT) { > for (String nodeLabel : queueCapacities.getExistingNodeLabels()) { > float sumOfWeight = 0; > for (CSQueue queue : childQueues) { > float weight = Math.max(0, > queue.getQueueCapacities().getWeight(nodeLabel)); > sumOfWeight += weight; > } > // When sum of weight == 0, skip setting normalized_weight (so > // normalized weight will be 0). > if (Math.abs(sumOfWeight) > 1e-6) { > for (CSQueue queue : childQueues) { > queue.getQueueCapacities().setNormalizedWeight(nodeLabel, > queue.getQueueCapacities().getWeight(nodeLabel) / > sumOfWeight); > } > } > } > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10767) Yarn Logs Command retrying on Standby RM for 30 times
[ https://issues.apache.org/jira/browse/YARN-10767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358857#comment-17358857 ] Hadoop QA commented on YARN-10767: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 18s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 2s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 28s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 42s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 27s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 19m 23s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 1m 36s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 37s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 33s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 36s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 15s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | |
[jira] [Updated] (YARN-10767) Yarn Logs Command retrying on Standby RM for 30 times
[ https://issues.apache.org/jira/browse/YARN-10767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] D M Murali Krishna Reddy updated YARN-10767: Attachment: YARN-10767.002.patch > Yarn Logs Command retrying on Standby RM for 30 times > - > > Key: YARN-10767 > URL: https://issues.apache.org/jira/browse/YARN-10767 > Project: Hadoop YARN > Issue Type: Bug >Reporter: D M Murali Krishna Reddy >Assignee: D M Murali Krishna Reddy >Priority: Major > Attachments: YARN-10767.001.patch, YARN-10767.002.patch > > > When ResourceManager HA is enabled and the first RM is unavailable, on > executing "bin/yarn logs -applicationId -am 1", we get > ConnectionException for connecting to the first RM, the ConnectionException > Occurs for 30 times before it tries to connect to the second RM. > > This can be optimized by trying to fetch the logs from the Active RM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10807) Parents node labels are incorrectly added to child queues in weight mode
[ https://issues.apache.org/jira/browse/YARN-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358799#comment-17358799 ] Hadoop QA commented on YARN-10807: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 18s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 8s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 47s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 3s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 20m 14s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 1m 49s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 51s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 40s{color} | {color:green}{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 0 new + 54 unchanged - 1 fixed = 54 total (was 55) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 47s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | |
[jira] [Commented] (YARN-10792) Set Completed AppAttempt LogsLink to Log Server Url
[ https://issues.apache.org/jira/browse/YARN-10792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358782#comment-17358782 ] Prabhu Joseph commented on YARN-10792: -- Thanks [~abhinaba.sarkar] for the patch. [^YARN-10792-003.patch] looks fine, +1. Failed test cases are not related to this patch. {{TestCapacitySchedulerSurgicalPreemption#testPriorityPreemptionWithNodeLabels}} and {{TestFSRMStateStore}} are running fine on local. Looks intermittent issue, will check if Jira exists to track the same. If not, will report the same. Will commit this Patch by tomorrow EOD, if no other comments. Thanks. > Set Completed AppAttempt LogsLink to Log Server Url > --- > > Key: YARN-10792 > URL: https://issues.apache.org/jira/browse/YARN-10792 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Abhinaba Sarkar >Priority: Major > Attachments: YARN-10792-001.patch, YARN-10792-002.patch, > YARN-10792-003.patch > > > Completed AppAttempts listed by YARN UI has logslink pointing to the > NodeManager containerlogs url. The completed container logs will be under > aggregated log path and so NM ContainerLogsPage redirects to Log Server Url. > On frequent Scale Down, these NMs won't be available and so makes difficulty > to look for appattempt logs of completed apps from RM UI. Setting the > logslink for Completed AppAttempts to LogServer url will avoid this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10792) Set Completed AppAttempt LogsLink to Log Server Url
[ https://issues.apache.org/jira/browse/YARN-10792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358782#comment-17358782 ] Prabhu Joseph edited comment on YARN-10792 at 6/7/21, 6:21 PM: --- Thanks [~abhinaba.sarkar] for the patch. [^YARN-10792-003.patch] looks good to me, +1. Failed test cases are not related to this patch. {{TestCapacitySchedulerSurgicalPreemption#testPriorityPreemptionWithNodeLabels}} and {{TestFSRMStateStore}} are running fine on local. Looks intermittent issue, will check if Jira exists to track the same. If not, will report the same. Will commit this Patch by tomorrow EOD, if no other comments. Thanks. was (Author: prabhu joseph): Thanks [~abhinaba.sarkar] for the patch. [^YARN-10792-003.patch] looks fine, +1. Failed test cases are not related to this patch. {{TestCapacitySchedulerSurgicalPreemption#testPriorityPreemptionWithNodeLabels}} and {{TestFSRMStateStore}} are running fine on local. Looks intermittent issue, will check if Jira exists to track the same. If not, will report the same. Will commit this Patch by tomorrow EOD, if no other comments. Thanks. > Set Completed AppAttempt LogsLink to Log Server Url > --- > > Key: YARN-10792 > URL: https://issues.apache.org/jira/browse/YARN-10792 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Abhinaba Sarkar >Priority: Major > Attachments: YARN-10792-001.patch, YARN-10792-002.patch, > YARN-10792-003.patch > > > Completed AppAttempts listed by YARN UI has logslink pointing to the > NodeManager containerlogs url. The completed container logs will be under > aggregated log path and so NM ContainerLogsPage redirects to Log Server Url. > On frequent Scale Down, these NMs won't be available and so makes difficulty > to look for appattempt logs of completed apps from RM UI. Setting the > logslink for Completed AppAttempts to LogServer url will avoid this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10807) Parents node labels are incorrectly added to child queues in weight mode
[ https://issues.apache.org/jira/browse/YARN-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358682#comment-17358682 ] Benjamin Teke commented on YARN-10807: -- [~zhuqi] Thanks for the review. I agree with the suggestion, added the check there as well. I also extended the UTs to ensure this won't be reintroduced later. > Parents node labels are incorrectly added to child queues in weight mode > - > > Key: YARN-10807 > URL: https://issues.apache.org/jira/browse/YARN-10807 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10807.001.patch, YARN-10807.002.patch > > > In ParentQueue.updateClusterResource when calculating the normalized weights > CS will iterate through the parent's nodelabels. If the parent has a node > label that a specific child doesn't it will incorrectly added to the child's > node label list through the queueCapacities.setNormalizedWeights(label, > weight) call: > {code:java} > // Normalize weight of children > if (getCapacityConfigurationTypeForQueues(childQueues) > == QueueCapacityType.WEIGHT) { > for (String nodeLabel : queueCapacities.getExistingNodeLabels()) { > float sumOfWeight = 0; > for (CSQueue queue : childQueues) { > float weight = Math.max(0, > queue.getQueueCapacities().getWeight(nodeLabel)); > sumOfWeight += weight; > } > // When sum of weight == 0, skip setting normalized_weight (so > // normalized weight will be 0). > if (Math.abs(sumOfWeight) > 1e-6) { > for (CSQueue queue : childQueues) { > queue.getQueueCapacities().setNormalizedWeight(nodeLabel, > queue.getQueueCapacities().getWeight(nodeLabel) / > sumOfWeight); > } > } > } > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10807) Parents node labels are incorrectly added to child queues in weight mode
[ https://issues.apache.org/jira/browse/YARN-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10807: - Attachment: YARN-10807.002.patch > Parents node labels are incorrectly added to child queues in weight mode > - > > Key: YARN-10807 > URL: https://issues.apache.org/jira/browse/YARN-10807 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10807.001.patch, YARN-10807.002.patch > > > In ParentQueue.updateClusterResource when calculating the normalized weights > CS will iterate through the parent's nodelabels. If the parent has a node > label that a specific child doesn't it will incorrectly added to the child's > node label list through the queueCapacities.setNormalizedWeights(label, > weight) call: > {code:java} > // Normalize weight of children > if (getCapacityConfigurationTypeForQueues(childQueues) > == QueueCapacityType.WEIGHT) { > for (String nodeLabel : queueCapacities.getExistingNodeLabels()) { > float sumOfWeight = 0; > for (CSQueue queue : childQueues) { > float weight = Math.max(0, > queue.getQueueCapacities().getWeight(nodeLabel)); > sumOfWeight += weight; > } > // When sum of weight == 0, skip setting normalized_weight (so > // normalized weight will be 0). > if (Math.abs(sumOfWeight) > 1e-6) { > for (CSQueue queue : childQueues) { > queue.getQueueCapacities().setNormalizedWeight(nodeLabel, > queue.getQueueCapacities().getWeight(nodeLabel) / > sumOfWeight); > } > } > } > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10792) Set Completed AppAttempt LogsLink to Log Server Url
[ https://issues.apache.org/jira/browse/YARN-10792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358668#comment-17358668 ] Hadoop QA commented on YARN-10792: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 19s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m 0s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 45s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 16m 59s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 20m 11s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 1m 50s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 52s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 46s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 49s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 14m 51s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s{color} | {color:green}{color} | {color:green} the patch passed with
[jira] [Updated] (YARN-10792) Set Completed AppAttempt LogsLink to Log Server Url
[ https://issues.apache.org/jira/browse/YARN-10792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinaba Sarkar updated YARN-10792: --- Attachment: YARN-10792-003.patch > Set Completed AppAttempt LogsLink to Log Server Url > --- > > Key: YARN-10792 > URL: https://issues.apache.org/jira/browse/YARN-10792 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Abhinaba Sarkar >Priority: Major > Attachments: YARN-10792-001.patch, YARN-10792-002.patch, > YARN-10792-003.patch > > > Completed AppAttempts listed by YARN UI has logslink pointing to the > NodeManager containerlogs url. The completed container logs will be under > aggregated log path and so NM ContainerLogsPage redirects to Log Server Url. > On frequent Scale Down, these NMs won't be available and so makes difficulty > to look for appattempt logs of completed apps from RM UI. Setting the > logslink for Completed AppAttempts to LogServer url will avoid this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9910) Make private localizer download resources in parallel
[ https://issues.apache.org/jira/browse/YARN-9910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Minni Mittal reassigned YARN-9910: -- Assignee: Minni Mittal (was: Abhishek Modi) > Make private localizer download resources in parallel > - > > Key: YARN-9910 > URL: https://issues.apache.org/jira/browse/YARN-9910 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Abhishek Modi >Assignee: Minni Mittal >Priority: Major > > Currently private localizer uses a single threaded pool to do the > localization. As part of this jira, private localizer will create a fixed > threadpool of configurable number of threads for localization. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10792) Set Completed AppAttempt LogsLink to Log Server Url
[ https://issues.apache.org/jira/browse/YARN-10792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358542#comment-17358542 ] Hadoop QA commented on YARN-10792: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 54s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 18s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 0s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 50s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 6s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 19m 10s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 22m 22s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 1m 52s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 45s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1044/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 56s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1044/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 56s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1044/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04.txt{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04. {color} | | {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 46s{color} | {color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1044/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10.txt{color} | {color:red} hadoop-yarn-server-resourcemanager in the patch failed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10. {color} | | {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 46s{color} |
[jira] [Commented] (YARN-8859) Add audit logs for router service
[ https://issues.apache.org/jira/browse/YARN-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358491#comment-17358491 ] Bibin Chundatt commented on YARN-8859: -- [~BilwaST] could you take a look .. > Add audit logs for router service > - > > Key: YARN-8859 > URL: https://issues.apache.org/jira/browse/YARN-8859 > Project: Hadoop YARN > Issue Type: Sub-task > Components: router >Reporter: Bibin Chundatt >Assignee: Minni Mittal >Priority: Major > Attachments: YARN-8859.v1.patch, YARN-8859.v2.patch, > YARN-8859.v3.patch > > > Similar to all other yarn services. > RouterClientRMService and RouterWebServices api/rest call should have Audit > logging. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10792) Set Completed AppAttempt LogsLink to Log Server Url
[ https://issues.apache.org/jira/browse/YARN-10792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinaba Sarkar updated YARN-10792: --- Attachment: YARN-10792-002.patch > Set Completed AppAttempt LogsLink to Log Server Url > --- > > Key: YARN-10792 > URL: https://issues.apache.org/jira/browse/YARN-10792 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp >Affects Versions: 3.3.0 >Reporter: Prabhu Joseph >Assignee: Abhinaba Sarkar >Priority: Major > Attachments: YARN-10792-001.patch, YARN-10792-002.patch > > > Completed AppAttempts listed by YARN UI has logslink pointing to the > NodeManager containerlogs url. The completed container logs will be under > aggregated log path and so NM ContainerLogsPage redirects to Log Server Url. > On frequent Scale Down, these NMs won't be available and so makes difficulty > to look for appattempt logs of completed apps from RM UI. Setting the > logslink for Completed AppAttempts to LogServer url will avoid this issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10807) Parents node labels are incorrectly added to child queues in weight mode
[ https://issues.apache.org/jira/browse/YARN-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358486#comment-17358486 ] Qi Zhu edited comment on YARN-10807 at 6/7/21, 9:54 AM: Thanks [~bteke] for this work. If we'd better skip the not existed node label also in sum logic, even thought we will sum 0 when it not exisit: {code:java} for (CSQueue queue : childQueues) { float weight = Math.max(0, queue.getQueueCapacities().getWeight(nodeLabel)); sumOfWeight += weight; } {code} Other things LGTM. Thanks. was (Author: zhuqi): Thanks [~bteke] for this work. If we can skip the not existed also in sum logic, even thought we will sum 0 when it not exisit: {code:java} for (CSQueue queue : childQueues) { float weight = Math.max(0, queue.getQueueCapacities().getWeight(nodeLabel)); sumOfWeight += weight; } {code} Other things LGTM. Thanks. > Parents node labels are incorrectly added to child queues in weight mode > - > > Key: YARN-10807 > URL: https://issues.apache.org/jira/browse/YARN-10807 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10807.001.patch > > > In ParentQueue.updateClusterResource when calculating the normalized weights > CS will iterate through the parent's nodelabels. If the parent has a node > label that a specific child doesn't it will incorrectly added to the child's > node label list through the queueCapacities.setNormalizedWeights(label, > weight) call: > {code:java} > // Normalize weight of children > if (getCapacityConfigurationTypeForQueues(childQueues) > == QueueCapacityType.WEIGHT) { > for (String nodeLabel : queueCapacities.getExistingNodeLabels()) { > float sumOfWeight = 0; > for (CSQueue queue : childQueues) { > float weight = Math.max(0, > queue.getQueueCapacities().getWeight(nodeLabel)); > sumOfWeight += weight; > } > // When sum of weight == 0, skip setting normalized_weight (so > // normalized weight will be 0). > if (Math.abs(sumOfWeight) > 1e-6) { > for (CSQueue queue : childQueues) { > queue.getQueueCapacities().setNormalizedWeight(nodeLabel, > queue.getQueueCapacities().getWeight(nodeLabel) / > sumOfWeight); > } > } > } > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10807) Parents node labels are incorrectly added to child queues in weight mode
[ https://issues.apache.org/jira/browse/YARN-10807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358486#comment-17358486 ] Qi Zhu commented on YARN-10807: --- Thanks [~bteke] for this work. If we can skip the not existed also in sum logic, even thought we will sum 0 when it not exisit: {code:java} for (CSQueue queue : childQueues) { float weight = Math.max(0, queue.getQueueCapacities().getWeight(nodeLabel)); sumOfWeight += weight; } {code} Other things LGTM. Thanks. > Parents node labels are incorrectly added to child queues in weight mode > - > > Key: YARN-10807 > URL: https://issues.apache.org/jira/browse/YARN-10807 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10807.001.patch > > > In ParentQueue.updateClusterResource when calculating the normalized weights > CS will iterate through the parent's nodelabels. If the parent has a node > label that a specific child doesn't it will incorrectly added to the child's > node label list through the queueCapacities.setNormalizedWeights(label, > weight) call: > {code:java} > // Normalize weight of children > if (getCapacityConfigurationTypeForQueues(childQueues) > == QueueCapacityType.WEIGHT) { > for (String nodeLabel : queueCapacities.getExistingNodeLabels()) { > float sumOfWeight = 0; > for (CSQueue queue : childQueues) { > float weight = Math.max(0, > queue.getQueueCapacities().getWeight(nodeLabel)); > sumOfWeight += weight; > } > // When sum of weight == 0, skip setting normalized_weight (so > // normalized weight will be 0). > if (Math.abs(sumOfWeight) > 1e-6) { > for (CSQueue queue : childQueues) { > queue.getQueueCapacities().setNormalizedWeight(nodeLabel, > queue.getQueueCapacities().getWeight(nodeLabel) / > sumOfWeight); > } > } > } > } > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org