[jira] [Commented] (YARN-10731) Please upgrade jquery to 3.6.0
[ https://issues.apache.org/jira/browse/YARN-10731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343701#comment-17343701 ] Wei-Chiu Chuang commented on YARN-10731: cherrypicked HADOOP-17283,HADOOP-17286,HADOOP-17302 into branch-3.3. Resolve. > Please upgrade jquery to 3.6.0 > -- > > Key: YARN-10731 > URL: https://issues.apache.org/jira/browse/YARN-10731 > Project: Hadoop YARN > Issue Type: Improvement > Components: buid, security, webapp >Affects Versions: 3.3.0, 3.2.1, 3.2.2 >Reporter: helen huang >Priority: Major > > Our fortify scan picked up a couple of security issues with the current > jquery lib being used by hadoop-yarn-common. Please upgrade it to the latest > version available (3.6.0). Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10324) Fetch data from NodeManager may case read timeout when disk is busy
[
https://issues.apache.org/jira/browse/YARN-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated YARN-10324:
---
Target Version/s: 3.3.2, 2.7.8 (was: 2.7.8, 3.3.1)
> Fetch data from NodeManager may case read timeout when disk is busy
> ---
>
> Key: YARN-10324
> URL: https://issues.apache.org/jira/browse/YARN-10324
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: auxservices
>Affects Versions: 2.7.0, 3.2.1
>Reporter: Yao Guangdong
>Priority: Minor
> Labels: patch
> Attachments: YARN-10324.001.patch, YARN-10324.002.patch
>
>
> With the cluster size become more and more big.The cost time on Reduce
> fetch Map's result from NodeManager become more and more long.We often see
> the WARN logs in the reduce's logs as follow.
> {quote}2020-06-19 15:43:15,522 WARN [fetcher#8]
> org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to
> TX-196-168-211.com:13562 with 5 map outputs
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735)
> at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587)
> at
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
> at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:434)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:400)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:271)
> at
> org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:330)
> at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198)
> {quote}
> We check the NodeManager server find that the disk IO util and connections
> became very high when the read timeout happened.We analyze that if we have
> 20,000 maps and 1,000 reduces which will make NodeManager generate 20 million
> times IO stream operate in the shuffle phase.If the reduce fetch data size is
> very small from map output files.Which make the disk IO util become very high
> in big cluster.Then read timeout happened frequently.The application finished
> time become longer.
> We find ShuffleHandler have IndexCache for cache file.out.index file.Then we
> want to change the small IO to big IO which can reduce the small disk IO
> times. So we try to cache all the small file data(file.out) in memory when
> the first fetch request come.Then the others fetch request only need read
> data from memory avoid disk IO operation.After we cache data to memory we
> find the read timeout disappeared.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10138) Document the new JHS API
[ https://issues.apache.org/jira/browse/YARN-10138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated YARN-10138: --- Target Version/s: 3.4.0, 3.3.2 (was: 3.4.0, 3.3.1) > Document the new JHS API > > > Key: YARN-10138 > URL: https://issues.apache.org/jira/browse/YARN-10138 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.3.0 >Reporter: Adam Antal >Assignee: Adam Antal >Priority: Major > > A new API has been introduced in YARN-10028, but we did not document it in > the JHS API documentation. Let's add it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10731) Please upgrade jquery to 3.6.0
[ https://issues.apache.org/jira/browse/YARN-10731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved YARN-10731. Resolution: Duplicate > Please upgrade jquery to 3.6.0 > -- > > Key: YARN-10731 > URL: https://issues.apache.org/jira/browse/YARN-10731 > Project: Hadoop YARN > Issue Type: Improvement > Components: buid, security, webapp >Affects Versions: 3.3.0, 3.2.1, 3.2.2 >Reporter: helen huang >Priority: Major > > Our fortify scan picked up a couple of security issues with the current > jquery lib being used by hadoop-yarn-common. Please upgrade it to the latest > version available (3.6.0). Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10731) Please upgrade jquery to 3.6.0
[ https://issues.apache.org/jira/browse/YARN-10731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated YARN-10731: --- Target Version/s: 3.3.1 (was: 3.4.0, 3.3.1) > Please upgrade jquery to 3.6.0 > -- > > Key: YARN-10731 > URL: https://issues.apache.org/jira/browse/YARN-10731 > Project: Hadoop YARN > Issue Type: Improvement > Components: buid, security, webapp >Affects Versions: 3.3.0, 3.2.1, 3.2.2 >Reporter: helen huang >Priority: Major > > Our fortify scan picked up a couple of security issues with the current > jquery lib being used by hadoop-yarn-common. Please upgrade it to the latest > version available (3.6.0). Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10731) Please upgrade jquery to 3.6.0
[ https://issues.apache.org/jira/browse/YARN-10731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343685#comment-17343685 ] Wei-Chiu Chuang commented on YARN-10731: branch-3.3 is still on jquery 3.4.1. I'll look into this. > Please upgrade jquery to 3.6.0 > -- > > Key: YARN-10731 > URL: https://issues.apache.org/jira/browse/YARN-10731 > Project: Hadoop YARN > Issue Type: Improvement > Components: buid, security, webapp >Affects Versions: 3.3.0, 3.2.1, 3.2.2 >Reporter: helen huang >Priority: Major > > Our fortify scan picked up a couple of security issues with the current > jquery lib being used by hadoop-yarn-common. Please upgrade it to the latest > version available (3.6.0). Thanks! -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10753) Document the removal of FS default queue creation
[
https://issues.apache.org/jira/browse/YARN-10753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343592#comment-17343592
]
Hadoop QA commented on YARN-10753:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 37m
16s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:blue}0{color} | {color:blue} markdownlint {color} | {color:blue} 0m
0s{color} | {color:blue}{color} | {color:blue} markdownlint was not available.
{color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 29m
42s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
48m 33s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
16s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
20s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 20s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} || ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
44s{color} | {color:green}{color} | {color:green} The patch does not generate
ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}106m 17s{color} |
{color:black}{color} | {color:black}{color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/981/artifact/out/Dockerfile
|
| JIRA Issue | YARN-10753 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13025386/YARN-10753.001.patch |
| Optional Tests | dupname asflicense mvnsite markdownlint |
| uname | Linux b199be2b3c0b 4.15.0-136-generic #140-Ubuntu SMP Thu Jan 28
05:20:47 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / fdd20a3cf4c |
| Max. process+thread count | 598 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site |
| Console output |
https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/981/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.13.0-SNAPSHOT https://yetus.apache.org |
This message was automatically generated.
> Document the removal of FS default queue creation
> -
>
> Key: YARN-10753
> URL: https://issues.apache.org/jira/browse/YARN-10753
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: fairscheduler
>Affects Versions: 3.4.0
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Attachments: YARN-10753.001.patch
>
>
> In YARN-7769 the auto creation of the root.default queue is removed from FS
> initialization. This implacts the the
> [documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html]
> in multiple places:
> * By default, all users share a single queue, named “default”.
> * user-as-default-queue says it falls back to the default queue
> * allow-undeclared-pools is another point that mentions the default queue
> This should be changed according to the new state.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For
[jira] [Commented] (YARN-7769) FS QueueManager should not create default queue at init
[ https://issues.apache.org/jira/browse/YARN-7769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343503#comment-17343503 ] Benjamin Teke commented on YARN-7769: - [~wilfreds], [~snemeth] did some testing, haven't found any inconsistencies. Also I uploaded the doc change to YARN-10753. > FS QueueManager should not create default queue at init > --- > > Key: YARN-7769 > URL: https://issues.apache.org/jira/browse/YARN-7769 > Project: Hadoop YARN > Issue Type: Bug > Components: fairscheduler >Affects Versions: 3.1.0 >Reporter: Wilfred Spiegelenburg >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-7769.001.patch, YARN-7769.002.patch, > YARN-7769.003.patch > > > Currently the FairScheduler QueueManager automatically creates the default > queue. However the default queue does not need to exist. We have two possible > cases which we should handle: > * Based on the placement rule "Default" the name for the default queue might > not be default and it should be created with a different name > * There might not be a "Default" placement rule at all which removes the need > to create the queue. > We should leave the creation of the default queue to the point in time that > we can assess if it is needed or not. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10753) Document the removal of FS default queue creation
[ https://issues.apache.org/jira/browse/YARN-10753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10753: - Attachment: YARN-10753.001.patch > Document the removal of FS default queue creation > - > > Key: YARN-10753 > URL: https://issues.apache.org/jira/browse/YARN-10753 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.4.0 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10753.001.patch > > > In YARN-7769 the auto creation of the root.default queue is removed from FS > initialization. This implacts the the > [documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html] > in multiple places: > * By default, all users share a single queue, named “default”. > * user-as-default-queue says it falls back to the default queue > * allow-undeclared-pools is another point that mentions the default queue > This should be changed according to the new state. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10753) Document the removal of FS default queue creation
[ https://issues.apache.org/jira/browse/YARN-10753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10753: - Attachment: (was: YARN-10753.001.patch) > Document the removal of FS default queue creation > - > > Key: YARN-10753 > URL: https://issues.apache.org/jira/browse/YARN-10753 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.4.0 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > > In YARN-7769 the auto creation of the root.default queue is removed from FS > initialization. This implacts the the > [documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html] > in multiple places: > * By default, all users share a single queue, named “default”. > * user-as-default-queue says it falls back to the default queue > * allow-undeclared-pools is another point that mentions the default queue > This should be changed according to the new state. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10753) Document the removal of FS default queue creation
[ https://issues.apache.org/jira/browse/YARN-10753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10753: - Attachment: YARN-10753.001.patch > Document the removal of FS default queue creation > - > > Key: YARN-10753 > URL: https://issues.apache.org/jira/browse/YARN-10753 > Project: Hadoop YARN > Issue Type: Improvement > Components: fairscheduler >Affects Versions: 3.4.0 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Major > Attachments: YARN-10753.001.patch > > > In YARN-7769 the auto creation of the root.default queue is removed from FS > initialization. This implacts the the > [documentation|https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/FairScheduler.html] > in multiple places: > * By default, all users share a single queue, named “default”. > * user-as-default-queue says it falls back to the default queue > * allow-undeclared-pools is another point that mentions the default queue > This should be changed according to the new state. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10763) add the speed of containers assigned metrics to ClusterMetrics
[
https://issues.apache.org/jira/browse/YARN-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343324#comment-17343324
]
Hadoop QA commented on YARN-10763:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
17s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
51s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 6s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 20m
21s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 1m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 43s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/980/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 29 unchanged - 0 fixed = 31 total (was 29) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 14s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} |
[jira] [Commented] (YARN-10763) add the speed of containers assigned metrics to ClusterMetrics
[
https://issues.apache.org/jira/browse/YARN-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343305#comment-17343305
]
Hadoop QA commented on YARN-10763:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
18s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
22s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 56s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
40s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 20m
6s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 1m
51s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/979/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 3 new + 29 unchanged - 0 fixed = 32 total (was 29) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:red}-1{color} | {color:red} shadedclient {color} | {color:red} 14m
58s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/979/artifact/out/patch-shadedclient.txt{color}
| {color:red} patch has errors when
[jira] [Comment Edited] (YARN-10571) Refactor dynamic queue handling logic
[ https://issues.apache.org/jira/browse/YARN-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343250#comment-17343250 ] Peter Bacsko edited comment on YARN-10571 at 5/12/21, 1:36 PM: --- Thanks [~gandras], committed to trunk. Also thanks [~shuzirra], [~zhuqi] for the review. was (Author: pbacsko): Thanks [~gandras], committed to trunk. > Refactor dynamic queue handling logic > - > > Key: YARN-10571 > URL: https://issues.apache.org/jira/browse/YARN-10571 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Minor > Attachments: YARN-10571.001.patch, YARN-10571.002.patch, > YARN-10571.003.patch, YARN-10571.004.patch > > > As per YARN-10506 we have introduced an other mode for auto queue creation > and a new class, which handles it. We should move the old, managed queue > related logic to CSAutoQueueHandler as well, and do additional cleanup > regarding queue management. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10571) Refactor dynamic queue handling logic
[ https://issues.apache.org/jira/browse/YARN-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343250#comment-17343250 ] Peter Bacsko commented on YARN-10571: - Thanks [~gandras], committed to trunk. > Refactor dynamic queue handling logic > - > > Key: YARN-10571 > URL: https://issues.apache.org/jira/browse/YARN-10571 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Minor > Attachments: YARN-10571.001.patch, YARN-10571.002.patch, > YARN-10571.003.patch, YARN-10571.004.patch > > > As per YARN-10506 we have introduced an other mode for auto queue creation > and a new class, which handles it. We should move the old, managed queue > related logic to CSAutoQueueHandler as well, and do additional cleanup > regarding queue management. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10763) add the speed of containers assigned metrics to ClusterMetrics
[ https://issues.apache.org/jira/browse/YARN-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chaosju updated YARN-10763: --- Attachment: YARN-10763.005.patch > add the speed of containers assigned metrics to ClusterMetrics > --- > > Key: YARN-10763 > URL: https://issues.apache.org/jira/browse/YARN-10763 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: chaosju >Assignee: chaosju >Priority: Minor > Attachments: YARN-10763.001.patch, YARN-10763.002.patch, > YARN-10763.003.patch, YARN-10763.004.patch, YARN-10763.005.patch, > screenshot-1.png > > > It'd be good to have ContainerAssignedNum/Second in ClusterMetrics for > measuring cluster throughput. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10763) add the speed of containers assigned metrics to ClusterMetrics
[ https://issues.apache.org/jira/browse/YARN-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343232#comment-17343232 ] chaosju commented on YARN-10763: [~pbacsko] Thank you for your detailed and patient reviews, I have got so much through your comments. I will fix it latter. Thank you again. > add the speed of containers assigned metrics to ClusterMetrics > --- > > Key: YARN-10763 > URL: https://issues.apache.org/jira/browse/YARN-10763 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: chaosju >Assignee: chaosju >Priority: Minor > Attachments: YARN-10763.001.patch, YARN-10763.002.patch, > YARN-10763.003.patch, YARN-10763.004.patch, screenshot-1.png > > > It'd be good to have ContainerAssignedNum/Second in ClusterMetrics for > measuring cluster throughput. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10571) Refactor dynamic queue handling logic
[ https://issues.apache.org/jira/browse/YARN-10571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343227#comment-17343227 ] Peter Bacsko commented on YARN-10571: - +1 LGTM > Refactor dynamic queue handling logic > - > > Key: YARN-10571 > URL: https://issues.apache.org/jira/browse/YARN-10571 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Minor > Attachments: YARN-10571.001.patch, YARN-10571.002.patch, > YARN-10571.003.patch, YARN-10571.004.patch > > > As per YARN-10506 we have introduced an other mode for auto queue creation > and a new class, which handles it. We should move the old, managed queue > related logic to CSAutoQueueHandler as well, and do additional cleanup > regarding queue management. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10763) add the speed of containers assigned metrics to ClusterMetrics
[
https://issues.apache.org/jira/browse/YARN-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343152#comment-17343152
]
Peter Bacsko commented on YARN-10763:
-
Some further comments:
1. {{private volatile AtomicLong numContainersAssigned = new AtomicLong(0);}} -
this doesn't have to be "volatile".
2. "containersAssignedCounter" should be renamed to something else. If this is
an executor, then use the name "assignCounterExecutor" or something like that.
3. Also, "containersAssignedCounter" should be not be a static field. It's part
of an object which just happens to be a singleton.
4. Initialize "containersAssignedCounter" in the constructor.
5. Use something else for the thread name instead of
"ContainersAssigned_Counter", for example, "ContainerAssignmentCounterThread".
6. "numContainersAssignedLast" should be int and not long. Therefore, you don't
need to the type cast at {{numContainerAssignedPerSecond.set((int)n);}}.
7. Also, it would be best to reset the counter of "numContainersAssigned" to 0.
Now it's AtomicLong and I think it's unlikely to overflow, but if we're just
counting in an interval, it makes sense to reset it. Also, it does not have to
be long, but just int. So this is preferred: {{numContainersAssignedLast =
numContainersAssigned.getAndSet(0)}}.
8. {{containersAssignedCounter.shutdown();}} --> use {{shutdownNow()}} to
immediately stop the executor.
9. The test case:
{noformat}
@Test
public void testClusterMetrics() throws Exception {
assert(metrics != null);
Assert.assertTrue(!metrics.numContainerAssignedPerSecond.changed());
metrics.incrNumContainerAssigned();
Thread.sleep(2000);
Assert.assertEquals(metrics.getnumContainerAssignedPerSecond(),1L);
}
{noformat}
The first is the default Java "assert", which only takes effect with the "-ea"
cmd line switch. Just remove it.
Also, instead of a fixed {{Thread.sleep()}}, use:
{noformat}
GenericTestUtils.waitFor(new Supplier() {
@Override
public Boolean get() {
return metrics.getnumContainerAssignedPerSecond() == 1;
}
}, 500, 5000);
{noformat}
> add the speed of containers assigned metrics to ClusterMetrics
> ---
>
> Key: YARN-10763
> URL: https://issues.apache.org/jira/browse/YARN-10763
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: chaosju
>Assignee: chaosju
>Priority: Minor
> Attachments: YARN-10763.001.patch, YARN-10763.002.patch,
> YARN-10763.003.patch, YARN-10763.004.patch, screenshot-1.png
>
>
> It'd be good to have ContainerAssignedNum/Second in ClusterMetrics for
> measuring cluster throughput.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10763) add the speed of containers assigned metrics to ClusterMetrics
[ https://issues.apache.org/jira/browse/YARN-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343139#comment-17343139 ] Peter Bacsko commented on YARN-10763: - [~chaosju] I added you to the list of contributors so you can assign the JIRA to yourself. Are the unit test failures related? The number of failed tests looks huge. > add the speed of containers assigned metrics to ClusterMetrics > --- > > Key: YARN-10763 > URL: https://issues.apache.org/jira/browse/YARN-10763 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: chaosju >Assignee: chaosju >Priority: Minor > Attachments: YARN-10763.001.patch, YARN-10763.002.patch, > YARN-10763.003.patch, YARN-10763.004.patch, screenshot-1.png > > > It'd be good to have ContainerAssignedNum/Second in ClusterMetrics for > measuring cluster throughput. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10763) add the speed of containers assigned metrics to ClusterMetrics
[ https://issues.apache.org/jira/browse/YARN-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko reassigned YARN-10763: --- Assignee: chaosju > add the speed of containers assigned metrics to ClusterMetrics > --- > > Key: YARN-10763 > URL: https://issues.apache.org/jira/browse/YARN-10763 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: chaosju >Assignee: chaosju >Priority: Minor > Attachments: YARN-10763.001.patch, YARN-10763.002.patch, > YARN-10763.003.patch, YARN-10763.004.patch, screenshot-1.png > > > It'd be good to have ContainerAssignedNum/Second in ClusterMetrics for > measuring cluster throughput. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10763) add the speed of containers assigned metrics to ClusterMetrics
[ https://issues.apache.org/jira/browse/YARN-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343131#comment-17343131 ] chaosju commented on YARN-10763: I have attached .004 patch with fixed whitespace、spotbugs etc > add the speed of containers assigned metrics to ClusterMetrics > --- > > Key: YARN-10763 > URL: https://issues.apache.org/jira/browse/YARN-10763 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: chaosju >Priority: Minor > Attachments: YARN-10763.001.patch, YARN-10763.002.patch, > YARN-10763.003.patch, YARN-10763.004.patch, screenshot-1.png > > > It'd be good to have ContainerAssignedNum/Second in ClusterMetrics for > measuring cluster throughput. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10763) add the speed of containers assigned metrics to ClusterMetrics
[ https://issues.apache.org/jira/browse/YARN-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chaosju updated YARN-10763: --- Attachment: YARN-10763.004.patch > add the speed of containers assigned metrics to ClusterMetrics > --- > > Key: YARN-10763 > URL: https://issues.apache.org/jira/browse/YARN-10763 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: chaosju >Priority: Minor > Attachments: YARN-10763.001.patch, YARN-10763.002.patch, > YARN-10763.003.patch, YARN-10763.004.patch, screenshot-1.png > > > It'd be good to have ContainerAssignedNum/Second in ClusterMetrics for > measuring cluster throughput. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10517) QueueMetrics has incorrect Allocated Resource when labelled partitions updated
[
https://issues.apache.org/jira/browse/YARN-10517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343084#comment-17343084
]
Qi Zhu commented on YARN-10517:
---
Thanks [~zhanqi.cai] for confirm.
cc [~pbacsko] [~ebadger] [~epayne]
> QueueMetrics has incorrect Allocated Resource when labelled partitions updated
> --
>
> Key: YARN-10517
> URL: https://issues.apache.org/jira/browse/YARN-10517
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.0, 3.3.0
>Reporter: sibyl.lv
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10517-branch-3.2.001.patch, YARN-10517.001.patch,
> wrong metrics.png
>
>
> After https://issues.apache.org/jira/browse/YARN-9596, QueueMetrics still has
> incorrect allocated jmx, such as {color:#660e7a}allocatedMB,
> {color}{color:#660e7a}allocatedVCores and
> {color}{color:#660e7a}allocatedContainers, {color}when the node partition is
> updated from "DEFAULT" to other label and there are running applications.
> Steps to reproduce
> ==
> # Configure capacity-scheduler.xml with label configuration
> # Submit one application to default partition and run
> # Add label "tpcds" to cluster and replace label on node1 and node2 to be
> "tpcds" when the above application is running
> # Note down "VCores Used" at Web UI
> # When the application is finished, the metrics get wrong (screenshots
> attached).
> ==
>
> FiCaSchedulerApp doesn't update queue metrics when CapacityScheduler handles
> this event {color:#660e7a}NODE_LABELS_UPDATE.{color}
> So we should release container resource from old partition and add used
> resource to new partition, just as updating queueUsage.
> {code:java}
> // code placeholder
> public void nodePartitionUpdated(RMContainer rmContainer, String oldPartition,
> String newPartition) {
> Resource containerResource = rmContainer.getAllocatedResource();
> this.attemptResourceUsage.decUsed(oldPartition, containerResource);
> this.attemptResourceUsage.incUsed(newPartition, containerResource);
> getCSLeafQueue().decUsedResource(oldPartition, containerResource, this);
> getCSLeafQueue().incUsedResource(newPartition, containerResource, this);
> // Update new partition name if container is AM and also update AM resource
> if (rmContainer.isAMContainer()) {
> setAppAMNodePartitionName(newPartition);
> this.attemptResourceUsage.decAMUsed(oldPartition, containerResource);
> this.attemptResourceUsage.incAMUsed(newPartition, containerResource);
> getCSLeafQueue().decAMUsedResource(oldPartition, containerResource, this);
> getCSLeafQueue().incAMUsedResource(newPartition, containerResource, this);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10763) add the speed of containers assigned metrics to ClusterMetrics
[
https://issues.apache.org/jira/browse/YARN-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343049#comment-17343049
]
Hadoop QA commented on YARN-10763:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
35s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 30s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 20m
45s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 1m
55s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
57s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/976/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 3 new + 29 unchanged - 0 fixed = 32 total (was 29) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/976/artifact/out/whitespace-eol.txt{color}
| {color:red} The patch has 3 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} shadedclient {color} |
[jira] [Commented] (YARN-10763) add the speed of containers assigned metrics to ClusterMetrics
[
https://issues.apache.org/jira/browse/YARN-10763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343045#comment-17343045
]
Hadoop QA commented on YARN-10763:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
15s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 1
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
14s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
56s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 11s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
39s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 20m
31s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 1m
58s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
52s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
0s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
49s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 40s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/975/artifact/out/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt{color}
| {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 3 new + 29 unchanged - 0 fixed = 32 total (was 29) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} |
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/975/artifact/out/whitespace-eol.txt{color}
| {color:red} The patch has 3 line(s) that end in whitespace. Use git apply
--whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} shadedclient {color} |
[jira] [Comment Edited] (YARN-10517) QueueMetrics has incorrect Allocated Resource when labelled partitions updated
[
https://issues.apache.org/jira/browse/YARN-10517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343037#comment-17343037
]
Zhanqi Cai edited comment on YARN-10517 at 5/12/21, 6:46 AM:
-
Thanks [~zhuqi] for your contribution.
It looks good to me and solves my problem.
was (Author: zhanqi.cai):
It looks good to me and works now.
> QueueMetrics has incorrect Allocated Resource when labelled partitions updated
> --
>
> Key: YARN-10517
> URL: https://issues.apache.org/jira/browse/YARN-10517
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.0, 3.3.0
>Reporter: sibyl.lv
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10517-branch-3.2.001.patch, YARN-10517.001.patch,
> wrong metrics.png
>
>
> After https://issues.apache.org/jira/browse/YARN-9596, QueueMetrics still has
> incorrect allocated jmx, such as {color:#660e7a}allocatedMB,
> {color}{color:#660e7a}allocatedVCores and
> {color}{color:#660e7a}allocatedContainers, {color}when the node partition is
> updated from "DEFAULT" to other label and there are running applications.
> Steps to reproduce
> ==
> # Configure capacity-scheduler.xml with label configuration
> # Submit one application to default partition and run
> # Add label "tpcds" to cluster and replace label on node1 and node2 to be
> "tpcds" when the above application is running
> # Note down "VCores Used" at Web UI
> # When the application is finished, the metrics get wrong (screenshots
> attached).
> ==
>
> FiCaSchedulerApp doesn't update queue metrics when CapacityScheduler handles
> this event {color:#660e7a}NODE_LABELS_UPDATE.{color}
> So we should release container resource from old partition and add used
> resource to new partition, just as updating queueUsage.
> {code:java}
> // code placeholder
> public void nodePartitionUpdated(RMContainer rmContainer, String oldPartition,
> String newPartition) {
> Resource containerResource = rmContainer.getAllocatedResource();
> this.attemptResourceUsage.decUsed(oldPartition, containerResource);
> this.attemptResourceUsage.incUsed(newPartition, containerResource);
> getCSLeafQueue().decUsedResource(oldPartition, containerResource, this);
> getCSLeafQueue().incUsedResource(newPartition, containerResource, this);
> // Update new partition name if container is AM and also update AM resource
> if (rmContainer.isAMContainer()) {
> setAppAMNodePartitionName(newPartition);
> this.attemptResourceUsage.decAMUsed(oldPartition, containerResource);
> this.attemptResourceUsage.incAMUsed(newPartition, containerResource);
> getCSLeafQueue().decAMUsedResource(oldPartition, containerResource, this);
> getCSLeafQueue().incAMUsedResource(newPartition, containerResource, this);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10517) QueueMetrics has incorrect Allocated Resource when labelled partitions updated
[
https://issues.apache.org/jira/browse/YARN-10517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343037#comment-17343037
]
Zhanqi Cai commented on YARN-10517:
---
It looks good to me and works now.
> QueueMetrics has incorrect Allocated Resource when labelled partitions updated
> --
>
> Key: YARN-10517
> URL: https://issues.apache.org/jira/browse/YARN-10517
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.0, 3.3.0
>Reporter: sibyl.lv
>Assignee: Qi Zhu
>Priority: Major
> Attachments: YARN-10517-branch-3.2.001.patch, YARN-10517.001.patch,
> wrong metrics.png
>
>
> After https://issues.apache.org/jira/browse/YARN-9596, QueueMetrics still has
> incorrect allocated jmx, such as {color:#660e7a}allocatedMB,
> {color}{color:#660e7a}allocatedVCores and
> {color}{color:#660e7a}allocatedContainers, {color}when the node partition is
> updated from "DEFAULT" to other label and there are running applications.
> Steps to reproduce
> ==
> # Configure capacity-scheduler.xml with label configuration
> # Submit one application to default partition and run
> # Add label "tpcds" to cluster and replace label on node1 and node2 to be
> "tpcds" when the above application is running
> # Note down "VCores Used" at Web UI
> # When the application is finished, the metrics get wrong (screenshots
> attached).
> ==
>
> FiCaSchedulerApp doesn't update queue metrics when CapacityScheduler handles
> this event {color:#660e7a}NODE_LABELS_UPDATE.{color}
> So we should release container resource from old partition and add used
> resource to new partition, just as updating queueUsage.
> {code:java}
> // code placeholder
> public void nodePartitionUpdated(RMContainer rmContainer, String oldPartition,
> String newPartition) {
> Resource containerResource = rmContainer.getAllocatedResource();
> this.attemptResourceUsage.decUsed(oldPartition, containerResource);
> this.attemptResourceUsage.incUsed(newPartition, containerResource);
> getCSLeafQueue().decUsedResource(oldPartition, containerResource, this);
> getCSLeafQueue().incUsedResource(newPartition, containerResource, this);
> // Update new partition name if container is AM and also update AM resource
> if (rmContainer.isAMContainer()) {
> setAppAMNodePartitionName(newPartition);
> this.attemptResourceUsage.decAMUsed(oldPartition, containerResource);
> this.attemptResourceUsage.incAMUsed(newPartition, containerResource);
> getCSLeafQueue().decAMUsedResource(oldPartition, containerResource, this);
> getCSLeafQueue().incAMUsedResource(newPartition, containerResource, this);
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10767) Yarn Logs Command retrying on Standby RM for 30 times
[
https://issues.apache.org/jira/browse/YARN-10767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17343026#comment-17343026
]
Hadoop QA commented on YARN-10767:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
58s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to
include any new or modified tests. Please justify why no new tests are needed
for this patch. Also please list what manual steps were performed to verify
this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
0s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 20s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
42s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 17m
53s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 1m
57s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
37s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch has no whitespace
issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 56s{color} | {color:green}{color} | {color:green} patch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
35s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} |
|
