[jira] [Commented] (YARN-11650) Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue
[
https://issues.apache.org/jira/browse/YARN-11650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17811738#comment-17811738
]
ASF GitHub Bot commented on YARN-11650:
---
hadoop-yetus commented on PR #6490:
URL: https://github.com/apache/hadoop/pull/6490#issuecomment-1913991706
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 45s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 39m 5s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 44s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 105m 18s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 250m 30s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6490/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6490 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 16629d116fa6 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 8fd3663da83ada01238f6ae04b2a8cef9e731a09 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6490/4/testReport/ |
| Max. process+thread count | 897 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-ya
[jira] [Commented] (YARN-11650) Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue
[ https://issues.apache.org/jira/browse/YARN-11650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17811874#comment-17811874 ] ASF GitHub Bot commented on YARN-11650: --- slfan1989 merged PR #6490: URL: https://github.com/apache/hadoop/pull/6490 > Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, > FiCaSchedulerApp and AbstractCSQueue > --- > > Key: YARN-11650 > URL: https://issues.apache.org/jira/browse/YARN-11650 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Labels: pull-request-available > > In classes related to MultiNodePolicy support, some variable names do not > accurately reflect their true meanings. For instance, a variable named > *queue* is actually representing the class name of a policy, and a variable > named *policyName* denotes the class name of the policy. This may cause > confusion for the readers of the code. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11650) Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue
[ https://issues.apache.org/jira/browse/YARN-11650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17811875#comment-17811875 ] ASF GitHub Bot commented on YARN-11650: --- slfan1989 commented on PR #6490: URL: https://github.com/apache/hadoop/pull/6490#issuecomment-1914714875 @yangjiandan Thanks for the contribution! merged into trunk. > Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, > FiCaSchedulerApp and AbstractCSQueue > --- > > Key: YARN-11650 > URL: https://issues.apache.org/jira/browse/YARN-11650 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, resourcemanager >Affects Versions: 3.5.0 >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > > In classes related to MultiNodePolicy support, some variable names do not > accurately reflect their true meanings. For instance, a variable named > *queue* is actually representing the class name of a policy, and a variable > named *policyName* denotes the class name of the policy. This may cause > confusion for the readers of the code. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11653) Add Totoal_Memory and Total_Vcores columns in Nodes page
[ https://issues.apache.org/jira/browse/YARN-11653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17811876#comment-17811876 ] ASF GitHub Bot commented on YARN-11653: --- slfan1989 commented on PR #6501: URL: https://github.com/apache/hadoop/pull/6501#issuecomment-1914716020 @yangjiandan Can we trigger compilation again? > Add Totoal_Memory and Total_Vcores columns in Nodes page > > > Key: YARN-11653 > URL: https://issues.apache.org/jira/browse/YARN-11653 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Labels: pull-request-available > > Currently, the RM nodes page includes used and available memory/vcore , but > it lacks a total column, which is not intuitive enough for users. When the > resource capacities of nodes in the cluster vary widely, we may need to sort > the nodes to facilitate the comparison of metrics among same types of nodes. > Therefore, it is necessary to add columns for total CPU/memory on the nodes > page. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11362) Fix several typos in YARN codebase of misspelled resource
[
https://issues.apache.org/jira/browse/YARN-11362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17811877#comment-17811877
]
ASF GitHub Bot commented on YARN-11362:
---
slfan1989 commented on code in PR #6474:
URL: https://github.com/apache/hadoop/pull/6474#discussion_r1469607772
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/exceptions/SchedulerInvalidResourceRequestException.java:
##
@@ -29,19 +29,19 @@
*/
@InterfaceAudience.Private
@InterfaceStability.Unstable
-public class SchedulerInvalidResoureRequestException extends
YarnRuntimeException {
+public class SchedulerInvalidResourceRequestException extends
YarnRuntimeException {
private static final long serialVersionUID = 10081123982L;
- public SchedulerInvalidResoureRequestException(String message) {
+ public SchedulerInvalidResourceRequestException(String message) {
super(message);
}
- public SchedulerInvalidResoureRequestException(Throwable cause) {
+ public SchedulerInvalidResourceRequestException(Throwable cause) {
super(cause);
}
- public SchedulerInvalidResoureRequestException(String message,
- Throwable cause) {
+ public SchedulerInvalidResourceRequestException(String message,
+ Throwable cause) {
Review Comment:
We need to pay attention to the indentation, usually 5 characters.
> Fix several typos in YARN codebase of misspelled resource
> -
>
> Key: YARN-11362
> URL: https://issues.apache.org/jira/browse/YARN-11362
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Labels: newbie, newbie++, pull-request-available
>
> I noticed that in YARN's codebase, there are several occassions of misspelled
> resource as "Resoure".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11362) Fix several typos in YARN codebase of misspelled resource
[
https://issues.apache.org/jira/browse/YARN-11362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812018#comment-17812018
]
ASF GitHub Bot commented on YARN-11362:
---
EremenkoValentin commented on code in PR #6474:
URL: https://github.com/apache/hadoop/pull/6474#discussion_r1470038367
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/exceptions/SchedulerInvalidResourceRequestException.java:
##
@@ -29,19 +29,19 @@
*/
@InterfaceAudience.Private
@InterfaceStability.Unstable
-public class SchedulerInvalidResoureRequestException extends
YarnRuntimeException {
+public class SchedulerInvalidResourceRequestException extends
YarnRuntimeException {
private static final long serialVersionUID = 10081123982L;
- public SchedulerInvalidResoureRequestException(String message) {
+ public SchedulerInvalidResourceRequestException(String message) {
super(message);
}
- public SchedulerInvalidResoureRequestException(Throwable cause) {
+ public SchedulerInvalidResourceRequestException(Throwable cause) {
super(cause);
}
- public SchedulerInvalidResoureRequestException(String message,
- Throwable cause) {
+ public SchedulerInvalidResourceRequestException(String message,
+ Throwable cause) {
Review Comment:
Thanks for your review. Corrected
> Fix several typos in YARN codebase of misspelled resource
> -
>
> Key: YARN-11362
> URL: https://issues.apache.org/jira/browse/YARN-11362
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Labels: newbie, newbie++, pull-request-available
>
> I noticed that in YARN's codebase, there are several occassions of misspelled
> resource as "Resoure".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11362) Fix several typos in YARN codebase of misspelled resource
[
https://issues.apache.org/jira/browse/YARN-11362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812080#comment-17812080
]
ASF GitHub Bot commented on YARN-11362:
---
hadoop-yetus commented on PR #6474:
URL: https://github.com/apache/hadoop/pull/6474#issuecomment-1915699065
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 48s | | trunk passed |
| +1 :green_heart: | compile | 3m 30s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 3m 17s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 57s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 33s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 27s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 26s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 19s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 37s | | the patch passed |
| +1 :green_heart: | compile | 3m 12s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 3m 12s | | the patch passed |
| +1 :green_heart: | compile | 3m 10s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 3m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 52s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 8s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 42s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 4m 35s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 18s | |
hadoop-yarn-server-applicationhistoryservice in the patch passed. |
| +1 :green_heart: | unit | 86m 11s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 25m 24s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 240m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6474/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6474 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 607eac38a085 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 220f46405413647d4acdf7d9b9114971fb53 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.a
[jira] [Commented] (YARN-11653) Add Totoal_Memory and Total_Vcores columns in Nodes page
[ https://issues.apache.org/jira/browse/YARN-11653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812116#comment-17812116 ] ASF GitHub Bot commented on YARN-11653: --- yangjiandan commented on PR #6501: URL: https://github.com/apache/hadoop/pull/6501#issuecomment-1915951700 @slfan1989 Thanks for reviewing very much, I'll tregger it. > Add Totoal_Memory and Total_Vcores columns in Nodes page > > > Key: YARN-11653 > URL: https://issues.apache.org/jira/browse/YARN-11653 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Labels: pull-request-available > > Currently, the RM nodes page includes used and available memory/vcore , but > it lacks a total column, which is not intuitive enough for users. When the > resource capacities of nodes in the cluster vary widely, we may need to sort > the nodes to facilitate the comparison of metrics among same types of nodes. > Therefore, it is necessary to add columns for total CPU/memory on the nodes > page. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11653) Add Totoal_Memory and Total_Vcores columns in Nodes page
[
https://issues.apache.org/jira/browse/YARN-11653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812162#comment-17812162
]
ASF GitHub Bot commented on YARN-11653:
---
hadoop-yetus commented on PR #6501:
URL: https://github.com/apache/hadoop/pull/6501#issuecomment-1916160462
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 34s | | trunk passed |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 51s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| +1 :green_heart: | compile | 0m 52s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 0m 48s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 44s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 42s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 0s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 100m 8s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 236m 52s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6501/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6501 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a0897eaf0fc6 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ffb54421e918d5bc467a6bcc930bba633a5324ef |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6501/2/testReport/ |
| Max. process+thread count | 966 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6501/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
T
[jira] [Commented] (YARN-7953) [GQ] Data structures for federation global queues calculations
[ https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812735#comment-17812735 ] ASF GitHub Bot commented on YARN-7953: -- slfan1989 commented on PR #6361: URL: https://github.com/apache/hadoop/pull/6361#issuecomment-1919291606 @goiri Can you help review this PR? Thank you very much! I will continue to follow the development of YARN-7402. > [GQ] Data structures for federation global queues calculations > -- > > Key: YARN-7953 > URL: https://issues.apache.org/jira/browse/YARN-7953 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Abhishek Modi >Priority: Major > Labels: pull-request-available > Attachments: YARN-7953-YARN-7402.v1.patch, > YARN-7953-YARN-7402.v2.patch, YARN-7953-YARN-7402.v3.patch, > YARN-7953-YARN-7402.v4.patch, YARN-7953-YARN-7402.v5.patch, > YARN-7953-YARN-7402.v6.patch, YARN-7953-YARN-7402.v7.patch, > YARN-7953-YARN-7402.v8.patch, YARN-7953.v1.patch > > > This Jira tracks data structures and helper classes used by the core > algorithms of YARN-7402 umbrella Jira (currently YARN-7403, and YARN-7834). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7953) [GQ] Data structures for federation global queues calculations
[
https://issues.apache.org/jira/browse/YARN-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812805#comment-17812805
]
ASF GitHub Bot commented on YARN-7953:
--
hadoop-yetus commented on PR #6361:
URL: https://github.com/apache/hadoop/pull/6361#issuecomment-1919525285
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | jsonlint | 0m 1s | | jsonlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 48m 4s | | trunk passed |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 41s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 1s | |
hadoop-yarn-server-globalpolicygenerator in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 137m 53s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6361/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6361 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle
jsonlint |
| uname | Linux d5a63d5a2eac 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 90890eb3b64a48d2bb1a7e760d1b478031ba2788 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6361/8/testReport/ |
| Max. process+thread count | 527 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator
|
| Console output |
https://ci-hadoop.apache.org/j
[jira] [Commented] (YARN-11362) Fix several typos in YARN codebase of misspelled resource
[ https://issues.apache.org/jira/browse/YARN-11362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17813922#comment-17813922 ] ASF GitHub Bot commented on YARN-11362: --- slfan1989 merged PR #6474: URL: https://github.com/apache/hadoop/pull/6474 > Fix several typos in YARN codebase of misspelled resource > - > > Key: YARN-11362 > URL: https://issues.apache.org/jira/browse/YARN-11362 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Labels: newbie, newbie++, pull-request-available > > I noticed that in YARN's codebase, there are several occassions of misspelled > resource as "Resoure". -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11362) Fix several typos in YARN codebase of misspelled resource
[ https://issues.apache.org/jira/browse/YARN-11362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17813923#comment-17813923 ] ASF GitHub Bot commented on YARN-11362: --- slfan1989 commented on PR #6474: URL: https://github.com/apache/hadoop/pull/6474#issuecomment-1925292462 @EremenkoValentin Thanks for the contribution, merged to trunk! > Fix several typos in YARN codebase of misspelled resource > - > > Key: YARN-11362 > URL: https://issues.apache.org/jira/browse/YARN-11362 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Labels: newbie, newbie++, pull-request-available > > I noticed that in YARN's codebase, there are several occassions of misspelled > resource as "Resoure". -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11362) Fix several typos in YARN codebase of misspelled resource
[ https://issues.apache.org/jira/browse/YARN-11362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17813931#comment-17813931 ] ASF GitHub Bot commented on YARN-11362: --- EremenkoValentin commented on PR #6474: URL: https://github.com/apache/hadoop/pull/6474#issuecomment-1925307454 > @EremenkoValentin Thanks for the contribution, merged to trunk! Do you have a JIRA account? Thanks @slfan1989 for your review and merge. I submitted a request to create an account but still have not received a response. > Fix several typos in YARN codebase of misspelled resource > - > > Key: YARN-11362 > URL: https://issues.apache.org/jira/browse/YARN-11362 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.5.0 >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Labels: newbie, newbie++, pull-request-available > Fix For: 3.5.0 > > > I noticed that in YARN's codebase, there are several occassions of misspelled > resource as "Resoure". -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11362) Fix several typos in YARN codebase of misspelled resource
[ https://issues.apache.org/jira/browse/YARN-11362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814006#comment-17814006 ] ASF GitHub Bot commented on YARN-11362: --- slfan1989 commented on PR #6474: URL: https://github.com/apache/hadoop/pull/6474#issuecomment-1925504736 > Thanks @slfan1989 for your review and merge. I submitted a request to create an account but still have not received a response. @EremenkoValentin your JIRA account has been created, Welcome to hadoop! > Fix several typos in YARN codebase of misspelled resource > - > > Key: YARN-11362 > URL: https://issues.apache.org/jira/browse/YARN-11362 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.5.0 >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Major > Labels: newbie, newbie++, pull-request-available > Fix For: 3.5.0 > > > I noticed that in YARN's codebase, there are several occassions of misspelled > resource as "Resoure". -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11654) [JDK17] TestLinuxContainerExecutorWithMocks.testStartLocalizer fails
[ https://issues.apache.org/jira/browse/YARN-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11654: -- Labels: pull-request-available (was: ) > [JDK17] TestLinuxContainerExecutorWithMocks.testStartLocalizer fails > > > Key: YARN-11654 > URL: https://issues.apache.org/jira/browse/YARN-11654 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.4.0 >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > Labels: pull-request-available > > [ERROR] TestLinuxContainerExecutorWithMocks.testStartLocalizer:310 > Expected size:<26> but was:<28> in: > <["nobody", > "test", > "0", > "application_0", > "12345", > "/bin/nmPrivateCTokensPath", > > "/Users/bilwa/code/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/tmp/nm-local-dir", > "src/test/resources", > > "/opt/homebrew/Cellar/openjdk@17/17.0.8/libexec/openjdk.jdk/Contents/Home/bin/java", > "-classpath", > > "/Users/bilwa/code/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/test-classes:/Users/bilwa/code/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/classes:/Users/bilwa/.m2/repository/org/apache/hadoop/hadoop-common/3.5.0-SNAPSHOT/hadoop-common-3.5.0-SNAPSHOT.jar:/Users/bilwa/.m2/repository/org/apache/hadoop/thirdparty/hadoop-shaded-protobuf_3_7/1.1.1/hadoop-shaded-protobuf_3_7-1.1.1.jar:/Users/bilwa/.m2/repository/com/google/guava/guava/27.0-jre/guava-27.0-jre.jar:/Users/bilwa/.m2/repository/com/google/guava/failureaccess/1.0/failureaccess-1.0.jar:/Users/bilwa/.m2/repository/com/google/guava/listenablefuture/.0-empty-to-avoid-conflict-with-guava/listenablefuture-.0-empty-to-avoid-conflict-with-guava.jar:/Users/bilwa/.m2/repository/org/checkerframework/checker-qual/2.5.2/checker-qual-2.5.2.jar:/Users/bilwa/.m2/repository/com/google/j2objc/j2objc-annotations/1.1/j2objc-annotations-1.1.jar:/Users/bilwa/.m2/repository/org/codehaus/mojo/animal-sniffer-annotations/1.17/animal-sniffer-annotations-1.17.jar:/Users/bilwa/.m2/repository/commons-cli/commons-cli/1.5.0/commons-cli-1.5.0.jar:/Users/bilwa/.m2/repository/org/apache/commons/commons-math3/3.6.1/commons-math3-3.6.1.jar:/Users/bilwa/.m2/repository/org/apache/httpcomponents/httpclient/4.5.13/httpclient-4.5.13.jar:/Users/bilwa/.m2/repository/org/apache/httpcomponents/httpcore/4.4.13/httpcore-4.4.13.jar:/Users/bilwa/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/Users/bilwa/.m2/repository/commons-io/commons-io/2.14.0/commons-io-2.14.0.jar:/Users/bilwa/.m2/repository/commons-net/commons-net/3.9.0/commons-net-3.9.0.jar:/Users/bilwa/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/bilwa/.m2/repository/jakarta/activation/jakarta.activation-api/1.2.1/jakarta.activation-api-1.2.1.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-server/9.4.53.v20231009/jetty-server-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-http/9.4.53.v20231009/jetty-http-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-io/9.4.53.v20231009/jetty-io-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-servlet/9.4.53.v20231009/jetty-servlet-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-security/9.4.53.v20231009/jetty-security-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-util-ajax/9.4.53.v20231009/jetty-util-ajax-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-webapp/9.4.53.v20231009/jetty-webapp-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-xml/9.4.53.v20231009/jetty-xml-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar:/Users/bilwa/.m2/repository/com/sun/jersey/jersey-servlet/1.19.4/jersey-servlet-1.19.4.jar:/Users/bilwa/.m2/repository/com/sun/jersey/jersey-server/1.19.4/jersey-server-1.19.4.jar:/Users/bilwa/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/bilwa/.m2/repository/commons-beanutils/commons-beanutils/1.9.4/commons-beanutils-1.9.4.jar:/Users/bilwa/.m2/repository/org/apache/commons/commons-configuration2/2.8.0/commons-configuration2-2.8.0.jar:/Users/bilwa/.m2/repository/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar:/Users/bilwa/.m2/repository/org/apache/commons/commons-text/1.10.0/commons-text-1.10.0.jar:/Users/bilwa/.m2/repository/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar:/Users/bilwa/.m2/repository/org/apache/avro/avro/1.9.2/avro-1.9.2.jar:/Users/bilwa/.m2/repository/com/google/re2j/re2j/1.1/re2j-1.1.jar:/Users/bilwa/.m2/
[jira] [Commented] (YARN-11654) [JDK17] TestLinuxContainerExecutorWithMocks.testStartLocalizer fails
[ https://issues.apache.org/jira/browse/YARN-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814334#comment-17814334 ] ASF GitHub Bot commented on YARN-11654: --- BilwaST opened a new pull request, #6528: URL: https://github.com/apache/hadoop/pull/6528 …lizer fails I have ran this test locally and verified > [JDK17] TestLinuxContainerExecutorWithMocks.testStartLocalizer fails > > > Key: YARN-11654 > URL: https://issues.apache.org/jira/browse/YARN-11654 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.4.0 >Reporter: Bilwa S T >Assignee: Bilwa S T >Priority: Major > > [ERROR] TestLinuxContainerExecutorWithMocks.testStartLocalizer:310 > Expected size:<26> but was:<28> in: > <["nobody", > "test", > "0", > "application_0", > "12345", > "/bin/nmPrivateCTokensPath", > > "/Users/bilwa/code/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/tmp/nm-local-dir", > "src/test/resources", > > "/opt/homebrew/Cellar/openjdk@17/17.0.8/libexec/openjdk.jdk/Contents/Home/bin/java", > "-classpath", > > "/Users/bilwa/code/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/test-classes:/Users/bilwa/code/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/classes:/Users/bilwa/.m2/repository/org/apache/hadoop/hadoop-common/3.5.0-SNAPSHOT/hadoop-common-3.5.0-SNAPSHOT.jar:/Users/bilwa/.m2/repository/org/apache/hadoop/thirdparty/hadoop-shaded-protobuf_3_7/1.1.1/hadoop-shaded-protobuf_3_7-1.1.1.jar:/Users/bilwa/.m2/repository/com/google/guava/guava/27.0-jre/guava-27.0-jre.jar:/Users/bilwa/.m2/repository/com/google/guava/failureaccess/1.0/failureaccess-1.0.jar:/Users/bilwa/.m2/repository/com/google/guava/listenablefuture/.0-empty-to-avoid-conflict-with-guava/listenablefuture-.0-empty-to-avoid-conflict-with-guava.jar:/Users/bilwa/.m2/repository/org/checkerframework/checker-qual/2.5.2/checker-qual-2.5.2.jar:/Users/bilwa/.m2/repository/com/google/j2objc/j2objc-annotations/1.1/j2objc-annotations-1.1.jar:/Users/bilwa/.m2/repository/org/codehaus/mojo/animal-sniffer-annotations/1.17/animal-sniffer-annotations-1.17.jar:/Users/bilwa/.m2/repository/commons-cli/commons-cli/1.5.0/commons-cli-1.5.0.jar:/Users/bilwa/.m2/repository/org/apache/commons/commons-math3/3.6.1/commons-math3-3.6.1.jar:/Users/bilwa/.m2/repository/org/apache/httpcomponents/httpclient/4.5.13/httpclient-4.5.13.jar:/Users/bilwa/.m2/repository/org/apache/httpcomponents/httpcore/4.4.13/httpcore-4.4.13.jar:/Users/bilwa/.m2/repository/commons-logging/commons-logging/1.1.3/commons-logging-1.1.3.jar:/Users/bilwa/.m2/repository/commons-io/commons-io/2.14.0/commons-io-2.14.0.jar:/Users/bilwa/.m2/repository/commons-net/commons-net/3.9.0/commons-net-3.9.0.jar:/Users/bilwa/.m2/repository/commons-collections/commons-collections/3.2.2/commons-collections-3.2.2.jar:/Users/bilwa/.m2/repository/jakarta/activation/jakarta.activation-api/1.2.1/jakarta.activation-api-1.2.1.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-server/9.4.53.v20231009/jetty-server-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-http/9.4.53.v20231009/jetty-http-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-io/9.4.53.v20231009/jetty-io-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-servlet/9.4.53.v20231009/jetty-servlet-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-security/9.4.53.v20231009/jetty-security-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-util-ajax/9.4.53.v20231009/jetty-util-ajax-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-webapp/9.4.53.v20231009/jetty-webapp-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/org/eclipse/jetty/jetty-xml/9.4.53.v20231009/jetty-xml-9.4.53.v20231009.jar:/Users/bilwa/.m2/repository/javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar:/Users/bilwa/.m2/repository/com/sun/jersey/jersey-servlet/1.19.4/jersey-servlet-1.19.4.jar:/Users/bilwa/.m2/repository/com/sun/jersey/jersey-server/1.19.4/jersey-server-1.19.4.jar:/Users/bilwa/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/Users/bilwa/.m2/repository/commons-beanutils/commons-beanutils/1.9.4/commons-beanutils-1.9.4.jar:/Users/bilwa/.m2/repository/org/apache/commons/commons-configuration2/2.8.0/commons-configuration2-2.8.0.jar:/Users/bilwa/.m2/repository/org/apache/commons/commons-lang3/3.12.0/commons-lang3-3.12.0.jar:/Users/bilwa/.m2/repository/org/apache/commons/commons-text/1.10.0/commons-text-1.10.0.jar:/Users/bilwa/.m2/repository/org/slf4j/slf4j-log4j12/1.7.30/slf4j-log4j12-1.7.30.jar:/Users/bilwa/.m2/repos
[jira] [Commented] (YARN-11654) [JDK17] TestLinuxContainerExecutorWithMocks.testStartLocalizer fails
[
https://issues.apache.org/jira/browse/YARN-11654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814390#comment-17814390
]
ASF GitHub Bot commented on YARN-11654:
---
hadoop-yetus commented on PR #6528:
URL: https://github.com/apache/hadoop/pull/6528#issuecomment-1927177125
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 46s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 32m 37s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 27s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 0 new + 3 unchanged - 1 fixed = 3 total (was 4) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 32m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 16s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 150m 0s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6528/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6528 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 274d96550475 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / be835e44742a7755c26d1b26d196db8cb2894e6a |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6528/1/testReport/ |
| Max. process+thread count | 558 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6528/1/console |
| vers
[jira] [Commented] (YARN-11387) [GPG] YARN GPG mistakenly deleted applicationid
[
https://issues.apache.org/jira/browse/YARN-11387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814839#comment-17814839
]
ASF GitHub Bot commented on YARN-11387:
---
slfan1989 commented on code in PR #6473:
URL: https://github.com/apache/hadoop/pull/6473#discussion_r1479929985
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/applicationcleaner/DefaultApplicationCleaner.java:
##
@@ -46,47 +49,37 @@ public void run() {
LOG.info("Application cleaner run at time {}", now);
FederationStateStoreFacade facade = getGPGContext().getStateStoreFacade();
+
try {
- // Get the candidate list from StateStore before calling router
- Set allStateStoreApps = new HashSet<>();
- List response =
+ // Step1. Get the candidate list from StateStore before calling router
+ List applicationHomeSubClusters =
facade.getApplicationsHomeSubCluster();
- for (ApplicationHomeSubCluster app : response) {
-allStateStoreApps.add(app.getApplicationId());
- }
- LOG.info("{} app entries in FederationStateStore",
allStateStoreApps.size());
-
- // Get the candidate list from Registry before calling router
- List allRegistryApps = getRegistryClient().getAllApplications();
- LOG.info("{} app entries in FederationRegistry",
allStateStoreApps.size());
-
- // Get the list of known apps from Router
- Set routerApps = getRouterKnownApplications();
- LOG.info("{} known applications from Router", routerApps.size());
+ LOG.info("FederationStateStore has {} applications.",
applicationHomeSubClusters.size());
- // Clean up StateStore entries
- Set toDelete =
- Sets.difference(allStateStoreApps, routerApps);
-
Review Comment:
Step 1: Retrieve all applications stored in the StateStore, which represents
all applications submitted to the Router.
Step 2: Use the Router's REST API to fetch all running tasks. This API will
invoke applications from all active SubClusters.
Step 3: Compare the results of Step 1 and Step 2 to identify applications
that exist in Step 1 but not in Step 2. Delete these applications.
> [GPG] YARN GPG mistakenly deleted applicationid
> ---
>
> Key: YARN-11387
> URL: https://issues.apache.org/jira/browse/YARN-11387
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.2.1, 3.4.0
>Reporter: zhangjunj
>Assignee: Shilun Fan
>Priority: Major
> Labels: federation, gpg, pull-request-available
> Attachments: YARN-11387-YARN-11387.v1.patch,
> yarn-gpg-mistakenly-deleted-applicationid.png
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In [YARN-7599|https://issues.apache.org/jira/browse/YARN-7599], the
> Federation can delete expired applicationid, but YARN GPG uses getRouter()
> method to obtain application information for multiple clusters. If there are
> too many applicationids that more than 200,000 , it will not be possible to
> pull all the applicationid information at one time, resulting in the
> possibility of accidental deletion. The following error is reported for spark
> component.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11387) [GPG] YARN GPG mistakenly deleted applicationid
[
https://issues.apache.org/jira/browse/YARN-11387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814841#comment-17814841
]
ASF GitHub Bot commented on YARN-11387:
---
slfan1989 commented on code in PR #6473:
URL: https://github.com/apache/hadoop/pull/6473#discussion_r1479929985
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/applicationcleaner/DefaultApplicationCleaner.java:
##
@@ -46,47 +49,37 @@ public void run() {
LOG.info("Application cleaner run at time {}", now);
FederationStateStoreFacade facade = getGPGContext().getStateStoreFacade();
+
try {
- // Get the candidate list from StateStore before calling router
- Set allStateStoreApps = new HashSet<>();
- List response =
+ // Step1. Get the candidate list from StateStore before calling router
+ List applicationHomeSubClusters =
facade.getApplicationsHomeSubCluster();
- for (ApplicationHomeSubCluster app : response) {
-allStateStoreApps.add(app.getApplicationId());
- }
- LOG.info("{} app entries in FederationStateStore",
allStateStoreApps.size());
-
- // Get the candidate list from Registry before calling router
- List allRegistryApps = getRegistryClient().getAllApplications();
- LOG.info("{} app entries in FederationRegistry",
allStateStoreApps.size());
-
- // Get the list of known apps from Router
- Set routerApps = getRouterKnownApplications();
- LOG.info("{} known applications from Router", routerApps.size());
+ LOG.info("FederationStateStore has {} applications.",
applicationHomeSubClusters.size());
- // Clean up StateStore entries
- Set toDelete =
- Sets.difference(allStateStoreApps, routerApps);
-
Review Comment:
Step 1: Retrieve all applications stored in the StateStore, which represents
all applications submitted to the Router.
Step 2: Use the Router's REST API to fetch all running tasks. This API will
invoke applications from all active SubClusters.
Step 3: Compare the results of `Step1` and `Step2` to identify applications
that exist in `Step1` but not in `Step2`. Delete these applications.
> [GPG] YARN GPG mistakenly deleted applicationid
> ---
>
> Key: YARN-11387
> URL: https://issues.apache.org/jira/browse/YARN-11387
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.2.1, 3.4.0
>Reporter: zhangjunj
>Assignee: Shilun Fan
>Priority: Major
> Labels: federation, gpg, pull-request-available
> Attachments: YARN-11387-YARN-11387.v1.patch,
> yarn-gpg-mistakenly-deleted-applicationid.png
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In [YARN-7599|https://issues.apache.org/jira/browse/YARN-7599], the
> Federation can delete expired applicationid, but YARN GPG uses getRouter()
> method to obtain application information for multiple clusters. If there are
> too many applicationids that more than 200,000 , it will not be possible to
> pull all the applicationid information at one time, resulting in the
> possibility of accidental deletion. The following error is reported for spark
> component.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11387) [GPG] YARN GPG mistakenly deleted applicationid
[
https://issues.apache.org/jira/browse/YARN-11387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814843#comment-17814843
]
ASF GitHub Bot commented on YARN-11387:
---
slfan1989 commented on code in PR #6473:
URL: https://github.com/apache/hadoop/pull/6473#discussion_r1479929985
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/applicationcleaner/DefaultApplicationCleaner.java:
##
@@ -46,47 +49,37 @@ public void run() {
LOG.info("Application cleaner run at time {}", now);
FederationStateStoreFacade facade = getGPGContext().getStateStoreFacade();
+
try {
- // Get the candidate list from StateStore before calling router
- Set allStateStoreApps = new HashSet<>();
- List response =
+ // Step1. Get the candidate list from StateStore before calling router
+ List applicationHomeSubClusters =
facade.getApplicationsHomeSubCluster();
- for (ApplicationHomeSubCluster app : response) {
-allStateStoreApps.add(app.getApplicationId());
- }
- LOG.info("{} app entries in FederationStateStore",
allStateStoreApps.size());
-
- // Get the candidate list from Registry before calling router
- List allRegistryApps = getRegistryClient().getAllApplications();
- LOG.info("{} app entries in FederationRegistry",
allStateStoreApps.size());
-
- // Get the list of known apps from Router
- Set routerApps = getRouterKnownApplications();
- LOG.info("{} known applications from Router", routerApps.size());
+ LOG.info("FederationStateStore has {} applications.",
applicationHomeSubClusters.size());
- // Clean up StateStore entries
- Set toDelete =
- Sets.difference(allStateStoreApps, routerApps);
-
Review Comment:
Step 1: Retrieve all applications stored in the StateStore, which represents
all applications submitted to the Router.
Step 2: Use the Router's REST API to fetch all running tasks. This API will
invoke applications from all active SubClusters.
Step 3: Compare the results of `Step1` and `Step2` to identify applications
that exist in `Step1` but not in `Step2`. Delete these applications.
There is a potential issue with this approach. If a particular SubCluster is
undergoing maintenance, such as RM restart, Step 2 will not be able to fetch
the complete list of running applications. As a result, during the comparison
in Step 3, there is a risk of mistakenly deleting applications that are still
running.
> [GPG] YARN GPG mistakenly deleted applicationid
> ---
>
> Key: YARN-11387
> URL: https://issues.apache.org/jira/browse/YARN-11387
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.2.1, 3.4.0
>Reporter: zhangjunj
>Assignee: Shilun Fan
>Priority: Major
> Labels: federation, gpg, pull-request-available
> Attachments: YARN-11387-YARN-11387.v1.patch,
> yarn-gpg-mistakenly-deleted-applicationid.png
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In [YARN-7599|https://issues.apache.org/jira/browse/YARN-7599], the
> Federation can delete expired applicationid, but YARN GPG uses getRouter()
> method to obtain application information for multiple clusters. If there are
> too many applicationids that more than 200,000 , it will not be possible to
> pull all the applicationid information at one time, resulting in the
> possibility of accidental deletion. The following error is reported for spark
> component.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11387) [GPG] YARN GPG mistakenly deleted applicationid
[
https://issues.apache.org/jira/browse/YARN-11387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17814845#comment-17814845
]
ASF GitHub Bot commented on YARN-11387:
---
slfan1989 commented on code in PR #6473:
URL: https://github.com/apache/hadoop/pull/6473#discussion_r1479929985
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/applicationcleaner/DefaultApplicationCleaner.java:
##
@@ -46,47 +49,37 @@ public void run() {
LOG.info("Application cleaner run at time {}", now);
FederationStateStoreFacade facade = getGPGContext().getStateStoreFacade();
+
try {
- // Get the candidate list from StateStore before calling router
- Set allStateStoreApps = new HashSet<>();
- List response =
+ // Step1. Get the candidate list from StateStore before calling router
+ List applicationHomeSubClusters =
facade.getApplicationsHomeSubCluster();
- for (ApplicationHomeSubCluster app : response) {
-allStateStoreApps.add(app.getApplicationId());
- }
- LOG.info("{} app entries in FederationStateStore",
allStateStoreApps.size());
-
- // Get the candidate list from Registry before calling router
- List allRegistryApps = getRegistryClient().getAllApplications();
- LOG.info("{} app entries in FederationRegistry",
allStateStoreApps.size());
-
- // Get the list of known apps from Router
- Set routerApps = getRouterKnownApplications();
- LOG.info("{} known applications from Router", routerApps.size());
+ LOG.info("FederationStateStore has {} applications.",
applicationHomeSubClusters.size());
- // Clean up StateStore entries
- Set toDelete =
- Sets.difference(allStateStoreApps, routerApps);
-
Review Comment:
Step 1: Retrieve all applications stored in the StateStore, which represents
all applications submitted to the Router.
Step 2: Use the Router's REST API to fetch all running tasks. This API will
invoke applications from all active SubClusters.
Step 3: Compare the results of `Step1` and `Step2` to identify applications
that exist in `Step1` but not in `Step2`. Delete these applications.
There is a potential issue with this approach. If a particular SubCluster is
undergoing maintenance, such as RM restart, `Step2` will not be able to fetch
the complete list of running applications. As a result, during the comparison
in `Step3`, there is a risk of mistakenly deleting applications that are still
running.
> [GPG] YARN GPG mistakenly deleted applicationid
> ---
>
> Key: YARN-11387
> URL: https://issues.apache.org/jira/browse/YARN-11387
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.2.1, 3.4.0
>Reporter: zhangjunj
>Assignee: Shilun Fan
>Priority: Major
> Labels: federation, gpg, pull-request-available
> Attachments: YARN-11387-YARN-11387.v1.patch,
> yarn-gpg-mistakenly-deleted-applicationid.png
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In [YARN-7599|https://issues.apache.org/jira/browse/YARN-7599], the
> Federation can delete expired applicationid, but YARN GPG uses getRouter()
> method to obtain application information for multiple clusters. If there are
> too many applicationids that more than 200,000 , it will not be possible to
> pull all the applicationid information at one time, resulting in the
> possibility of accidental deletion. The following error is reported for spark
> component.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11387) [GPG] YARN GPG mistakenly deleted applicationid
[
https://issues.apache.org/jira/browse/YARN-11387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815020#comment-17815020
]
ASF GitHub Bot commented on YARN-11387:
---
slfan1989 commented on code in PR #6473:
URL: https://github.com/apache/hadoop/pull/6473#discussion_r1479929985
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/applicationcleaner/DefaultApplicationCleaner.java:
##
@@ -46,47 +49,37 @@ public void run() {
LOG.info("Application cleaner run at time {}", now);
FederationStateStoreFacade facade = getGPGContext().getStateStoreFacade();
+
try {
- // Get the candidate list from StateStore before calling router
- Set allStateStoreApps = new HashSet<>();
- List response =
+ // Step1. Get the candidate list from StateStore before calling router
+ List applicationHomeSubClusters =
facade.getApplicationsHomeSubCluster();
- for (ApplicationHomeSubCluster app : response) {
-allStateStoreApps.add(app.getApplicationId());
- }
- LOG.info("{} app entries in FederationStateStore",
allStateStoreApps.size());
-
- // Get the candidate list from Registry before calling router
- List allRegistryApps = getRegistryClient().getAllApplications();
- LOG.info("{} app entries in FederationRegistry",
allStateStoreApps.size());
-
- // Get the list of known apps from Router
- Set routerApps = getRouterKnownApplications();
- LOG.info("{} known applications from Router", routerApps.size());
+ LOG.info("FederationStateStore has {} applications.",
applicationHomeSubClusters.size());
- // Clean up StateStore entries
- Set toDelete =
- Sets.difference(allStateStoreApps, routerApps);
-
Review Comment:
Step 1: Retrieve all applications stored in the StateStore, which represents
all applications submitted to the Router.
Step 2: Use the Router's REST API to fetch all running tasks. This API will
invoke applications from all active SubClusters.
Step 3: Compare the results of `Step1` and `Step2` to identify applications
that exist in `Step1` but not in `Step2`. Delete these applications.
There is a potential issue with this approach. If a particular SubCluster is
undergoing maintenance, such as RM restart, `Step2` will not be able to fetch
the complete list of running applications. As a result, during the comparison
in `Step3`, there is a risk of mistakenly deleting applications that are still
running.
We have three SubClusters: `subClusterA`, `subClusterB`, and `subClusterC`,
with an equal allocation ratio of 1:1:1.
We submit six applications through `routerA`.
- `app1` and `app2` are allocated to `subClusterA`
- `app3` and `app4` to `subClusterB`
- `app5` and `app6` to `subClusterC`.
Among these, `app1`, `app3`, and `app5` have completed their execution,
and we expect to retain `app2`, `app4`, and `app6` in the StateStore.
> [GPG] YARN GPG mistakenly deleted applicationid
> ---
>
> Key: YARN-11387
> URL: https://issues.apache.org/jira/browse/YARN-11387
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.2.1, 3.4.0
>Reporter: zhangjunj
>Assignee: Shilun Fan
>Priority: Major
> Labels: federation, gpg, pull-request-available
> Attachments: YARN-11387-YARN-11387.v1.patch,
> yarn-gpg-mistakenly-deleted-applicationid.png
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In [YARN-7599|https://issues.apache.org/jira/browse/YARN-7599], the
> Federation can delete expired applicationid, but YARN GPG uses getRouter()
> method to obtain application information for multiple clusters. If there are
> too many applicationids that more than 200,000 , it will not be possible to
> pull all the applicationid information at one time, resulting in the
> possibility of accidental deletion. The following error is reported for spark
> component.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11387) [GPG] YARN GPG mistakenly deleted applicationid
[
https://issues.apache.org/jira/browse/YARN-11387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815021#comment-17815021
]
ASF GitHub Bot commented on YARN-11387:
---
slfan1989 commented on code in PR #6473:
URL: https://github.com/apache/hadoop/pull/6473#discussion_r1479929985
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/applicationcleaner/DefaultApplicationCleaner.java:
##
@@ -46,47 +49,37 @@ public void run() {
LOG.info("Application cleaner run at time {}", now);
FederationStateStoreFacade facade = getGPGContext().getStateStoreFacade();
+
try {
- // Get the candidate list from StateStore before calling router
- Set allStateStoreApps = new HashSet<>();
- List response =
+ // Step1. Get the candidate list from StateStore before calling router
+ List applicationHomeSubClusters =
facade.getApplicationsHomeSubCluster();
- for (ApplicationHomeSubCluster app : response) {
-allStateStoreApps.add(app.getApplicationId());
- }
- LOG.info("{} app entries in FederationStateStore",
allStateStoreApps.size());
-
- // Get the candidate list from Registry before calling router
- List allRegistryApps = getRegistryClient().getAllApplications();
- LOG.info("{} app entries in FederationRegistry",
allStateStoreApps.size());
-
- // Get the list of known apps from Router
- Set routerApps = getRouterKnownApplications();
- LOG.info("{} known applications from Router", routerApps.size());
+ LOG.info("FederationStateStore has {} applications.",
applicationHomeSubClusters.size());
- // Clean up StateStore entries
- Set toDelete =
- Sets.difference(allStateStoreApps, routerApps);
-
Review Comment:
Step 1: Retrieve all applications stored in the StateStore, which represents
all applications submitted to the Router.
Step 2: Use the Router's REST API to fetch all running tasks. This API will
invoke applications from all active SubClusters.
Step 3: Compare the results of `Step1` and `Step2` to identify applications
that exist in `Step1` but not in `Step2`. Delete these applications.
There is a potential issue with this approach. If a particular SubCluster is
undergoing maintenance, such as RM restart, `Step2` will not be able to fetch
the complete list of running applications. As a result, during the comparison
in `Step3`, there is a risk of mistakenly deleting applications that are still
running.
We have three SubClusters: `subClusterA`, `subClusterB`, and `subClusterC`,
with an equal allocation ratio of 1:1:1.
We submit six applications through `routerA`.
- `app1` and `app2` are allocated to `subClusterA`
- `app3` and `app4` to `subClusterB`
- `app5` and `app6` to `subClusterC`.
Among these, `app1`, `app3`, and `app5` have completed their execution, and
we expect to retain `app2`, `app4`, and `app6` in the StateStore.
> [GPG] YARN GPG mistakenly deleted applicationid
> ---
>
> Key: YARN-11387
> URL: https://issues.apache.org/jira/browse/YARN-11387
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.2.1, 3.4.0
>Reporter: zhangjunj
>Assignee: Shilun Fan
>Priority: Major
> Labels: federation, gpg, pull-request-available
> Attachments: YARN-11387-YARN-11387.v1.patch,
> yarn-gpg-mistakenly-deleted-applicationid.png
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In [YARN-7599|https://issues.apache.org/jira/browse/YARN-7599], the
> Federation can delete expired applicationid, but YARN GPG uses getRouter()
> method to obtain application information for multiple clusters. If there are
> too many applicationids that more than 200,000 , it will not be possible to
> pull all the applicationid information at one time, resulting in the
> possibility of accidental deletion. The following error is reported for spark
> component.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11387) [GPG] YARN GPG mistakenly deleted applicationid
[
https://issues.apache.org/jira/browse/YARN-11387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815023#comment-17815023
]
ASF GitHub Bot commented on YARN-11387:
---
slfan1989 commented on code in PR #6473:
URL: https://github.com/apache/hadoop/pull/6473#discussion_r1479929985
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/applicationcleaner/DefaultApplicationCleaner.java:
##
@@ -46,47 +49,37 @@ public void run() {
LOG.info("Application cleaner run at time {}", now);
FederationStateStoreFacade facade = getGPGContext().getStateStoreFacade();
+
try {
- // Get the candidate list from StateStore before calling router
- Set allStateStoreApps = new HashSet<>();
- List response =
+ // Step1. Get the candidate list from StateStore before calling router
+ List applicationHomeSubClusters =
facade.getApplicationsHomeSubCluster();
- for (ApplicationHomeSubCluster app : response) {
-allStateStoreApps.add(app.getApplicationId());
- }
- LOG.info("{} app entries in FederationStateStore",
allStateStoreApps.size());
-
- // Get the candidate list from Registry before calling router
- List allRegistryApps = getRegistryClient().getAllApplications();
- LOG.info("{} app entries in FederationRegistry",
allStateStoreApps.size());
-
- // Get the list of known apps from Router
- Set routerApps = getRouterKnownApplications();
- LOG.info("{} known applications from Router", routerApps.size());
+ LOG.info("FederationStateStore has {} applications.",
applicationHomeSubClusters.size());
- // Clean up StateStore entries
- Set toDelete =
- Sets.difference(allStateStoreApps, routerApps);
-
Review Comment:
Step 1: Retrieve all applications stored in the StateStore, which represents
all applications submitted to the Router.
Step 2: Use the Router's REST API to fetch all running tasks. This API will
invoke applications from all active SubClusters.
Step 3: Compare the results of `Step1` and `Step2` to identify applications
that exist in `Step1` but not in `Step2`. Delete these applications.
There is a potential issue with this approach. If a particular SubCluster is
undergoing maintenance, such as RM restart, `Step2` will not be able to fetch
the complete list of running applications. As a result, during the comparison
in `Step3`, there is a risk of mistakenly deleting applications that are still
running.
We have three SubClusters: `subClusterA`, `subClusterB`, and `subClusterC`,
with an equal allocation ratio of 1:1:1.
We submit six applications through `routerA`.
- `app1` and `app2` are allocated to `subClusterA`
- `app3` and `app4` to `subClusterB`
- `app5` and `app6` to `subClusterC`.
Among these, `app1`, `app3`, and `app5` have completed their execution, and
we expect to retain `app2`, `app4`, and `app6` in the StateStore.
In the normal scenario:
Comparing the steps mentioned above:
Step 1: We will retrieve six applications [app1, app2, app3, app4, app5,
app6] from the StateStore.
Step 2: We will fetch three applications [app2, app4, app6] from the
Router's REST interface.
Step 3: By comparing Step 1 and Step 2, we can identify that applications
[app1, app3, app5] should be deleted.
> [GPG] YARN GPG mistakenly deleted applicationid
> ---
>
> Key: YARN-11387
> URL: https://issues.apache.org/jira/browse/YARN-11387
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.2.1, 3.4.0
>Reporter: zhangjunj
>Assignee: Shilun Fan
>Priority: Major
> Labels: federation, gpg, pull-request-available
> Attachments: YARN-11387-YARN-11387.v1.patch,
> yarn-gpg-mistakenly-deleted-applicationid.png
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In [YARN-7599|https://issues.apache.org/jira/browse/YARN-7599], the
> Federation can delete expired applicationid, but YARN GPG uses getRouter()
> method to obtain application information for multiple clusters. If there are
> too many applicationids that more than 200,000 , it will not be possible to
> pull all the applicationid information at one time, resulting in the
> possibility of accidental deletion. The following error is reported for spark
> component.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11387) [GPG] YARN GPG mistakenly deleted applicationid
[
https://issues.apache.org/jira/browse/YARN-11387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815025#comment-17815025
]
ASF GitHub Bot commented on YARN-11387:
---
slfan1989 commented on code in PR #6473:
URL: https://github.com/apache/hadoop/pull/6473#discussion_r1479929985
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/applicationcleaner/DefaultApplicationCleaner.java:
##
@@ -46,47 +49,37 @@ public void run() {
LOG.info("Application cleaner run at time {}", now);
FederationStateStoreFacade facade = getGPGContext().getStateStoreFacade();
+
try {
- // Get the candidate list from StateStore before calling router
- Set allStateStoreApps = new HashSet<>();
- List response =
+ // Step1. Get the candidate list from StateStore before calling router
+ List applicationHomeSubClusters =
facade.getApplicationsHomeSubCluster();
- for (ApplicationHomeSubCluster app : response) {
-allStateStoreApps.add(app.getApplicationId());
- }
- LOG.info("{} app entries in FederationStateStore",
allStateStoreApps.size());
-
- // Get the candidate list from Registry before calling router
- List allRegistryApps = getRegistryClient().getAllApplications();
- LOG.info("{} app entries in FederationRegistry",
allStateStoreApps.size());
-
- // Get the list of known apps from Router
- Set routerApps = getRouterKnownApplications();
- LOG.info("{} known applications from Router", routerApps.size());
+ LOG.info("FederationStateStore has {} applications.",
applicationHomeSubClusters.size());
- // Clean up StateStore entries
- Set toDelete =
- Sets.difference(allStateStoreApps, routerApps);
-
Review Comment:
Step 1: Retrieve all applications stored in the StateStore, which represents
all applications submitted to the Router.
Step 2: Use the Router's REST API to fetch all running tasks. This API will
invoke applications from all active SubClusters.
Step 3: Compare the results of `Step1` and `Step2` to identify applications
that exist in `Step1` but not in `Step2`. Delete these applications.
There is a potential issue with this approach. If a particular SubCluster is
undergoing maintenance, such as RM restart, `Step2` will not be able to fetch
the complete list of running applications. As a result, during the comparison
in `Step3`, there is a risk of mistakenly deleting applications that are still
running.
We have three SubClusters: `subClusterA`, `subClusterB`, and `subClusterC`,
with an equal allocation ratio of 1:1:1.
We submit six applications through `routerA`.
- `app1` and `app2` are allocated to `subClusterA`
- `app3` and `app4` to `subClusterB`
- `app5` and `app6` to `subClusterC`.
Among these, `app1`, `app3`, and `app5` have completed their execution, and
we expect to retain `app2`, `app4`, and `app6` in the StateStore.
In the normal scenario:
Comparing the steps mentioned above:
Step 1: We will retrieve six applications [app1, app2, app3, app4, app5,
app6] from the StateStore.
Step 2: We will fetch three applications [app2, app4, app6] from the
Router's REST interface.
Step 3: By comparing Step 1 and Step 2, we can identify that applications
[app1, app3, app5] should be deleted.
In the exceptional scenario:
Comparing the steps mentioned above:
Step 1: We will retrieve six applications [app1, app2, app3, app4, app5,
app6] from the StateStore.
Step 2: We will fetch the list of running applications from the Router's
REST interface. However, due to maintenance in subClusterB and subClusterC, we
can only obtain the applications running in subClusterA [app2].
Step 3: By comparing Step 1 and Step 3, we can identify that applications
[app1, app3, app4, app5, app6] should be deleted.
In this case, we had an error deletion.
> [GPG] YARN GPG mistakenly deleted applicationid
> ---
>
> Key: YARN-11387
> URL: https://issues.apache.org/jira/browse/YARN-11387
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.2.1, 3.4.0
>Reporter: zhangjunj
>Assignee: Shilun Fan
>Priority: Major
> Labels: federation, gpg, pull-request-available
> Attachments: YARN-11387-YARN-11387.v1.patch,
> yarn-gpg-mistakenly-deleted-applicationid.png
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In [YARN-7599|https://issues.apache.org/jira/browse/YARN-7599], the
> Federation can delete expired applicationid, but YARN GPG uses getRouter()
> method to obtain appli
[jira] [Commented] (YARN-11387) [GPG] YARN GPG mistakenly deleted applicationid
[
https://issues.apache.org/jira/browse/YARN-11387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815230#comment-17815230
]
ASF GitHub Bot commented on YARN-11387:
---
hadoop-yetus commented on PR #6473:
URL: https://github.com/apache/hadoop/pull/6473#issuecomment-1931745994
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 4s | | trunk passed |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 13s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-globalpolicygenerator.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6473/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-globalpolicygenerator.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator:
The patch generated 5 new + 0 unchanged - 0 fixed = 5 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 0m 58s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-globalpolicygenerator.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6473/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-globalpolicygenerator.txt)
| hadoop-yarn-server-globalpolicygenerator in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 157m 21s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.globalpolicygenerator.applicationcleaner.TestDefaultApplicationCleaner
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6473/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6473 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a8d0e4abcb07 5.15.0-88-generic #98-Ubuntu SMP Mon Oc
[jira] [Commented] (YARN-11387) [GPG] YARN GPG mistakenly deleted applicationid
[
https://issues.apache.org/jira/browse/YARN-11387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815256#comment-17815256
]
ASF GitHub Bot commented on YARN-11387:
---
hadoop-yetus commented on PR #6473:
URL: https://github.com/apache/hadoop/pull/6473#issuecomment-1931867203
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 22s | | trunk passed |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 22s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 17s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 17s | | the patch passed |
| +1 :green_heart: | compile | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 12s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-globalpolicygenerator.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6473/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-globalpolicygenerator.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator:
The patch generated 10 new + 0 unchanged - 0 fixed = 10 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 59s | |
hadoop-yarn-server-globalpolicygenerator in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 129m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6473/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6473 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a244f44d355e 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fa157666cbc6cdc1894473c748d16682b8ee8e9b |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6473/4/testReport/ |
| Max. process+thread co
[jira] [Commented] (YARN-11649) YARN Federation getNewApplication returns different maxresourcecapability
[ https://issues.apache.org/jira/browse/YARN-11649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17815455#comment-17815455 ] ASF GitHub Bot commented on YARN-11649: --- jchanggg commented on PR #6481: URL: https://github.com/apache/hadoop/pull/6481#issuecomment-1933105891 Fixed all the lint issues. Could you help review this change @slfan1989 ? Thank you! > YARN Federation getNewApplication returns different maxresourcecapability > - > > Key: YARN-11649 > URL: https://issues.apache.org/jira/browse/YARN-11649 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jeffrey Chang >Assignee: Jeffrey Chang >Priority: Minor > Labels: pull-request-available > > When getNewApplication is called against YARN Router with Federation on, its > possible we get different maxResourceCapabilities in its response on > different calls. This is because getNewApplication is called against a random > cluster on each call, which may return different maxResourceCapability based > on the cluster that the call is executed on. ie in response in HTTP > [https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_New_Application_API] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11649) YARN Federation getNewApplication returns different maxresourcecapability
[ https://issues.apache.org/jira/browse/YARN-11649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17817230#comment-17817230 ] ASF GitHub Bot commented on YARN-11649: --- jchanggg commented on PR #6481: URL: https://github.com/apache/hadoop/pull/6481#issuecomment-1943201285 Gentle ping @slfan1989 , thank you! > YARN Federation getNewApplication returns different maxresourcecapability > - > > Key: YARN-11649 > URL: https://issues.apache.org/jira/browse/YARN-11649 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jeffrey Chang >Assignee: Jeffrey Chang >Priority: Minor > Labels: pull-request-available > > When getNewApplication is called against YARN Router with Federation on, its > possible we get different maxResourceCapabilities in its response on > different calls. This is because getNewApplication is called against a random > cluster on each call, which may return different maxResourceCapability based > on the cluster that the call is executed on. ie in response in HTTP > [https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_New_Application_API] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11649) YARN Federation getNewApplication returns different maxresourcecapability
[
https://issues.apache.org/jira/browse/YARN-11649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17818298#comment-17818298
]
ASF GitHub Bot commented on YARN-11649:
---
slfan1989 commented on code in PR #6481:
URL: https://github.com/apache/hadoop/pull/6481#discussion_r1493789622
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java:
##
@@ -2927,11 +2927,11 @@ public static boolean isAclEnabled(Configuration conf) {
// Web Proxy Configs
public static final String PROXY_PREFIX = "yarn.web-proxy.";
-
+
Review Comment:
These changes are unrelated to the content we modified. We shouldn't alter
this section.
> YARN Federation getNewApplication returns different maxresourcecapability
> -
>
> Key: YARN-11649
> URL: https://issues.apache.org/jira/browse/YARN-11649
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jeffrey Chang
>Assignee: Jeffrey Chang
>Priority: Minor
> Labels: pull-request-available
>
> When getNewApplication is called against YARN Router with Federation on, its
> possible we get different maxResourceCapabilities in its response on
> different calls. This is because getNewApplication is called against a random
> cluster on each call, which may return different maxResourceCapability based
> on the cluster that the call is executed on. ie in response in HTTP
> [https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_New_Application_API]
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11649) YARN Federation getNewApplication returns different maxresourcecapability
[ https://issues.apache.org/jira/browse/YARN-11649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17818299#comment-17818299 ] ASF GitHub Bot commented on YARN-11649: --- slfan1989 commented on PR #6481: URL: https://github.com/apache/hadoop/pull/6481#issuecomment-1951358892 > Gentle ping @slfan1989 , thank you! @jchanggg Sorry for the delayed response, I've been a bit busy lately. I will continue to follow up on this PR. > YARN Federation getNewApplication returns different maxresourcecapability > - > > Key: YARN-11649 > URL: https://issues.apache.org/jira/browse/YARN-11649 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jeffrey Chang >Assignee: Jeffrey Chang >Priority: Minor > Labels: pull-request-available > > When getNewApplication is called against YARN Router with Federation on, its > possible we get different maxResourceCapabilities in its response on > different calls. This is because getNewApplication is called against a random > cluster on each call, which may return different maxResourceCapability based > on the cluster that the call is executed on. ie in response in HTTP > [https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_New_Application_API] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11655) modify default value of Allocated GPUs and Reserved GPUs in yarn scheduler webui from -1 to 0
[ https://issues.apache.org/jira/browse/YARN-11655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11655: -- Labels: pull-request-available (was: ) > modify default value of Allocated GPUs and Reserved GPUs in yarn scheduler > webui from -1 to 0 > -- > > Key: YARN-11655 > URL: https://issues.apache.org/jira/browse/YARN-11655 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-common >Affects Versions: 3.3.3 >Reporter: wangzhongwei >Assignee: wangzhongwei >Priority: Minor > Labels: pull-request-available > Attachments: image-2024-02-20-15-15-34-996.png > > > in yarn scheduler webui,the value of Allocated GPUs and Reserved GPUs be set > to 0 by default may be better. when GPUs not used,these values should be 0 > !image-2024-02-20-15-15-34-996.png|width=486,height=235! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11655) modify default value of Allocated GPUs and Reserved GPUs in yarn scheduler webui from -1 to 0
[ https://issues.apache.org/jira/browse/YARN-11655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17818676#comment-17818676 ] ASF GitHub Bot commented on YARN-11655: --- granewang opened a new pull request, #6564: URL: https://github.com/apache/hadoop/pull/6564 in yarn scheduler webui,the value of Allocated GPUs and Reserved GPUs be set to 0 by default may be better. when GPUs not used,these values should be 0. > modify default value of Allocated GPUs and Reserved GPUs in yarn scheduler > webui from -1 to 0 > -- > > Key: YARN-11655 > URL: https://issues.apache.org/jira/browse/YARN-11655 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-common >Affects Versions: 3.3.3 >Reporter: wangzhongwei >Assignee: wangzhongwei >Priority: Minor > Attachments: image-2024-02-20-15-15-34-996.png > > > in yarn scheduler webui,the value of Allocated GPUs and Reserved GPUs be set > to 0 by default may be better. when GPUs not used,these values should be 0 > !image-2024-02-20-15-15-34-996.png|width=486,height=235! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11655) modify default value of Allocated GPUs and Reserved GPUs in yarn scheduler webui from -1 to 0
[
https://issues.apache.org/jira/browse/YARN-11655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17818717#comment-17818717
]
ASF GitHub Bot commented on YARN-11655:
---
hadoop-yetus commented on PR #6564:
URL: https://github.com/apache/hadoop/pull/6564#issuecomment-1953815967
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 12s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 35m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 52s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 132m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6564/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6564 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 26ac412a9bc6 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6dd3f57c6e90f0aeb92a042e7306db9345cdd524 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6564/1/testReport/ |
| Max. process+thread count | 699 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6564/1/console |
| versions | git=2.25.1 maven=3.
[jira] [Commented] (YARN-11656) RMStateStore event queue blocked
[
https://issues.apache.org/jira/browse/YARN-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819377#comment-17819377
]
ASF GitHub Bot commented on YARN-11656:
---
K0K0V0K opened a new pull request, #6569:
URL: https://github.com/apache/hadoop/pull/6569
### Description of PR
Created a MultiDispatcher class which implements the Dispatcher interface.
The dispatcher replace the AsyncDispatcher in the RMStateStore.
The Dispatcher creates a separate metric object called Event metrics for
"rm-state-store" where we can see
- how many unhandled events are currently present in the event queue for the
specific event type
- how many events were handled for the specific event type
- average execution time for the specific event
### How was this patch tested?
- Unit test
- Deployed cluster and load test with PI example job with the following
implementations
- - FileSystemRMStateStore
- - LeveldbRMStateStore
- - ZKRMStateStore
- Check RM stop - start can't corrupt RM
> RMStateStore event queue blocked
>
>
> Key: YARN-11656
> URL: https://issues.apache.org/jira/browse/YARN-11656
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.4.1
>Reporter: Bence Kosztolnik
>Assignee: Bence Kosztolnik
>Priority: Major
> Attachments: issue.png, log.png
>
>
> h2. Problem statement
>
> I observed Yarn cluster has pending and available resources as well, but the
> cluster utilization is usually around ~50%. The cluster had loaded with 200
> parallel PI example job (from hadoop-mapreduce-examples) with 20 map and 20
> reduce containers configured, on a 50 nodes cluster, where each node had 8
> cores, and a lot of memory (there was cpu bottleneck).
> Finally, I realized the RM had some IO bottleneck and needed 1~20 seconds to
> persist a RMStateStoreEvent (using FileSystemRMStateStore).
> To reduce the impact of the issue:
> - create a dispatcher where events can persist in parallel threads
> - create metric data for the RMStateStore event queue to be able easily to
> identify the problem if occurs on a cluster
> {panel:title=Issue visible on UI2}
> !issue.png|height=250!
> {panel}
> Also another way to identify the issue if we can see too much time is
> required to store info for app after reach new_saving state
> {panel:title=How issue can look like in log}
> !log.png|height=250!
> {panel}
> h2. Solution
> Created a *MultiDispatcher* class which implements the Dispatcher interface.
> The Dispatcher creates a separate metric object called _Event metrics for
> "rm-state-store"_ where we can see
> - how many unhandled events are currently present in the event queue for the
> specific event type
> - how many events were handled for the specific event type
> - average execution time for the specific event
> The dispatcher has the following configs ( the placeholder is for the
> dispatcher name, for example, rm-state-store )
> ||Config name||Description||Default value||
> |yarn.dispatcher.multi-thread.{}.*default-pool-size*|How many parallel
> threads should execute the parallel event execution| 4|
> |yarn.dispatcher.multi-thread.{}.*max-pool-size*|If the event queue is full
> the execution threads will scale up to this many|8|
> |yarn.dispatcher.multi-thread.{}.*keep-alive-seconds*|Execution threads will
> be destroyed after this many seconds|10|
> |yarn.dispatcher.multi-thread.{}.*queue-size*|Size of the eventqueue|1 000
> 000|
> |yarn.dispatcher.multi-thread.{}.*monitor-seconds*|The size of the event
> queue will be logged with this frequency (if not zero) |30|
> |yarn.dispatcher.multi-thread.{}.*graceful-stop-seconds*|After the stop
> signal the dispatcher will wait this many seconds to be able to process the
> incoming events before terminating them|60|
> {panel:title=Example output from RM JMX api}
> {noformat}
> ...
> {
> "name": "Hadoop:service=ResourceManager,name=Event metrics for
> rm-state-store",
> "modelerType": "Event metrics for rm-state-store",
> "tag.Context": "yarn",
> "tag.Hostname": CENSORED
> "RMStateStoreEventType#STORE_APP_ATTEMPT_Current": 51,
> "RMStateStoreEventType#STORE_APP_ATTEMPT_NumOps": 0,
> "RMStateStoreEventType#STORE_APP_ATTEMPT_AvgTime": 0.0,
> "RMStateStoreEventType#STORE_APP_Current": 124,
> "RMStateStoreEventType#STORE_APP_NumOps": 46,
> "RMStateStoreEventType#STORE_APP_AvgTime": 3318.25,
> "RMStateStoreEventType#UPDATE_APP_Current": 31,
> "RMStateStoreEventType#UPDATE_APP_NumOps": 16,
> "RMStateStoreEventType#UPDATE_APP_AvgTime": 2629.5,
> "RMStateStoreEventType#UPDATE_APP_ATTEMPT_Current": 31,
> "RMStateStoreEventType#UPDATE_APP_ATTEMPT_Nu
[jira] [Updated] (YARN-11656) RMStateStore event queue blocked
[
https://issues.apache.org/jira/browse/YARN-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YARN-11656:
--
Labels: pull-request-available (was: )
> RMStateStore event queue blocked
>
>
> Key: YARN-11656
> URL: https://issues.apache.org/jira/browse/YARN-11656
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.4.1
>Reporter: Bence Kosztolnik
>Assignee: Bence Kosztolnik
>Priority: Major
> Labels: pull-request-available
> Attachments: issue.png, log.png
>
>
> h2. Problem statement
>
> I observed Yarn cluster has pending and available resources as well, but the
> cluster utilization is usually around ~50%. The cluster had loaded with 200
> parallel PI example job (from hadoop-mapreduce-examples) with 20 map and 20
> reduce containers configured, on a 50 nodes cluster, where each node had 8
> cores, and a lot of memory (there was cpu bottleneck).
> Finally, I realized the RM had some IO bottleneck and needed 1~20 seconds to
> persist a RMStateStoreEvent (using FileSystemRMStateStore).
> To reduce the impact of the issue:
> - create a dispatcher where events can persist in parallel threads
> - create metric data for the RMStateStore event queue to be able easily to
> identify the problem if occurs on a cluster
> {panel:title=Issue visible on UI2}
> !issue.png|height=250!
> {panel}
> Also another way to identify the issue if we can see too much time is
> required to store info for app after reach new_saving state
> {panel:title=How issue can look like in log}
> !log.png|height=250!
> {panel}
> h2. Solution
> Created a *MultiDispatcher* class which implements the Dispatcher interface.
> The Dispatcher creates a separate metric object called _Event metrics for
> "rm-state-store"_ where we can see
> - how many unhandled events are currently present in the event queue for the
> specific event type
> - how many events were handled for the specific event type
> - average execution time for the specific event
> The dispatcher has the following configs ( the placeholder is for the
> dispatcher name, for example, rm-state-store )
> ||Config name||Description||Default value||
> |yarn.dispatcher.multi-thread.{}.*default-pool-size*|How many parallel
> threads should execute the parallel event execution| 4|
> |yarn.dispatcher.multi-thread.{}.*max-pool-size*|If the event queue is full
> the execution threads will scale up to this many|8|
> |yarn.dispatcher.multi-thread.{}.*keep-alive-seconds*|Execution threads will
> be destroyed after this many seconds|10|
> |yarn.dispatcher.multi-thread.{}.*queue-size*|Size of the eventqueue|1 000
> 000|
> |yarn.dispatcher.multi-thread.{}.*monitor-seconds*|The size of the event
> queue will be logged with this frequency (if not zero) |30|
> |yarn.dispatcher.multi-thread.{}.*graceful-stop-seconds*|After the stop
> signal the dispatcher will wait this many seconds to be able to process the
> incoming events before terminating them|60|
> {panel:title=Example output from RM JMX api}
> {noformat}
> ...
> {
> "name": "Hadoop:service=ResourceManager,name=Event metrics for
> rm-state-store",
> "modelerType": "Event metrics for rm-state-store",
> "tag.Context": "yarn",
> "tag.Hostname": CENSORED
> "RMStateStoreEventType#STORE_APP_ATTEMPT_Current": 51,
> "RMStateStoreEventType#STORE_APP_ATTEMPT_NumOps": 0,
> "RMStateStoreEventType#STORE_APP_ATTEMPT_AvgTime": 0.0,
> "RMStateStoreEventType#STORE_APP_Current": 124,
> "RMStateStoreEventType#STORE_APP_NumOps": 46,
> "RMStateStoreEventType#STORE_APP_AvgTime": 3318.25,
> "RMStateStoreEventType#UPDATE_APP_Current": 31,
> "RMStateStoreEventType#UPDATE_APP_NumOps": 16,
> "RMStateStoreEventType#UPDATE_APP_AvgTime": 2629.5,
> "RMStateStoreEventType#UPDATE_APP_ATTEMPT_Current": 31,
> "RMStateStoreEventType#UPDATE_APP_ATTEMPT_NumOps": 12,
> "RMStateStoreEventType#UPDATE_APP_ATTEMPT_AvgTime": 2048.5,
> "RMStateStoreEventType#REMOVE_APP_Current": 12,
> "RMStateStoreEventType#REMOVE_APP_NumOps": 3,
> "RMStateStoreEventType#REMOVE_APP_AvgTime": 1378.0,
> "RMStateStoreEventType#REMOVE_APP_ATTEMPT_Current": 0,
> "RMStateStoreEventType#REMOVE_APP_ATTEMPT_NumOps": 0,
> "RMStateStoreEventType#REMOVE_APP_ATTEMPT_AvgTime": 0.0,
> "RMStateStoreEventType#FENCED_Current": 0,
> "RMStateStoreEventType#FENCED_NumOps": 0,
> "RMStateStoreEventType#FENCED_AvgTime": 0.0,
> "RMStateStoreEventType#STORE_MASTERKEY_Current": 0,
> "RMStateStoreEventType#STORE_MASTERKEY_NumOps": 0,
> "RMStateStoreEventType#STORE_MASTERKEY_AvgTime": 0.0,
> "RMStateStoreEventType#REMOVE_MASTERKEY_Current
[jira] [Commented] (YARN-11656) RMStateStore event queue blocked
[
https://issues.apache.org/jira/browse/YARN-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819432#comment-17819432
]
ASF GitHub Bot commented on YARN-11656:
---
hadoop-yetus commented on PR #6569:
URL: https://github.com/apache/hadoop/pull/6569#issuecomment-1958479227
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 58s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 49s | | trunk passed |
| +1 :green_heart: | compile | 7m 32s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 6m 59s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 11s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 1s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 6m 51s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 6m 51s | | the patch passed |
| +1 :green_heart: | compile | 6m 57s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 6m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 47s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 12 new + 71 unchanged
- 1 fixed = 83 total (was 72) |
| +1 :green_heart: | mvnsite | 1m 57s | | the patch passed |
| -1 :x: | javadoc | 0m 58s |
[/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/1/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdkUbuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04.txt)
| hadoop-yarn-common in the patch failed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04. |
| -1 :x: | javadoc | 0m 57s |
[/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/1/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common-jdkPrivateBuild-1.8.0_392-8u392-ga-1~20.04-b08.txt)
| hadoop-yarn-common in the patch failed with JDK Private
Build-1.8.0_392-8u392-ga-1~20.04-b08. |
| +1 :green_heart: | spotbugs | 4m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 35s | | hadoop-yarn-common in the patch
passed. |
| -1 :x: | unit | 106m 58s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| -1 :x: | asflicense | 0m 58s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/1/artifact/out/results-asflicense.txt)
| The patch generated 2 ASF License warning
[jira] [Commented] (YARN-11656) RMStateStore event queue blocked
[
https://issues.apache.org/jira/browse/YARN-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819654#comment-17819654
]
ASF GitHub Bot commented on YARN-11656:
---
slfan1989 commented on code in PR #6569:
URL: https://github.com/apache/hadoop/pull/6569#discussion_r1499283399
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/Event.java:
##
@@ -32,4 +32,14 @@ public interface Event> {
TYPE getType();
long getTimestamp();
String toString();
+
+ /**
+ * In case of parallel execution of events in the same dispatcher,
+ * the result of this method will be used as semaphore.
+ * If method returns null, then a default semaphore will be used.
+ * @return the semaphore
+ */
+ default String getLockKey() {
+return null;
Review Comment:
Is returning null expected here?
> RMStateStore event queue blocked
>
>
> Key: YARN-11656
> URL: https://issues.apache.org/jira/browse/YARN-11656
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.4.1
>Reporter: Bence Kosztolnik
>Assignee: Bence Kosztolnik
>Priority: Major
> Labels: pull-request-available
> Attachments: issue.png, log.png
>
>
> h2. Problem statement
>
> I observed Yarn cluster has pending and available resources as well, but the
> cluster utilization is usually around ~50%. The cluster had loaded with 200
> parallel PI example job (from hadoop-mapreduce-examples) with 20 map and 20
> reduce containers configured, on a 50 nodes cluster, where each node had 8
> cores, and a lot of memory (there was cpu bottleneck).
> Finally, I realized the RM had some IO bottleneck and needed 1~20 seconds to
> persist a RMStateStoreEvent (using FileSystemRMStateStore).
> To reduce the impact of the issue:
> - create a dispatcher where events can persist in parallel threads
> - create metric data for the RMStateStore event queue to be able easily to
> identify the problem if occurs on a cluster
> {panel:title=Issue visible on UI2}
> !issue.png|height=250!
> {panel}
> Also another way to identify the issue if we can see too much time is
> required to store info for app after reach new_saving state
> {panel:title=How issue can look like in log}
> !log.png|height=250!
> {panel}
> h2. Solution
> Created a *MultiDispatcher* class which implements the Dispatcher interface.
> The Dispatcher creates a separate metric object called _Event metrics for
> "rm-state-store"_ where we can see
> - how many unhandled events are currently present in the event queue for the
> specific event type
> - how many events were handled for the specific event type
> - average execution time for the specific event
> The dispatcher has the following configs ( the placeholder is for the
> dispatcher name, for example, rm-state-store )
> ||Config name||Description||Default value||
> |yarn.dispatcher.multi-thread.{}.*default-pool-size*|How many parallel
> threads should execute the parallel event execution| 4|
> |yarn.dispatcher.multi-thread.{}.*max-pool-size*|If the event queue is full
> the execution threads will scale up to this many|8|
> |yarn.dispatcher.multi-thread.{}.*keep-alive-seconds*|Execution threads will
> be destroyed after this many seconds|10|
> |yarn.dispatcher.multi-thread.{}.*queue-size*|Size of the eventqueue|1 000
> 000|
> |yarn.dispatcher.multi-thread.{}.*monitor-seconds*|The size of the event
> queue will be logged with this frequency (if not zero) |30|
> |yarn.dispatcher.multi-thread.{}.*graceful-stop-seconds*|After the stop
> signal the dispatcher will wait this many seconds to be able to process the
> incoming events before terminating them|60|
> {panel:title=Example output from RM JMX api}
> {noformat}
> ...
> {
> "name": "Hadoop:service=ResourceManager,name=Event metrics for
> rm-state-store",
> "modelerType": "Event metrics for rm-state-store",
> "tag.Context": "yarn",
> "tag.Hostname": CENSORED
> "RMStateStoreEventType#STORE_APP_ATTEMPT_Current": 51,
> "RMStateStoreEventType#STORE_APP_ATTEMPT_NumOps": 0,
> "RMStateStoreEventType#STORE_APP_ATTEMPT_AvgTime": 0.0,
> "RMStateStoreEventType#STORE_APP_Current": 124,
> "RMStateStoreEventType#STORE_APP_NumOps": 46,
> "RMStateStoreEventType#STORE_APP_AvgTime": 3318.25,
> "RMStateStoreEventType#UPDATE_APP_Current": 31,
> "RMStateStoreEventType#UPDATE_APP_NumOps": 16,
> "RMStateStoreEventType#UPDATE_APP_AvgTime": 2629.5,
> "RMStateStoreEventType#UPDATE_APP_ATTEMPT_Current": 31,
> "RMStateStoreEventType#UPDATE_APP_ATTEMPT_NumOps": 12,
> "RMStateStoreEventType#UPDATE_APP_ATTEMPT_AvgTime": 2048.5,
> "RMStateStoreEventType#REM
[jira] [Commented] (YARN-11656) RMStateStore event queue blocked
[
https://issues.apache.org/jira/browse/YARN-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819659#comment-17819659
]
ASF GitHub Bot commented on YARN-11656:
---
K0K0V0K commented on code in PR #6569:
URL: https://github.com/apache/hadoop/pull/6569#discussion_r1499304409
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/event/Event.java:
##
@@ -32,4 +32,14 @@ public interface Event> {
TYPE getType();
long getTimestamp();
String toString();
+
+ /**
+ * In case of parallel execution of events in the same dispatcher,
+ * the result of this method will be used as semaphore.
+ * If method returns null, then a default semaphore will be used.
+ * @return the semaphore
+ */
+ default String getLockKey() {
+return null;
Review Comment:
Hi @slfan1989 !
Thanks for the review.
Yes, that is expected.
If we dont specify lockKey for an event we should return with null, so these
events will be executed in sequential not parallel. The method is used in the
MultiDispatcherLocks.
> RMStateStore event queue blocked
>
>
> Key: YARN-11656
> URL: https://issues.apache.org/jira/browse/YARN-11656
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.4.1
>Reporter: Bence Kosztolnik
>Assignee: Bence Kosztolnik
>Priority: Major
> Labels: pull-request-available
> Attachments: issue.png, log.png
>
>
> h2. Problem statement
>
> I observed Yarn cluster has pending and available resources as well, but the
> cluster utilization is usually around ~50%. The cluster had loaded with 200
> parallel PI example job (from hadoop-mapreduce-examples) with 20 map and 20
> reduce containers configured, on a 50 nodes cluster, where each node had 8
> cores, and a lot of memory (there was cpu bottleneck).
> Finally, I realized the RM had some IO bottleneck and needed 1~20 seconds to
> persist a RMStateStoreEvent (using FileSystemRMStateStore).
> To reduce the impact of the issue:
> - create a dispatcher where events can persist in parallel threads
> - create metric data for the RMStateStore event queue to be able easily to
> identify the problem if occurs on a cluster
> {panel:title=Issue visible on UI2}
> !issue.png|height=250!
> {panel}
> Also another way to identify the issue if we can see too much time is
> required to store info for app after reach new_saving state
> {panel:title=How issue can look like in log}
> !log.png|height=250!
> {panel}
> h2. Solution
> Created a *MultiDispatcher* class which implements the Dispatcher interface.
> The Dispatcher creates a separate metric object called _Event metrics for
> "rm-state-store"_ where we can see
> - how many unhandled events are currently present in the event queue for the
> specific event type
> - how many events were handled for the specific event type
> - average execution time for the specific event
> The dispatcher has the following configs ( the placeholder is for the
> dispatcher name, for example, rm-state-store )
> ||Config name||Description||Default value||
> |yarn.dispatcher.multi-thread.{}.*default-pool-size*|How many parallel
> threads should execute the parallel event execution| 4|
> |yarn.dispatcher.multi-thread.{}.*max-pool-size*|If the event queue is full
> the execution threads will scale up to this many|8|
> |yarn.dispatcher.multi-thread.{}.*keep-alive-seconds*|Execution threads will
> be destroyed after this many seconds|10|
> |yarn.dispatcher.multi-thread.{}.*queue-size*|Size of the eventqueue|1 000
> 000|
> |yarn.dispatcher.multi-thread.{}.*monitor-seconds*|The size of the event
> queue will be logged with this frequency (if not zero) |30|
> |yarn.dispatcher.multi-thread.{}.*graceful-stop-seconds*|After the stop
> signal the dispatcher will wait this many seconds to be able to process the
> incoming events before terminating them|60|
> {panel:title=Example output from RM JMX api}
> {noformat}
> ...
> {
> "name": "Hadoop:service=ResourceManager,name=Event metrics for
> rm-state-store",
> "modelerType": "Event metrics for rm-state-store",
> "tag.Context": "yarn",
> "tag.Hostname": CENSORED
> "RMStateStoreEventType#STORE_APP_ATTEMPT_Current": 51,
> "RMStateStoreEventType#STORE_APP_ATTEMPT_NumOps": 0,
> "RMStateStoreEventType#STORE_APP_ATTEMPT_AvgTime": 0.0,
> "RMStateStoreEventType#STORE_APP_Current": 124,
> "RMStateStoreEventType#STORE_APP_NumOps": 46,
> "RMStateStoreEventType#STORE_APP_AvgTime": 3318.25,
> "RMStateStoreEventType#UPDATE_APP_Current": 31,
> "RMStateStoreEventType#UPDATE_APP_NumOps": 16,
> "RMStateStoreEventType#UPDATE_APP_AvgTime": 2629.5,
> "RMStateStore
[jira] [Updated] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YARN-11657:
--
Labels: pull-request-available (was: )
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819719#comment-17819719
]
ASF GitHub Bot commented on YARN-11657:
---
steveloughran opened a new pull request, #6575:
URL: https://github.com/apache/hadoop/pull/6575
### How was this patch tested?
hadoop release support verifies that protobuf-2.5 isn't published as a
dependency of
any artifact.
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819727#comment-17819727
]
ASF GitHub Bot commented on YARN-11657:
---
steveloughran commented on PR #6575:
URL: https://github.com/apache/hadoop/pull/6575#issuecomment-1959869568
doing a build shows it is still coming in to timeline service. that's
because it is talking to hbase...to fix that we'd need to move to an hbase
client which was more current.
```
find . -name \*.jar -print | grep proto
./share/hadoop/yarn/csi/lib/grpc-protobuf-1.53.0.jar
./share/hadoop/yarn/csi/lib/grpc-protobuf-lite-1.53.0.jar
./share/hadoop/yarn/csi/lib/proto-google-common-protos-2.9.0.jar
./share/hadoop/yarn/timelineservice/lib/protobuf-java-2.5.0.jar
./share/hadoop/yarn/timelineservice/lib/hbase-protocol-1.7.1.jar
./share/hadoop/common/lib/hadoop-shaded-protobuf_3_21-1.2.0.jar
./share/hadoop/hdfs/lib/hadoop-shaded-protobuf_3_21-1.2.0.jar
```
Issue: should we explicitly list it in LICENSE-binary?
Given we still ship it, I believe we should
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819729#comment-17819729
]
ASF GitHub Bot commented on YARN-11657:
---
steveloughran commented on PR #6575:
URL: https://github.com/apache/hadoop/pull/6575#issuecomment-1959871746
comes in because of
```
[INFO] +- org.apache.hbase:hbase-common:jar:1.7.1:compile
[INFO] | +- org.apache.hbase:hbase-protocol:jar:1.7.1:compile
[INFO] | +- org.apache.hbase:hbase-annotations:jar:1.7.1:compile
[INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | +- org.apache.hbase.thirdparty:hbase-shaded-gson:jar:3.0.0:compile
[INFO] | +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] | +- org.apache.htrace:htrace-core:jar:3.1.0-incubating:compile
[INFO] | \-
com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
```
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819792#comment-17819792
]
ASF GitHub Bot commented on YARN-11657:
---
hadoop-yetus commented on PR #6575:
URL: https://github.com/apache/hadoop/pull/6575#issuecomment-1960180302
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.4 Compile Tests _ |
| +0 :ok: | mvndep | 4m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 41m 49s | | branch-3.4 passed |
| +1 :green_heart: | compile | 7m 30s | | branch-3.4 passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 6m 54s | | branch-3.4 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | mvnsite | 2m 31s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 2m 27s | | branch-3.4 passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 16s | | branch-3.4 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 103m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 35s | | the patch passed |
| +1 :green_heart: | compile | 7m 8s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 7m 8s | | the patch passed |
| +1 :green_heart: | compile | 7m 31s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 7m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 2m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 7s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 4s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 42m 51s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 7s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 4m 46s | |
hadoop-yarn-server-applicationhistoryservice in the patch passed. |
| +1 :green_heart: | unit | 24m 40s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 55s | | The patch does not
generate ASF License warnings. |
| | | 200m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6575/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6575 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux 3bae1998fb94 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / 18223bfbcb4cec81dff8f82ea4ec8ec50cf2b9de |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6575/1/testReport/ |
| Max. process+thread count | 2773 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoo
[jira] [Updated] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YARN-11626:
--
Labels: pull-request-available (was: )
> Optimization of the safeDelete operation in ZKRMStateStore
> --
>
> Key: YARN-11626
> URL: https://issues.apache.org/jira/browse/YARN-11626
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Minor
> Labels: pull-request-available
>
> h1. Description
> * We can be observed that removing app info started at 06:17:20, but the
> NoNodeException was received at 06:17:35.
> * During the 15s interval, Curator was retrying the metadata operation. Due
> to the non-idempotent nature of the Zookeeper deletion operation, in one of
> the retry attempts, the metadata operation was successful but no response was
> received. In the next retry it resulted in a NoNodeException, triggering the
> STATE_STORE_FENCED event and ultimately causing the current ResourceManager
> to switch to standby .
> {code:java}
> 2023-10-28 06:17:20,359 INFO recovery.RMStateStore
> (RMStateStore.java:transition(333)) - Removing info for app:
> application_1697410508608_140368
> 2023-10-28 06:17:20,359 INFO resourcemanager.RMAppManager
> (RMAppManager.java:checkAppNumCompletedLimit(303)) - Application should be
> expired, max number of completed apps kept in memory met:
> maxCompletedAppsInMemory = 1000, removing app
> application_1697410508608_140368 from memory:
> 2023-10-28 06:17:35,665 ERROR recovery.RMStateStore
> (RMStateStore.java:transition(337)) - Error removing app:
> application_1697410508608_140368
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> 2023-10-28 06:17:35,666 INFO recovery.RMStateStore
> (RMStateStore.java:handleStoreEvent(1147)) - RMStateStore state change from
> ACTIVE to FENCED
> 2023-10-28 06:17:35,666 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> STATE_STORE_FENCED, caused by
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> 2023-10-28 06:17:35,666 INFO resourcemanager.ResourceManager
> (ResourceManager.java:transitionToStandby(1309)) - Transitioning to standby
> state
> {code}
> h1. Solution
> The NoNodeException clearly indicates that the Znode no longer exists, so we
> can safely ignore this exception to avoid triggering a larger impact on the
> cluster caused by ResourceManager failover.
> h1. Other
> We also need to discuss and optimize the same issues in safeCreate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819896#comment-17819896
]
ASF GitHub Bot commented on YARN-11626:
---
XbaoWu opened a new pull request, #6577:
URL: https://github.com/apache/hadoop/pull/6577
### Description of PR
For more information about this PR, please refer to the following issue：
[YARN-11626](https://issues.apache.org/jira/browse/YARN-11626) Optimization
of the safeDelete operation in ZKRMStateStore
### How was this patch tested?
add TestCheckRemoveZKNodeRMStateStore.testSafeDeleteZKNode()
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [x] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Optimization of the safeDelete operation in ZKRMStateStore
> --
>
> Key: YARN-11626
> URL: https://issues.apache.org/jira/browse/YARN-11626
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Minor
>
> h1. Description
> * We can be observed that removing app info started at 06:17:20, but the
> NoNodeException was received at 06:17:35.
> * During the 15s interval, Curator was retrying the metadata operation. Due
> to the non-idempotent nature of the Zookeeper deletion operation, in one of
> the retry attempts, the metadata operation was successful but no response was
> received. In the next retry it resulted in a NoNodeException, triggering the
> STATE_STORE_FENCED event and ultimately causing the current ResourceManager
> to switch to standby .
> {code:java}
> 2023-10-28 06:17:20,359 INFO recovery.RMStateStore
> (RMStateStore.java:transition(333)) - Removing info for app:
> application_1697410508608_140368
> 2023-10-28 06:17:20,359 INFO resourcemanager.RMAppManager
> (RMAppManager.java:checkAppNumCompletedLimit(303)) - Application should be
> expired, max number of completed apps kept in memory met:
> maxCompletedAppsInMemory = 1000, removing app
> application_1697410508608_140368 from memory:
> 2023-10-28 06:17:35,665 ERROR recovery.RMStateStore
> (RMStateStore.java:transition(337)) - Error removing app:
> application_1697410508608_140368
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> 2023-10-28 06:17:35,666 INFO recovery.RMStateStore
> (RMStateStore.java:handleStoreEvent(1147)) - RMStateStore state change from
> ACTIVE to FENCED
> 2023-10-28 06:17:35,666 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> STATE_STORE_FENCED, caused by
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> 2023-10-28 06:17:35,666 INFO resourcemanager.ResourceManager
> (ResourceManager.java:transitionToStandby(1309)) - Transitioning to standby
> state
> {code}
> h1. Solution
> The NoNodeException clearly indicates that the Znode no longer exists, so we
> can safely ignore this exception to avoid triggering a larger impact on the
> cluster caused by ResourceManager failover.
> h1. Other
> We also need to discuss and optimize the same issues in safeCreate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819900#comment-17819900
]
ASF GitHub Bot commented on YARN-11626:
---
hadoop-yetus commented on PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#issuecomment-1960704380
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 4m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| -1 :x: | mvninstall | 0m 20s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/1/artifact/out/branch-mvninstall-root.txt)
| root in branch-3.3 failed. |
| -1 :x: | compile | 0m 22s |
[/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/1/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in branch-3.3 failed. |
| -0 :warning: | checkstyle | 0m 20s |
[/buildtool-branch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/1/artifact/out/buildtool-branch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| The patch fails to run checkstyle in hadoop-yarn-server-resourcemanager |
| -1 :x: | mvnsite | 0m 22s |
[/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/1/artifact/out/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in branch-3.3 failed. |
| -1 :x: | javadoc | 0m 20s |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/1/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in branch-3.3 failed. |
| -1 :x: | spotbugs | 0m 20s |
[/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/1/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in branch-3.3 failed. |
| +1 :green_heart: | shadedclient | 2m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 21s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/1/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 0m 21s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | javac | 0m 21s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/1/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/buildtool-patch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_h
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17819926#comment-17819926
]
ASF GitHub Bot commented on YARN-11657:
---
ayushtkn commented on code in PR #6575:
URL: https://github.com/apache/hadoop/pull/6575#discussion_r1500242344
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml:
##
@@ -72,6 +72,7 @@
com.google.protobuf
protobuf-java
+ ${transient.protobuf2.scope}
Review Comment:
Any pointers why are we putting scope in every declarition rather than just
putting it up in the parent pom?
https://github.com/apache/hadoop/commit/c373d3fa39013e8a8f6a6122c3ca230b4aa10abe
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820020#comment-17820020
]
ASF GitHub Bot commented on YARN-11657:
---
hadoop-yetus commented on PR #6575:
URL: https://github.com/apache/hadoop/pull/6575#issuecomment-1961147527
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.4 Compile Tests _ |
| +0 :ok: | mvndep | 4m 22s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 43m 1s | | branch-3.4 passed |
| +1 :green_heart: | compile | 17m 43s | | branch-3.4 passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 16m 59s | | branch-3.4 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | mvnsite | 22m 44s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 8m 38s | | branch-3.4 passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 39s | | branch-3.4 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 50m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 31m 55s | | the patch passed |
| +1 :green_heart: | compile | 16m 49s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 16m 49s | | the patch passed |
| +1 :green_heart: | compile | 16m 4s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 16m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 15m 35s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 29s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 7m 44s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 49m 49s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 799m 43s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6575/2/artifact/out/patch-unit-root.txt)
| root in the patch failed. |
| +1 :green_heart: | asflicense | 1m 37s | | The patch does not
generate ASF License warnings. |
| | | 1094m 1s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.yarn.server.timeline.webapp.TestTimelineWebServices |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6575/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6575 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs |
| uname | Linux 35c13fbe0370 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / a726eb4e4079467e91b46316dcfd56bb220589d8 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-ope
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820112#comment-17820112
]
ASF GitHub Bot commented on YARN-11657:
---
steveloughran commented on PR #6575:
URL: https://github.com/apache/hadoop/pull/6575#issuecomment-1961575509
I don't want to backport this to 3.4.0; i'm going to merge into branch-3.4
and trunk but not 3.4.0 as the lib is still needed in one place and there's a
risk of breaking things.
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820113#comment-17820113
]
ASF GitHub Bot commented on YARN-11657:
---
steveloughran commented on code in PR #6575:
URL: https://github.com/apache/hadoop/pull/6575#discussion_r1500859099
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice/pom.xml:
##
@@ -72,6 +72,7 @@
com.google.protobuf
protobuf-java
+ ${transient.protobuf2.scope}
Review Comment:
1. The dependent poms (hadoop-common...) aren't publishing it any more, just
using it for compilation.
2. I don't trust the base declaration to do the right thing.
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820135#comment-17820135
]
ASF GitHub Bot commented on YARN-11657:
---
steveloughran commented on PR #6575:
URL: https://github.com/apache/hadoop/pull/6575#issuecomment-1961692880
pushed up a change which updates the building doc
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820137#comment-17820137
]
ASF GitHub Bot commented on YARN-11657:
---
steveloughran merged PR #6575:
URL: https://github.com/apache/hadoop/pull/6575
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820138#comment-17820138
]
ASF GitHub Bot commented on YARN-11657:
---
steveloughran opened a new pull request, #6580:
URL: https://github.com/apache/hadoop/pull/6580
The import of protobuf-java-2.5 in the hadoop-yarn-api module is downgraded
from "compile" to "provided"
This removes it from share/hadoop/yarn/lib/protobuf-java-2.5.0.jar
It is still found under
share/hadoop/yarn/timelineservice/lib/protobuf-java-2.5.0.jar
### For code changes:
- [X] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820139#comment-17820139
]
ASF GitHub Bot commented on YARN-11657:
---
steveloughran commented on PR #6580:
URL: https://github.com/apache/hadoop/pull/6580#issuecomment-1961701109
This is #6575 for trunk
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820147#comment-17820147
]
ASF GitHub Bot commented on YARN-11657:
---
steveloughran opened a new pull request, #6581:
URL: https://github.com/apache/hadoop/pull/6581
PR #6575 for branch-3.3
The import of protobuf-java-2.5 in the hadoop-yarn-api module is downgraded
from "compile" to "provided"
This removes it from share/hadoop/yarn/lib/protobuf-java-2.5.0.jar
It is still found under
share/hadoop/yarn/timelineservice/lib/protobuf-java-2.5.0.jar
Note: this PR also updates LICENSE-binary reference to protobuf 3.6 to
3.21.12
to match what in hadoop-thirdparty; the merge created conflict at the text
level, and rather than fix it by picking 3.6 or 3.7 I fixed it to the current
value
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [X] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820261#comment-17820261
]
ASF GitHub Bot commented on YARN-11626:
---
hiwangzhihui commented on code in PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#discussion_r1501328992
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java:
##
@@ -938,28 +938,36 @@ private void handleApplicationAttemptStateOp(
attemptStateDataPB.getProto().toByteArray();
LOG.debug("{} info for attempt: {} at: {}", operation, appAttemptId, path);
-switch (operation) {
-case UPDATE:
- if (exists(path)) {
-zkManager.safeSetData(path, attemptStateData, -1, zkAcl,
-fencingNodePath);
- } else {
-zkManager.safeCreate(path, attemptStateData, zkAcl,
-CreateMode.PERSISTENT, zkAcl, fencingNodePath);
-LOG.debug("Path {} for {} didn't exist. Created a new znode to update"
-+ " the application attempt state.", path, appAttemptId);
+try {
+ switch (operation) {
+ case UPDATE:
+if (exists(path)) {
+ zkManager.safeSetData(path, attemptStateData, -1, zkAcl,
+ fencingNodePath);
+} else {
+ zkManager.safeCreate(path, attemptStateData, zkAcl,
+ CreateMode.PERSISTENT, zkAcl, fencingNodePath);
+ LOG.debug("Path {} for {} didn't exist. Created a new znode to
update"
+ + " the application attempt state.", path, appAttemptId);
+}
+break;
+ case STORE:
+zkManager.safeCreate(path, attemptStateData, zkAcl,
CreateMode.PERSISTENT,
+zkAcl, fencingNodePath);
+break;
+ case REMOVE:
+zkManager.safeDelete(path, zkAcl, fencingNodePath);
+break;
+ default:
+break;
+ }
+} catch (KeeperException.NoNodeException nne){
+ if(!exists(path)){
Review Comment:
Here, You just need to focus on the NoNodeException of REMOVE action
> Optimization of the safeDelete operation in ZKRMStateStore
> --
>
> Key: YARN-11626
> URL: https://issues.apache.org/jira/browse/YARN-11626
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Minor
> Labels: pull-request-available
>
> h1. Description
> * We can be observed that removing app info started at 06:17:20, but the
> NoNodeException was received at 06:17:35.
> * During the 15s interval, Curator was retrying the metadata operation. Due
> to the non-idempotent nature of the Zookeeper deletion operation, in one of
> the retry attempts, the metadata operation was successful but no response was
> received. In the next retry it resulted in a NoNodeException, triggering the
> STATE_STORE_FENCED event and ultimately causing the current ResourceManager
> to switch to standby .
> {code:java}
> 2023-10-28 06:17:20,359 INFO recovery.RMStateStore
> (RMStateStore.java:transition(333)) - Removing info for app:
> application_1697410508608_140368
> 2023-10-28 06:17:20,359 INFO resourcemanager.RMAppManager
> (RMAppManager.java:checkAppNumCompletedLimit(303)) - Application should be
> expired, max number of completed apps kept in memory met:
> maxCompletedAppsInMemory = 1000, removing app
> application_1697410508608_140368 from memory:
> 2023-10-28 06:17:35,665 ERROR recovery.RMStateStore
> (RMStateStore.java:transition(337)) - Error removing app:
> application_1697410508608_140368
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> 2023-10-28 06:17:35,666 INFO recovery.RMStateStore
> (RMStateStore.java:handleStoreEvent(1147)) - RMStateStore state change from
> ACTIVE to FENCED
> 2023-10-28 06:17:35,666 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> STATE_STORE_FENCED, caused by
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> 2023-10-28 06:17:35,666 INFO resourcemanager.ResourceManager
> (ResourceManager.java:transitionToStandby(1309)) - Transitioning to standby
> state
> {code}
> h1. Solution
> The NoNodeException clearly indicates that the Znode no longer exists, so we
> can safely ignore this exception to avoid triggering a larger impact on the
> cluster caused by ResourceManager failover.
> h1. Other
> We also need to discuss and optimize the same issues in safeCreate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820278#comment-17820278
]
ASF GitHub Bot commented on YARN-11626:
---
XbaoWu commented on code in PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#discussion_r1501352665
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/recovery/ZKRMStateStore.java:
##
@@ -938,28 +938,36 @@ private void handleApplicationAttemptStateOp(
attemptStateDataPB.getProto().toByteArray();
LOG.debug("{} info for attempt: {} at: {}", operation, appAttemptId, path);
-switch (operation) {
-case UPDATE:
- if (exists(path)) {
-zkManager.safeSetData(path, attemptStateData, -1, zkAcl,
-fencingNodePath);
- } else {
-zkManager.safeCreate(path, attemptStateData, zkAcl,
-CreateMode.PERSISTENT, zkAcl, fencingNodePath);
-LOG.debug("Path {} for {} didn't exist. Created a new znode to update"
-+ " the application attempt state.", path, appAttemptId);
+try {
+ switch (operation) {
+ case UPDATE:
+if (exists(path)) {
+ zkManager.safeSetData(path, attemptStateData, -1, zkAcl,
+ fencingNodePath);
+} else {
+ zkManager.safeCreate(path, attemptStateData, zkAcl,
+ CreateMode.PERSISTENT, zkAcl, fencingNodePath);
+ LOG.debug("Path {} for {} didn't exist. Created a new znode to
update"
+ + " the application attempt state.", path, appAttemptId);
+}
+break;
+ case STORE:
+zkManager.safeCreate(path, attemptStateData, zkAcl,
CreateMode.PERSISTENT,
+zkAcl, fencingNodePath);
+break;
+ case REMOVE:
+zkManager.safeDelete(path, zkAcl, fencingNodePath);
+break;
+ default:
+break;
+ }
+} catch (KeeperException.NoNodeException nne){
+ if(!exists(path)){
Review Comment:
Thank you. I've modified and tested this.
> Optimization of the safeDelete operation in ZKRMStateStore
> --
>
> Key: YARN-11626
> URL: https://issues.apache.org/jira/browse/YARN-11626
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Minor
> Labels: pull-request-available
>
> h1. Description
> * We can be observed that removing app info started at 06:17:20, but the
> NoNodeException was received at 06:17:35.
> * During the 15s interval, Curator was retrying the metadata operation. Due
> to the non-idempotent nature of the Zookeeper deletion operation, in one of
> the retry attempts, the metadata operation was successful but no response was
> received. In the next retry it resulted in a NoNodeException, triggering the
> STATE_STORE_FENCED event and ultimately causing the current ResourceManager
> to switch to standby .
> {code:java}
> 2023-10-28 06:17:20,359 INFO recovery.RMStateStore
> (RMStateStore.java:transition(333)) - Removing info for app:
> application_1697410508608_140368
> 2023-10-28 06:17:20,359 INFO resourcemanager.RMAppManager
> (RMAppManager.java:checkAppNumCompletedLimit(303)) - Application should be
> expired, max number of completed apps kept in memory met:
> maxCompletedAppsInMemory = 1000, removing app
> application_1697410508608_140368 from memory:
> 2023-10-28 06:17:35,665 ERROR recovery.RMStateStore
> (RMStateStore.java:transition(337)) - Error removing app:
> application_1697410508608_140368
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> 2023-10-28 06:17:35,666 INFO recovery.RMStateStore
> (RMStateStore.java:handleStoreEvent(1147)) - RMStateStore state change from
> ACTIVE to FENCED
> 2023-10-28 06:17:35,666 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> STATE_STORE_FENCED, caused by
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> 2023-10-28 06:17:35,666 INFO resourcemanager.ResourceManager
> (ResourceManager.java:transitionToStandby(1309)) - Transitioning to standby
> state
> {code}
> h1. Solution
> The NoNodeException clearly indicates that the Znode no longer exists, so we
> can safely ignore this exception to avoid triggering a larger impact on the
> cluster caused by ResourceManager failover.
> h1. Other
> We also need to discuss and optimize the same issues in safeCreate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820282#comment-17820282
]
ASF GitHub Bot commented on YARN-11657:
---
hadoop-yetus commented on PR #6581:
URL: https://github.com/apache/hadoop/pull/6581#issuecomment-1962275833
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 13m 39s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 24m 11s | | branch-3.3 passed |
| +1 :green_heart: | compile | 11m 34s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 18m 11s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 4m 42s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 26m 13s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 24m 39s | | the patch passed |
| +1 :green_heart: | compile | 11m 20s | | the patch passed |
| +1 :green_heart: | javac | 11m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 14m 26s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 4m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 607m 54s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6581/1/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 1m 7s | | The patch does not
generate ASF License warnings. |
| | | 784m 59s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestReadStripedFileWithDNFailure |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.TestDFSStripedInputStreamWithRandomECPolicy |
| | hadoop.hdfs.TestErasureCodingPoliciesWithRandomECPolicy |
| | hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6581/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6581 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvnsite unit shadedclient xmllint shellcheck shelldocs |
| uname | Linux 1044b159e79c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / f92c528071ae93f216bee610eff68ec33401d8b9 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6581/1/testReport/ |
| Max. process+thread count | 4223 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-applicationhistoryservice
. U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6581/1/console |
| versions | git=2.
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820283#comment-17820283
]
ASF GitHub Bot commented on YARN-11657:
---
hadoop-yetus commented on PR #6575:
URL: https://github.com/apache/hadoop/pull/6575#issuecomment-1962277549
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.4 Compile Tests _ |
| +0 :ok: | mvndep | 4m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 44s | | branch-3.4 passed |
| +1 :green_heart: | compile | 10m 32s | | branch-3.4 passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 37s | | branch-3.4 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | mvnsite | 14m 38s | | branch-3.4 passed |
| +1 :green_heart: | javadoc | 5m 11s | | branch-3.4 passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 29s | | branch-3.4 passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 34m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 20m 21s | | the patch passed |
| +1 :green_heart: | compile | 9m 38s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 9m 38s | | the patch passed |
| +1 :green_heart: | compile | 8m 19s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 8m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 10m 13s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 5m 7s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 25s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 33m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 628m 0s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6575/3/artifact/out/patch-unit-root.txt)
| root in the patch failed. |
| +1 :green_heart: | asflicense | 1m 5s | | The patch does not
generate ASF License warnings. |
| | | 817m 32s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.TestReconstructStripedFileWithValidator |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| |
hadoop.yarn.server.timelineservice.security.TestTimelineAuthFilterForV2 |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6575/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6575 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvnsite unit shadedclient xmllint shellcheck shelldocs |
| uname | Linux 05a9ef72435e 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.4 / 24633bcb891f85b9425a9f64e2dc27d12886f3dc |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubu
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820302#comment-17820302
]
ASF GitHub Bot commented on YARN-11626:
---
hadoop-yetus commented on PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#issuecomment-1962315822
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 58m 19s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 58s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 44s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 1m 3s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 50s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 59s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 35m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 0s | | the patch passed |
| +1 :green_heart: | compile | 0m 49s | | the patch passed |
| +1 :green_heart: | javac | 0m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 32s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 8 new + 5 unchanged - 0 fixed = 13 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 54s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 98m 43s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 244m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6577 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux f4c8ec9c4bde 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / baf56d632a0ee56dd9096b55e675129d71c4d5b2 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/2/testReport/ |
| Max. process+thread count | 949 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/2/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Optimization of the safeDelete operation in ZKRMStateStore
> --
>
> Key: YARN-11626
> URL: https://issues.apache.org/jira/browse/YARN-11626
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820309#comment-17820309
]
ASF GitHub Bot commented on YARN-11657:
---
hadoop-yetus commented on PR #6580:
URL: https://github.com/apache/hadoop/pull/6580#issuecomment-1962329670
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 47s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 17s | | trunk passed |
| +1 :green_heart: | compile | 19m 32s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 18m 43s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | mvnsite | 23m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 52s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 8m 12s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 49m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 31m 22s | | the patch passed |
| +1 :green_heart: | compile | 19m 59s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 19m 59s | | the patch passed |
| +1 :green_heart: | compile | 19m 17s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 19m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 16m 45s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 40s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 8m 9s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 50m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 756m 4s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6580/1/artifact/out/patch-unit-root.txt)
| root in the patch failed. |
| +1 :green_heart: | asflicense | 1m 23s | | The patch does not
generate ASF License warnings. |
| | | 1072m 39s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.hdfs.server.namenode.TestFSImage |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6580/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6580 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvnsite unit shadedclient xmllint shellcheck shelldocs |
| uname | Linux 0184bc318077 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 16ad42c172ef16f6b1aa9bc7c67926b6c29e7a2d |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-am
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820311#comment-17820311
]
ASF GitHub Bot commented on YARN-11657:
---
hadoop-yetus commented on PR #6580:
URL: https://github.com/apache/hadoop/pull/6580#issuecomment-1962332574
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 42s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 4s | | trunk passed |
| +1 :green_heart: | compile | 19m 17s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 18m 22s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | mvnsite | 23m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 47s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 8m 9s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 49m 38s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 30m 56s | | the patch passed |
| +1 :green_heart: | compile | 19m 46s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 19m 46s | | the patch passed |
| +1 :green_heart: | compile | 19m 0s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 19m 0s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 17m 8s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 40s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 8m 12s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 50m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 769m 45s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6580/2/artifact/out/patch-unit-root.txt)
| root in the patch failed. |
| +1 :green_heart: | asflicense | 1m 37s | | The patch does not
generate ASF License warnings. |
| | | 1073m 44s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.TestErasureCodingPolicyWithSnapshotWithRandomECPolicy |
| | hadoop.hdfs.protocol.TestBlockListAsLongs |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.server.namenode.TestCacheDirectivesWithViewDFS |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
| | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6580/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6580 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvnsite unit shadedclient xmllint shellcheck shelldocs |
| uname | Linux fa7dfa5768b6 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 16ad42c172ef16f6
[jira] [Commented] (YARN-11656) RMStateStore event queue blocked
[
https://issues.apache.org/jira/browse/YARN-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820312#comment-17820312
]
ASF GitHub Bot commented on YARN-11656:
---
hadoop-yetus commented on PR #6569:
URL: https://github.com/apache/hadoop/pull/6569#issuecomment-1962333672
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 28m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 55s | | trunk passed |
| +1 :green_heart: | compile | 3m 46s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 3m 31s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 18s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 21s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| +1 :green_heart: | compile | 3m 26s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 3m 26s | | the patch passed |
| +1 :green_heart: | compile | 3m 23s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 3m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 57s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 12 new + 71 unchanged
- 1 fixed = 83 total (was 72) |
| +1 :green_heart: | mvnsite | 1m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 8s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 16s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 4m 38s | | hadoop-yarn-common in the patch
passed. |
| -1 :x: | unit | 67m 54s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/4/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 197m 20s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairSchedulerQueueACLs |
| |
hadoop.yarn.server.resourcemanager.TestApplicationMasterServiceCapacity |
| | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueParsing
|
| |
hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesConfigurationMutation
|
| |
hadoop.yarn.server.resourcemanager.scheduler.policy.TestFairOrderingPolicy |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestSchedulingRequestContainerAllocation
|
| |
hadoop.yarn.server.res
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820510#comment-17820510
]
ASF GitHub Bot commented on YARN-11626:
---
hadoop-yetus commented on PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#issuecomment-1963034340
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 4m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 59m 51s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 36s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 29s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 41s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 36s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 17s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 23m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 8 new + 5 unchanged - 0 fixed = 13 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 25s | | the patch passed |
| -1 :x: | shadedclient | 26m 52s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 30m 17s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +0 :ok: | asflicense | 0m 25s | | ASF License check generated no
output? |
| | | 155m 30s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA |
| | hadoop.yarn.server.resourcemanager.TestRMHA |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6577 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 9d0efe4fa13d 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / c4094ebd48ed921b6b60587827eea68aa28bfc29 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/3/testReport/ |
| Max. process+thread count | 706 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/3/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820568#comment-17820568
]
ASF GitHub Bot commented on YARN-11626:
---
hadoop-yetus commented on PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#issuecomment-1963416005
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 40s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 36s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 28s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 38s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 32s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 14s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 21m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 80m 45s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/4/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 177m 30s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.recovery.TestCheckRemoveZKNodeRMStateStore |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6577 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 62f075e61cd1 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 1d50dd5546633b2c9991ce3f847086397d95acc5 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/4/testReport/ |
| Max. process+thread count | 934 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/4/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820660#comment-17820660
]
ASF GitHub Bot commented on YARN-11626:
---
hadoop-yetus commented on PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#issuecomment-1963800849
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 7s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 34s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 28s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 41s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 35s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 16s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 21m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 16s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 32s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 80m 42s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/5/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 26s | | The patch does not
generate ASF License warnings. |
| | | 181m 56s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.recovery.TestCheckRemoveZKNodeRMStateStore |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6577 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux bf6af790a69f 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / d27aa4a3fe290663dca1036447602bfcafe06d12 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/5/testReport/ |
| Max. process+thread count | 911 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/5/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820751#comment-17820751
]
ASF GitHub Bot commented on YARN-11626:
---
hadoop-yetus commented on PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#issuecomment-1964483913
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 44s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 35s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 27s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 39s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 32s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 15s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 21m 36s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 80m 24s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/6/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 26s | | The patch does not
generate ASF License warnings. |
| | | 166m 21s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.recovery.TestCheckRemoveZKNodeRMStateStore |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6577 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 863b191a11d6 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 847f51117995f954fdc734fd295ff2fc7153ca52 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/6/testReport/ |
| Max. process+thread count | 937 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/6/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820954#comment-17820954
]
ASF GitHub Bot commented on YARN-11626:
---
hadoop-yetus commented on PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#issuecomment-1965764333
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 24s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| -1 :x: | mvninstall | 1m 4s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/7/artifact/out/branch-mvninstall-root.txt)
| root in branch-3.3 failed. |
| -1 :x: | compile | 0m 22s |
[/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/7/artifact/out/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in branch-3.3 failed. |
| -0 :warning: | checkstyle | 0m 20s |
[/buildtool-branch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/7/artifact/out/buildtool-branch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| The patch fails to run checkstyle in hadoop-yarn-server-resourcemanager |
| -1 :x: | mvnsite | 4m 7s |
[/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/7/artifact/out/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in branch-3.3 failed. |
| +1 :green_heart: | javadoc | 0m 30s | | branch-3.3 passed |
| -1 :x: | spotbugs | 0m 19s |
[/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/7/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in branch-3.3 failed. |
| -1 :x: | shadedclient | 6m 8s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 22s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/7/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 0m 22s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/7/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | javac | 0m 22s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/7/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/buildtool-patch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/7/artifact/out/buildtool-patch-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| The patch fails to run checkstyle in hadoop-yarn-server-resourcemanager |
| -1 :x: | mvnsit
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17821171#comment-17821171
]
ASF GitHub Bot commented on YARN-11626:
---
hadoop-yetus commented on PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#issuecomment-1966285056
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 12s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 37s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 29s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 40s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 33s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 16s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 21m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 19s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/8/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 30s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 11s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 80m 40s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/8/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 166m 5s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.recovery.TestCheckRemoveZKNodeRMStateStore |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6577 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 80a05f23c3c6 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / dd5be7c235ea01f2ddeb70ff8bc27687f2f31625 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/8/testReport/ |
| Max. process+thread count | 945 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/8/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0
[jira] [Commented] (YARN-11656) RMStateStore event queue blocked
[
https://issues.apache.org/jira/browse/YARN-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17822185#comment-17822185
]
ASF GitHub Bot commented on YARN-11656:
---
hadoop-yetus commented on PR #6569:
URL: https://github.com/apache/hadoop/pull/6569#issuecomment-1971420241
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 14s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 58s | | trunk passed |
| +1 :green_heart: | compile | 9m 3s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 13s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 2m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 4s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 59s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 8m 10s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 8m 10s | | the patch passed |
| +1 :green_heart: | compile | 9m 47s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 9m 47s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 56s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 14 new + 71 unchanged
- 1 fixed = 85 total (was 72) |
| +1 :green_heart: | mvnsite | 2m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 50s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 49s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 10s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/5/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 40m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 43s | | hadoop-yarn-common in the patch
passed. |
| -1 :x: | unit | 127m 31s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/5/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not
generate ASF License warnings. |
| | | 334m 6s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| | new
org.apache.hadoop.yarn.event.multidispatcher.MultiDispatcherExecutor(Logger,
MultiDispatcherConfig, String) invokes
org.apache.hadoop.yarn.event.multidispatcher.MultiDispatcherExecutor$MultiDispatcherExecutorThread.start()
At
MultiDispatcherExecutor.java:org.apache.hadoop.yarn.even
[jira] [Commented] (YARN-11656) RMStateStore event queue blocked
[
https://issues.apache.org/jira/browse/YARN-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17822190#comment-17822190
]
ASF GitHub Bot commented on YARN-11656:
---
hadoop-yetus commented on PR #6569:
URL: https://github.com/apache/hadoop/pull/6569#issuecomment-1971438007
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 8s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 37m 48s | | trunk passed |
| +1 :green_heart: | compile | 10m 14s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 13s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 2m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 6s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 58s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 9m 8s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 9m 8s | | the patch passed |
| +1 :green_heart: | compile | 8m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 8m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 58s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 14 new + 71 unchanged
- 1 fixed = 85 total (was 72) |
| +1 :green_heart: | mvnsite | 1m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 57s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 55s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 9s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/6/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 40m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 44s | | hadoop-yarn-common in the patch
passed. |
| -1 :x: | unit | 130m 39s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/6/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not
generate ASF License warnings. |
| | | 339m 47s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| | new
org.apache.hadoop.yarn.event.multidispatcher.MultiDispatcherExecutor(Logger,
MultiDispatcherConfig, String) invokes
org.apache.hadoop.yarn.event.multidispatcher.MultiDispatcherExecutor$MultiDispatcherExecutorThread.start()
At
MultiDispatcherExecutor.java:org.apache.hadoop.yarn.even
[jira] [Commented] (YARN-11656) RMStateStore event queue blocked
[
https://issues.apache.org/jira/browse/YARN-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17822283#comment-17822283
]
ASF GitHub Bot commented on YARN-11656:
---
hadoop-yetus commented on PR #6569:
URL: https://github.com/apache/hadoop/pull/6569#issuecomment-1971796048
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 34s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 48s | | trunk passed |
| +1 :green_heart: | compile | 9m 17s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 7m 55s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 0s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 2s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 54s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 8m 1s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 8m 1s | | the patch passed |
| +1 :green_heart: | compile | 7m 45s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 7m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 51s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 14 new + 71 unchanged
- 1 fixed = 85 total (was 72) |
| +1 :green_heart: | mvnsite | 1m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 49s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 44s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 1s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/7/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 41m 12s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 45s | | hadoop-yarn-common in the patch
passed. |
| -1 :x: | unit | 136m 21s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/7/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not
generate ASF License warnings. |
| | | 338m 42s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| | new
org.apache.hadoop.yarn.event.multidispatcher.MultiDispatcherExecutor(Logger,
MultiDispatcherConfig, String) invokes
org.apache.hadoop.yarn.event.multidispatcher.MultiDispatcherExecutor$MultiDispatcherExecutorThread.start()
At
MultiDispatcherExecutor.java:org.apache.hadoop.yarn.even
[jira] [Commented] (YARN-11656) RMStateStore event queue blocked
[
https://issues.apache.org/jira/browse/YARN-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17822289#comment-17822289
]
ASF GitHub Bot commented on YARN-11656:
---
hadoop-yetus commented on PR #6569:
URL: https://github.com/apache/hadoop/pull/6569#issuecomment-1971843834
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 34s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 38m 55s | | trunk passed |
| +1 :green_heart: | compile | 8m 22s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 7m 59s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 2m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 3s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 53s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 7m 58s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 7m 58s | | the patch passed |
| +1 :green_heart: | compile | 7m 45s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 7m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 51s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/8/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 14 new + 71 unchanged
- 1 fixed = 85 total (was 72) |
| +1 :green_heart: | mvnsite | 1m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 52s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 47s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 1s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/8/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 40m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 40s | | hadoop-yarn-common in the patch
passed. |
| -1 :x: | unit | 118m 47s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/8/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 1m 27s | | The patch does not
generate ASF License warnings. |
| | | 325m 15s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| | Bad attempt to compute absolute value of signed 32-bit hashcode in
org.apache.hadoop.yarn.event.multidispatcher.MultiDispatcherExecutor.execute(Event,
Runnable) At MultiDispatcherExecutor.java:value of signed 32-bit hashcode in
org.apache.hadoop.yarn.event.multidispatcher.MultiDispatcher
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17822941#comment-17822941
]
ASF GitHub Bot commented on YARN-11626:
---
hadoop-yetus commented on PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#issuecomment-1975171815
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 3m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 58s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 35s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 28s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 41s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 32s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 20s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 21m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 17s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/9/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 5 unchanged - 0 fixed = 7 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 81m 3s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 170m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6577 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux ebf28a942b8c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 97f7996875d5ca1f8ba369dcb54ed86e2429ad4c |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/9/testReport/ |
| Max. process+thread count | 926 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/9/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Optimization of the safeDelete operation in ZKRMStateStore
> --
>
> Key: YARN-11626
> URL: https://issues.apache.org/jira/browse/YARN-11626
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823536#comment-17823536
]
ASF GitHub Bot commented on YARN-11657:
---
steveloughran commented on PR #6580:
URL: https://github.com/apache/hadoop/pull/6580#issuecomment-1978423882
Can I hold back on this and go for the "cut hbase 1 timeline service" I
think stuff may fall out from that automatically. Then we can revisit this to
see what to do
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823557#comment-17823557
]
ASF GitHub Bot commented on YARN-11657:
---
steveloughran merged PR #6580:
URL: https://github.com/apache/hadoop/pull/6580
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11657) Remove protobuf-2.5 as dependency of hadoop-yarn-api
[
https://issues.apache.org/jira/browse/YARN-11657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17823559#comment-17823559
]
ASF GitHub Bot commented on YARN-11657:
---
steveloughran merged PR #6581:
URL: https://github.com/apache/hadoop/pull/6581
> Remove protobuf-2.5 as dependency of hadoop-yarn-api
>
>
> Key: YARN-11657
> URL: https://issues.apache.org/jira/browse/YARN-11657
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: api
>Affects Versions: 3.4.0
>Reporter: Steve Loughran
>Assignee: Steve Loughran
>Priority: Major
> Labels: pull-request-available
>
> hadoop-yarn-api is still exporting protobuf-2.5.
> if we can cut this, we should
> {code}
> [echo] [INFO] +-
> org.apache.hadoop:hadoop-yarn-server-common:jar:3.4.0:compile
> [echo] [INFO] | +- org.apache.hadoop:hadoop-yarn-api:jar:3.4.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-guava:jar:1.2.0:compile - omitted
> for duplicate)
> [echo] [INFO] | | +- (javax.xml.bind:jaxb-api:jar:2.2.11:compile -
> omitted for duplicate)
> [echo] [INFO] | | +-
> (org.apache.hadoop:hadoop-annotations:jar:3.4.0:compile - omitted for
> duplicate)
> [echo] [INFO] | | +-
> com.google.protobuf:protobuf-java:jar:2.5.0:compile
> [echo] [INFO] | | +-
> (org.apache.hadoop.thirdparty:hadoop-shaded-protobuf_3_21:jar:1.2.0:compile -
> omitted for duplicate)
> [echo] [INFO] | | \-
> (com.fasterxml.jackson.core:jackson-annotations:jar:2.12.7:compile - omitted
> for duplicate)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824049#comment-17824049
]
ASF GitHub Bot commented on YARN-11626:
---
hadoop-yetus commented on PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#issuecomment-1981087718
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 36s | | branch-3.3 passed |
| +1 :green_heart: | compile | 0m 30s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 0m 25s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 0m 36s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 0m 34s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 1m 22s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 25m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 21s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/10/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed |
| +1 :green_heart: | spotbugs | 1m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 30s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 80m 12s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 23s | | The patch does not
generate ASF License warnings. |
| | | 177m 27s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6577 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 460b1e90ebc9 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / 0451694b8d0066271a5a500fdfcb9b2e318cc8bb |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~18.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/10/testReport/ |
| Max. process+thread count | 950 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6577/10/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Optimization of the safeDelete operation in ZKRMStateStore
> --
>
> Key: YARN-11626
> URL: https://issues.apache.org/jira/browse/YARN-11626
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanage
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824330#comment-17824330
]
ASF GitHub Bot commented on YARN-11626:
---
Hexiaoqiao commented on PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#issuecomment-1983061802
Hi @XbaoWu Please submit PR to trunk first, if approved and committed to
trunk, then backport to other active branches if necessary.
> Optimization of the safeDelete operation in ZKRMStateStore
> --
>
> Key: YARN-11626
> URL: https://issues.apache.org/jira/browse/YARN-11626
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Minor
> Labels: pull-request-available
>
> h1. Description
> * We can be observed that removing app info started at 06:17:20, but the
> NoNodeException was received at 06:17:35.
> * During the 15s interval, Curator was retrying the metadata operation. Due
> to the non-idempotent nature of the Zookeeper deletion operation, in one of
> the retry attempts, the metadata operation was successful but no response was
> received. In the next retry it resulted in a NoNodeException, triggering the
> STATE_STORE_FENCED event and ultimately causing the current ResourceManager
> to switch to standby .
> {code:java}
> 2023-10-28 06:17:20,359 INFO recovery.RMStateStore
> (RMStateStore.java:transition(333)) - Removing info for app:
> application_1697410508608_140368
> 2023-10-28 06:17:20,359 INFO resourcemanager.RMAppManager
> (RMAppManager.java:checkAppNumCompletedLimit(303)) - Application should be
> expired, max number of completed apps kept in memory met:
> maxCompletedAppsInMemory = 1000, removing app
> application_1697410508608_140368 from memory:
> 2023-10-28 06:17:35,665 ERROR recovery.RMStateStore
> (RMStateStore.java:transition(337)) - Error removing app:
> application_1697410508608_140368
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> 2023-10-28 06:17:35,666 INFO recovery.RMStateStore
> (RMStateStore.java:handleStoreEvent(1147)) - RMStateStore state change from
> ACTIVE to FENCED
> 2023-10-28 06:17:35,666 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> STATE_STORE_FENCED, caused by
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> 2023-10-28 06:17:35,666 INFO resourcemanager.ResourceManager
> (ResourceManager.java:transitionToStandby(1309)) - Transitioning to standby
> state
> {code}
> h1. Solution
> The NoNodeException clearly indicates that the Znode no longer exists, so we
> can safely ignore this exception to avoid triggering a larger impact on the
> cluster caused by ResourceManager failover.
> h1. Other
> We also need to discuss and optimize the same issues in safeCreate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824360#comment-17824360
]
ASF GitHub Bot commented on YARN-11626:
---
XbaoWu opened a new pull request, #6616:
URL: https://github.com/apache/hadoop/pull/6616
### Description of PR
For more information about this PR, please refer to the following issue：
[YARN-11626](https://issues.apache.org/jira/browse/YARN-11626) Optimization
of the safeDelete operation in ZKRMStateStore
The NoNodeException clearly indicates that the Znode no longer exists, so if
we check again and find that the node does not actually exist, we can safely
ignore this exception to avoid triggering a larger impact on the cluster caused
by ResourceManager failover.
### How was this patch tested?
add TestCheckRemoveZKNodeRMStateStore.testSafeDeleteZKNode()
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [x] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [x] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Optimization of the safeDelete operation in ZKRMStateStore
> --
>
> Key: YARN-11626
> URL: https://issues.apache.org/jira/browse/YARN-11626
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Minor
> Labels: pull-request-available
>
> h1. Description
> * We can be observed that removing app info started at 06:17:20, but the
> NoNodeException was received at 06:17:35.
> * During the 15s interval, Curator was retrying the metadata operation. Due
> to the non-idempotent nature of the Zookeeper deletion operation, in one of
> the retry attempts, the metadata operation was successful but no response was
> received. In the next retry it resulted in a NoNodeException, triggering the
> STATE_STORE_FENCED event and ultimately causing the current ResourceManager
> to switch to standby .
> {code:java}
> 2023-10-28 06:17:20,359 INFO recovery.RMStateStore
> (RMStateStore.java:transition(333)) - Removing info for app:
> application_1697410508608_140368
> 2023-10-28 06:17:20,359 INFO resourcemanager.RMAppManager
> (RMAppManager.java:checkAppNumCompletedLimit(303)) - Application should be
> expired, max number of completed apps kept in memory met:
> maxCompletedAppsInMemory = 1000, removing app
> application_1697410508608_140368 from memory:
> 2023-10-28 06:17:35,665 ERROR recovery.RMStateStore
> (RMStateStore.java:transition(337)) - Error removing app:
> application_1697410508608_140368
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> 2023-10-28 06:17:35,666 INFO recovery.RMStateStore
> (RMStateStore.java:handleStoreEvent(1147)) - RMStateStore state change from
> ACTIVE to FENCED
> 2023-10-28 06:17:35,666 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> STATE_STORE_FENCED, caused by
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> 2023-10-28 06:17:35,666 INFO resourcemanager.ResourceManager
> (ResourceManager.java:transitionToStandby(1309)) - Transitioning to standby
> state
> {code}
> h1. Solution
> The NoNodeException clearly indicates that the Znode no longer exists, so we
> can safely ignore this exception to avoid triggering a larger impact on the
> cluster caused by ResourceManager failover.
> h1. Other
> We also need to discuss and optimize the same issues in safeCreate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824376#comment-17824376
]
ASF GitHub Bot commented on YARN-11626:
---
XbaoWu commented on PR #6577:
URL: https://github.com/apache/hadoop/pull/6577#issuecomment-1983324064
> Hi @XbaoWu Please submit PR to trunk first, if approved and committed to
trunk, then backport to other active branches if necessary.
Okay, thank you for your reminder
> Optimization of the safeDelete operation in ZKRMStateStore
> --
>
> Key: YARN-11626
> URL: https://issues.apache.org/jira/browse/YARN-11626
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: resourcemanager
>Affects Versions: 3.0.0-alpha4, 3.1.1, 3.3.0
>Reporter: wangzhihui
>Priority: Minor
> Labels: pull-request-available
>
> h1. Description
> * We can be observed that removing app info started at 06:17:20, but the
> NoNodeException was received at 06:17:35.
> * During the 15s interval, Curator was retrying the metadata operation. Due
> to the non-idempotent nature of the Zookeeper deletion operation, in one of
> the retry attempts, the metadata operation was successful but no response was
> received. In the next retry it resulted in a NoNodeException, triggering the
> STATE_STORE_FENCED event and ultimately causing the current ResourceManager
> to switch to standby .
> {code:java}
> 2023-10-28 06:17:20,359 INFO recovery.RMStateStore
> (RMStateStore.java:transition(333)) - Removing info for app:
> application_1697410508608_140368
> 2023-10-28 06:17:20,359 INFO resourcemanager.RMAppManager
> (RMAppManager.java:checkAppNumCompletedLimit(303)) - Application should be
> expired, max number of completed apps kept in memory met:
> maxCompletedAppsInMemory = 1000, removing app
> application_1697410508608_140368 from memory:
> 2023-10-28 06:17:35,665 ERROR recovery.RMStateStore
> (RMStateStore.java:transition(337)) - Error removing app:
> application_1697410508608_140368
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
> 2023-10-28 06:17:35,666 INFO recovery.RMStateStore
> (RMStateStore.java:handleStoreEvent(1147)) - RMStateStore state change from
> ACTIVE to FENCED
> 2023-10-28 06:17:35,666 ERROR resourcemanager.ResourceManager
> (ResourceManager.java:handle(898)) - Received RMFatalEvent of type
> STATE_STORE_FENCED, caused by
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode
> 2023-10-28 06:17:35,666 INFO resourcemanager.ResourceManager
> (ResourceManager.java:transitionToStandby(1309)) - Transitioning to standby
> state
> {code}
> h1. Solution
> The NoNodeException clearly indicates that the Znode no longer exists, so we
> can safely ignore this exception to avoid triggering a larger impact on the
> cluster caused by ResourceManager failover.
> h1. Other
> We also need to discuss and optimize the same issues in safeCreate.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824449#comment-17824449
]
ASF GitHub Bot commented on YARN-11626:
---
hadoop-yetus commented on PR #6616:
URL: https://github.com/apache/hadoop/pull/6616#issuecomment-1983768684
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 18m 24s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 45s | | trunk passed |
| +1 :green_heart: | compile | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 51s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| +1 :green_heart: | compile | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 0m 45s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 44s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6616/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 108m 3s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 269m 12s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6616/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6616 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux ba39eed75d7a 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ce613be2e53778022e910c86be78f0d8c6ba1ec8 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-mu
[jira] [Commented] (YARN-11656) RMStateStore event queue blocked
[
https://issues.apache.org/jira/browse/YARN-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824884#comment-17824884
]
ASF GitHub Bot commented on YARN-11656:
---
hadoop-yetus commented on PR #6569:
URL: https://github.com/apache/hadoop/pull/6569#issuecomment-1986567474
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 47s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 36s | | trunk passed |
| +1 :green_heart: | compile | 8m 6s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 7m 21s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 6s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 43m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 33s | | the patch passed |
| +1 :green_heart: | compile | 9m 18s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 9m 18s | | the patch passed |
| +1 :green_heart: | compile | 9m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 9m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 55s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/9/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 14 new + 95 unchanged
- 3 fixed = 109 total (was 98) |
| +1 :green_heart: | mvnsite | 1m 52s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 52s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 48s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 1m 56s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/9/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.html)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 40m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 37s | | hadoop-yarn-common in the patch
passed. |
| -1 :x: | unit | 104m 43s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/9/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 55s | | The patch does not
generate ASF License warnings. |
| | | 328m 36s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| | Bad attempt to compute absolute value of signed 32-bit hashcode in
org.apache.hadoop.yarn.event.multidispatcher.MultiDispatcherExecutor.execute(Event,
Runnable) At MultiDispatcherExecutor.java:value of signed 32-bit hashcode in
org.apache.hadoop.yarn.event.multidispatcher.MultiD
[jira] [Commented] (YARN-11656) RMStateStore event queue blocked
[
https://issues.apache.org/jira/browse/YARN-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824960#comment-17824960
]
ASF GitHub Bot commented on YARN-11656:
---
hadoop-yetus commented on PR #6569:
URL: https://github.com/apache/hadoop/pull/6569#issuecomment-1986880158
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 8s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 50s | | trunk passed |
| +1 :green_heart: | compile | 8m 56s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 7m 41s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 2m 8s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 0s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 3s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 53s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 8m 3s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 8m 3s | | the patch passed |
| +1 :green_heart: | compile | 8m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 8m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 3s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/10/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 20 new + 95 unchanged
- 3 fixed = 115 total (was 98) |
| +1 :green_heart: | mvnsite | 2m 5s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 53s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 59s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 40s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 108m 43s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 313m 2s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6569 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 2c6393f94f3c 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 465a0de4f481ee2dc7383e6fb40b93e32135ba3c |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.
[jira] [Commented] (YARN-11656) RMStateStore event queue blocked
[
https://issues.apache.org/jira/browse/YARN-11656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17824961#comment-17824961
]
ASF GitHub Bot commented on YARN-11656:
---
hadoop-yetus commented on PR #6569:
URL: https://github.com/apache/hadoop/pull/6569#issuecomment-1986880841
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 4s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 37m 43s | | trunk passed |
| +1 :green_heart: | compile | 9m 47s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 7m 57s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 2m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 10s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 2m 0s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 12s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 24s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 8m 57s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 8m 57s | | the patch passed |
| +1 :green_heart: | compile | 8m 18s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 8m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 56s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/11/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 20 new + 95 unchanged
- 3 fixed = 115 total (was 98) |
| +1 :green_heart: | mvnsite | 1m 58s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 59s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 52s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 41m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 42s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 108m 8s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 55s | | The patch does not
generate ASF License warnings. |
| | | 314m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6569/11/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6569 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 815f880bc435 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 465a0de4f481ee2dc7383e6fb40b93e32135ba3c |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.
[jira] [Updated] (YARN-11660) SingleConstraintAppPlacementAllocator performance regression
[ https://issues.apache.org/jira/browse/YARN-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11660: -- Labels: pull-request-available (was: ) > SingleConstraintAppPlacementAllocator performance regression > > > Key: YARN-11660 > URL: https://issues.apache.org/jira/browse/YARN-11660 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11660) SingleConstraintAppPlacementAllocator performance regression
[ https://issues.apache.org/jira/browse/YARN-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825562#comment-17825562 ] ASF GitHub Bot commented on YARN-11660: --- zuston opened a new pull request, #6623: URL: https://github.com/apache/hadoop/pull/6623 ### Description of PR When using the `SingleConstraintAppPlacementAllocator` with scheduling request in our internal cluster, I found the huge performance regression from the metric of `allocateAvgTime`. After digging this bug, I found this dangerous bug that will always checking the non-pending resource apps due to missing of desc pending resource for one app for the async scheduling threads. ### How was this patch tested? Has been applied in our internal cluster ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > SingleConstraintAppPlacementAllocator performance regression > > > Key: YARN-11660 > URL: https://issues.apache.org/jira/browse/YARN-11660 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11660) SingleConstraintAppPlacementAllocator performance regression
[
https://issues.apache.org/jira/browse/YARN-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825632#comment-17825632
]
ASF GitHub Bot commented on YARN-11660:
---
hadoop-yetus commented on PR #6623:
URL: https://github.com/apache/hadoop/pull/6623#issuecomment-1991401418
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 31m 49s | | trunk passed |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 9s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 88m 49s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 171m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6623/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6623 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5a9aac54ba70 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4eba7856a91f4b35a57b40e6cabb6ed3cb58d5c9 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6623/1/testReport/ |
| Max. process+thread count | 926 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6623/1/consol
[jira] [Commented] (YARN-11660) SingleConstraintAppPlacementAllocator performance regression
[
https://issues.apache.org/jira/browse/YARN-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825703#comment-17825703
]
ASF GitHub Bot commented on YARN-11660:
---
slfan1989 commented on code in PR #6623:
URL: https://github.com/apache/hadoop/pull/6623#discussion_r1521601149
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/SingleConstraintAppPlacementAllocator.java:
##
@@ -309,6 +309,10 @@ private void decreasePendingNumAllocation() {
// Deduct pending #allocations by 1
ResourceSizing sizing = schedulingRequest.getResourceSizing();
sizing.setNumAllocations(sizing.getNumAllocations() - 1);
+
+appSchedulingInfo.decPendingResource(
Review Comment:
one line may be better.
> SingleConstraintAppPlacementAllocator performance regression
>
>
> Key: YARN-11660
> URL: https://issues.apache.org/jira/browse/YARN-11660
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11660) SingleConstraintAppPlacementAllocator performance regression
[
https://issues.apache.org/jira/browse/YARN-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825884#comment-17825884
]
ASF GitHub Bot commented on YARN-11660:
---
zuston commented on code in PR #6623:
URL: https://github.com/apache/hadoop/pull/6623#discussion_r1522391003
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/SingleConstraintAppPlacementAllocator.java:
##
@@ -309,6 +309,10 @@ private void decreasePendingNumAllocation() {
// Deduct pending #allocations by 1
ResourceSizing sizing = schedulingRequest.getResourceSizing();
sizing.setNumAllocations(sizing.getNumAllocations() - 1);
+
+appSchedulingInfo.decPendingResource(
Review Comment:
Done
> SingleConstraintAppPlacementAllocator performance regression
>
>
> Key: YARN-11660
> URL: https://issues.apache.org/jira/browse/YARN-11660
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11660) SingleConstraintAppPlacementAllocator performance regression
[
https://issues.apache.org/jira/browse/YARN-11660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825905#comment-17825905
]
ASF GitHub Bot commented on YARN-11660:
---
hadoop-yetus commented on PR #6623:
URL: https://github.com/apache/hadoop/pull/6623#issuecomment-1993566244
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 51s | | trunk passed |
| +1 :green_heart: | compile | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 8s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 86m 39s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 25s | | The patch does not
generate ASF License warnings. |
| | | 170m 10s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6623/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6623 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5dcf70804e69 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3cd69ab870947d5b597703b2658957e9fef4cf52 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6623/2/testReport/ |
| Max. process+thread count | 939 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6623/2/consol
[jira] [Updated] (YARN-5305) Yarn Application Log Aggregation fails due to NM can not get correct HDFS delegation token III
[ https://issues.apache.org/jira/browse/YARN-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-5305: - Labels: pull-request-available (was: ) > Yarn Application Log Aggregation fails due to NM can not get correct HDFS > delegation token III > -- > > Key: YARN-5305 > URL: https://issues.apache.org/jira/browse/YARN-5305 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Xianyin Xin >Assignee: Peter Szucs >Priority: Major > Labels: pull-request-available > > Different with YARN-5098 and YARN-5302, this problem happens when AM submits > a startContainer request with a new HDFS token (say, tokenB) which is not > managed by YARN, so two tokens exist in the credentials of the user on NM, > one is tokenB, the other is the one renewed on RM (tokenA). If tokenB is > selected when connect to HDFS and tokenB expires, exception happens. > Supplementary: this problem happen due to that AM didn't use the service name > as the token alias in credentials, so two tokens for the same service can > co-exist in one credentials. TokenSelector can only select the first matched > token, it doesn't care if the token is valid or not. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5305) Yarn Application Log Aggregation fails due to NM can not get correct HDFS delegation token III
[ https://issues.apache.org/jira/browse/YARN-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17826048#comment-17826048 ] ASF GitHub Bot commented on YARN-5305: -- p-szucs opened a new pull request, #6625: URL: https://github.com/apache/hadoop/pull/6625 Change-Id: I47937134c8d8e1537c6af1033cc590dd528dda11 ### Description of PR Quoting the upstream ticket for problem statement: > This problem happens when AM submits a startContainer request with a new HDFS token (say, tokenB) which is not managed by YARN, so two tokens exist in the credentials of the user on NM, one is tokenB, the other is the one renewed on RM (tokenA). If tokenB is selected when connect to HDFS and tokenB expires, exception happens. This change removes the expired delegation tokens from the credentials before the log aggregation tries to write the logs into HDFS, so we can avoid getting InvalidToken exceptions. ### How was this patch tested? Added unit test. Tested the change on a cluster having an expired HDFS delegation token present in the credentials when trying to use HDFS. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Yarn Application Log Aggregation fails due to NM can not get correct HDFS > delegation token III > -- > > Key: YARN-5305 > URL: https://issues.apache.org/jira/browse/YARN-5305 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn >Reporter: Xianyin Xin >Assignee: Peter Szucs >Priority: Major > > Different with YARN-5098 and YARN-5302, this problem happens when AM submits > a startContainer request with a new HDFS token (say, tokenB) which is not > managed by YARN, so two tokens exist in the credentials of the user on NM, > one is tokenB, the other is the one renewed on RM (tokenA). If tokenB is > selected when connect to HDFS and tokenB expires, exception happens. > Supplementary: this problem happen due to that AM didn't use the service name > as the token alias in credentials, so two tokens for the same service can > co-exist in one credentials. TokenSelector can only select the first matched > token, it doesn't care if the token is valid or not. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5305) Yarn Application Log Aggregation fails due to NM can not get correct HDFS delegation token III
[
https://issues.apache.org/jira/browse/YARN-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17826808#comment-17826808
]
ASF GitHub Bot commented on YARN-5305:
--
hadoop-yetus commented on PR #6625:
URL: https://github.com/apache/hadoop/pull/6625#issuecomment-1995139286
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 41s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 41s | | trunk passed |
| +1 :green_heart: | compile | 17m 33s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 17m 48s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 39s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 11s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 50s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 35s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6625/1/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 33m 58s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 32s | | the patch passed |
| +1 :green_heart: | compile | 16m 40s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 16m 40s | | the patch passed |
| +1 :green_heart: | compile | 17m 9s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 17m 9s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6625/1/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 26s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6625/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 3 new + 197 unchanged - 0 fixed = 200 total (was
197) |
| +1 :green_heart: | mvnsite | 2m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 58s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 45s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 1m 50s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6625/1/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 35m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 20m 13s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 24m 43s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 58s | | The patch does not
generate ASF License warnings. |
| | | 270m 37s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| |
org.apache.hadoop.yarn.server.nodemanager.containermanager.
[jira] [Commented] (YARN-5305) Yarn Application Log Aggregation fails due to NM can not get correct HDFS delegation token III
[
https://issues.apache.org/jira/browse/YARN-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17826860#comment-17826860
]
ASF GitHub Bot commented on YARN-5305:
--
hadoop-yetus commented on PR #6625:
URL: https://github.com/apache/hadoop/pull/6625#issuecomment-1995665423
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 38s | | trunk passed |
| +1 :green_heart: | compile | 18m 25s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 17m 57s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 13s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 46s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 43s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6625/2/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 34m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 32s | | the patch passed |
| +1 :green_heart: | compile | 16m 37s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 16m 37s | | the patch passed |
| +1 :green_heart: | compile | 16m 0s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 16m 0s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6625/2/artifact/out/blanks-eol.txt)
| The patch has 2 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 4m 20s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6625/2/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 5 new + 197 unchanged - 0 fixed = 202 total (was
197) |
| +1 :green_heart: | mvnsite | 2m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 8s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 52s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 1m 51s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6625/2/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | shadedclient | 33m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 36s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 24m 41s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not
generate ASF License warnings. |
| | | 271m 59s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| |
org.apache.hadoop.yarn.server.nodemanager.containermanager.
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827022#comment-17827022
]
ASF GitHub Bot commented on YARN-11626:
---
hadoop-yetus commented on PR #6616:
URL: https://github.com/apache/hadoop/pull/6616#issuecomment-1997036293
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 57s | | trunk passed |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 22s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6616/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 89m 6s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 23s | | The patch does not
generate ASF License warnings. |
| | | 172m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6616/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6616 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 52e3af425cb9 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ce613be2e53778022e910c86be78f0d8c6ba1ec8 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-mu
[jira] [Commented] (YARN-11626) Optimization of the safeDelete operation in ZKRMStateStore
[
https://issues.apache.org/jira/browse/YARN-11626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827056#comment-17827056
]
ASF GitHub Bot commented on YARN-11626:
---
hadoop-yetus commented on PR #6616:
URL: https://github.com/apache/hadoop/pull/6616#issuecomment-1997193295
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 51s | | trunk passed |
| +1 :green_heart: | compile | 1m 1s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| +1 :green_heart: | compile | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 0m 45s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 43s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6616/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 3 new + 5 unchanged - 0 fixed = 8 total (was 5) |
| +1 :green_heart: | mvnsite | 0m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 39s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 39s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 107m 59s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 255m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6616/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6616 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux a582a2f4c055 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ce613be2e53778022e910c86be78f0d8c6ba1ec8 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-mu
[jira] [Commented] (YARN-5305) Yarn Application Log Aggregation fails due to NM can not get correct HDFS delegation token III
[
https://issues.apache.org/jira/browse/YARN-5305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827139#comment-17827139
]
ASF GitHub Bot commented on YARN-5305:
--
hadoop-yetus commented on PR #6625:
URL: https://github.com/apache/hadoop/pull/6625#issuecomment-1997734839
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 35s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 29s | | trunk passed |
| +1 :green_heart: | compile | 17m 27s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 15m 52s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 12s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 49s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| -1 :x: | spotbugs | 2m 36s |
[/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6625/3/artifact/out/branch-spotbugs-hadoop-common-project_hadoop-common-warnings.html)
| hadoop-common-project/hadoop-common in trunk has 1 extant spotbugs
warnings. |
| +1 :green_heart: | shadedclient | 33m 55s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 16m 51s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 16m 51s | | the patch passed |
| +1 :green_heart: | compile | 15m 49s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 15m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 17s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6625/3/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 197 unchanged - 0 fixed = 198 total (was
197) |
| +1 :green_heart: | mvnsite | 2m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 6s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 50s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 33s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 46s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 24m 40s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 1m 4s | | The patch does not
generate ASF License warnings. |
| | | 265m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6625/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6625 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ddd33f3ff055 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a27924044ad204e0e5a704152a7912986d87c38e |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd6
