[jira] [Commented] (YARN-11584) [CS] Attempting to create Leaf Queue with empty shortname should fail without crashing RM
[
https://issues.apache.org/jira/browse/YARN-11584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1503#comment-1503
]
ASF GitHub Bot commented on YARN-11584:
---
hadoop-yetus commented on PR #6148:
URL: https://github.com/apache/hadoop/pull/6148#issuecomment-1771840256
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 27s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 34m 10s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 27s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 11s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 85m 52s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6148/4/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 175m 30s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6148/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6148 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 2a7576f3fc04 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5d7604ba55c168481eb0226d0273c685bf694fc7 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6148/4/testReport/ |
| Max. process+thread count | 973 (vs. ulimit of
[jira] [Commented] (YARN-11593) [Federation] Improve command line help information.
[
https://issues.apache.org/jira/browse/YARN-11593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1478#comment-1478
]
ASF GitHub Bot commented on YARN-11593:
---
hadoop-yetus commented on PR #6199:
URL: https://github.com/apache/hadoop/pull/6199#issuecomment-1771749746
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 14s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 30s | | trunk passed |
| +1 :green_heart: | compile | 7m 49s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 7m 3s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 23s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 14s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +0 :ok: | spotbugs | 0m 36s | |
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 32m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 6m 29s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 6m 29s | | the patch passed |
| +1 :green_heart: | compile | 6m 27s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 6m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 43s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6199/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 5 new + 0 unchanged -
0 fixed = 5 total (was 0) |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 8s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 6s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +0 :ok: | spotbugs | 0m 29s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 32m 5s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 0m 41s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6199/7/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-client in the patch failed. |
| +1 :green_heart: | unit | 0m 31s | | hadoop-yarn-site in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not
generate ASF License warnings. |
| | | 166m 20s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6199/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6199 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux d26636e5bac8 5.15.0-86-generic #96-Ubuntu SMP Wed Sep 20
08:23:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-9013) [GPG] fix order of steps cleaning Registry entries in ApplicationCleaner
[
https://issues.apache.org/jira/browse/YARN-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1462#comment-1462
]
ASF GitHub Bot commented on YARN-9013:
--
hadoop-yetus commented on PR #6147:
URL: https://github.com/apache/hadoop/pull/6147#issuecomment-1771671879
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 52s | | trunk passed |
| +1 :green_heart: | compile | 7m 54s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 7m 6s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 2m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 31s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 18s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 37m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 7m 13s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 7m 13s | | the patch passed |
| +1 :green_heart: | compile | 7m 3s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 7m 3s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 51s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 11s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 13s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 8s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 5m 10s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 9s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 36s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 1m 2s | |
hadoop-yarn-server-globalpolicygenerator in the patch passed. |
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not
generate ASF License warnings. |
| | | 201m 24s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6147/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6147 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 34e5236cc430 5.15.0-78-generic #85-Ubuntu SMP Fri Jul 7
15:25:09 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ded9f555f526fa748e262e6e008ce0e163093910 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6147/6/testReport/ |
| Max. process+thread count | 529 (vs. ulimit of 5500) |
[jira] [Commented] (YARN-11584) [CS] Attempting to create Leaf Queue with empty shortname should fail without crashing RM
[
https://issues.apache.org/jira/browse/YARN-11584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1461#comment-1461
]
ASF GitHub Bot commented on YARN-11584:
---
bgoerlitz commented on code in PR #6148:
URL: https://github.com/apache/hadoop/pull/6148#discussion_r1366074215
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java:
##
@@ -509,6 +509,12 @@ public AbstractLeafQueue createQueue(QueuePath queue)
String leafQueueName = queue.getLeafName();
String parentQueueName = queue.getParent();
+if (StringUtils.isEmpty(leafQueueName)) {
+ throw new SchedulerDynamicEditException(
+ "Trying to create new queue=" + queue.getFullPath()
+ + ", which has empty leaf shortname.");
+}
+
Review Comment:
Agreed with this. This path currently does fail due to the fact that
`QueuePath.hasEmptyParts()` does not actually return true for an empty leaf
name, but the added unit test for `QueueManager.createQueue(QueuePath)` is not
necessary, so I will remove that in the next commit.
> [CS] Attempting to create Leaf Queue with empty shortname should fail without
> crashing RM
> -
>
> Key: YARN-11584
> URL: https://issues.apache.org/jira/browse/YARN-11584
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Brian Goerlitz
>Assignee: Brian Goerlitz
>Priority: Major
> Labels: pull-request-available
>
> If an app submission results in attempting to auto-create a leaf queue with
> an empty short name, the app submission should be rejected without the RM
> crashing. Currently, the queue will be created, but the RM encounters a FATAL
> exception due to metrics collision.
> For example, if an app is placed to 'root.' the RM will fail with the below.
> {noformat}
> 2023-09-12 20:23:43,294 FATAL org.apache.hadoop.yarn.event.EventDispatcher:
> Error in handling event type APP_ADDED to the Event Dispatcher
> org.apache.hadoop.metrics2.MetricsException: Metrics source
> QueueMetrics,q0=root already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueMetrics.forQueue(CSQueueMetrics.java:309)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.(AbstractCSQueue.java:147)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractLeafQueue.(AbstractLeafQueue.java:148)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:42)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.createNewQueue(ParentQueue.java:495)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.addDynamicChildQueue(ParentQueue.java:563)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.addDynamicLeafQueue(ParentQueue.java:517)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.createAutoQueue(CapacitySchedulerQueueManager.java:678)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.createQueue(CapacitySchedulerQueueManager.java:511)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getOrCreateQueueFromPlacementContext(CapacityScheduler.java:898)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplication(CapacityScheduler.java:962)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1920)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:170)
> at
> org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Commented] (YARN-11584) [CS] Attempting to create Leaf Queue with empty shortname should fail without crashing RM
[
https://issues.apache.org/jira/browse/YARN-11584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1460#comment-1460
]
ASF GitHub Bot commented on YARN-11584:
---
bgoerlitz commented on PR #6148:
URL: https://github.com/apache/hadoop/pull/6148#issuecomment-1771653899
Thanks for the comments @brumi1024. In fact it turns out that
`QueuePath.hasEmptyParts()` does not detect an empty leaf, so I will revise the
patch to address the issue there instead.
> [CS] Attempting to create Leaf Queue with empty shortname should fail without
> crashing RM
> -
>
> Key: YARN-11584
> URL: https://issues.apache.org/jira/browse/YARN-11584
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Reporter: Brian Goerlitz
>Assignee: Brian Goerlitz
>Priority: Major
> Labels: pull-request-available
>
> If an app submission results in attempting to auto-create a leaf queue with
> an empty short name, the app submission should be rejected without the RM
> crashing. Currently, the queue will be created, but the RM encounters a FATAL
> exception due to metrics collision.
> For example, if an app is placed to 'root.' the RM will fail with the below.
> {noformat}
> 2023-09-12 20:23:43,294 FATAL org.apache.hadoop.yarn.event.EventDispatcher:
> Error in handling event type APP_ADDED to the Event Dispatcher
> org.apache.hadoop.metrics2.MetricsException: Metrics source
> QueueMetrics,q0=root already exists!
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.newSourceName(DefaultMetricsSystem.java:152)
> at
> org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.sourceName(DefaultMetricsSystem.java:125)
> at
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl.register(MetricsSystemImpl.java:229)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CSQueueMetrics.forQueue(CSQueueMetrics.java:309)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue.(AbstractCSQueue.java:147)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractLeafQueue.(AbstractLeafQueue.java:148)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue.(LeafQueue.java:42)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.createNewQueue(ParentQueue.java:495)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.addDynamicChildQueue(ParentQueue.java:563)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.ParentQueue.addDynamicLeafQueue(ParentQueue.java:517)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.createAutoQueue(CapacitySchedulerQueueManager.java:678)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager.createQueue(CapacitySchedulerQueueManager.java:511)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.getOrCreateQueueFromPlacementContext(CapacityScheduler.java:898)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.addApplication(CapacityScheduler.java:962)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:1920)
> at
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler.handle(CapacityScheduler.java:170)
> at
> org.apache.hadoop.yarn.event.EventDispatcher$EventProcessor.run(EventDispatcher.java:66)
> at java.base/java.lang.Thread.run(Thread.java:834)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11592) Add timeout to GPGUtils#invokeRMWebService.
[
https://issues.apache.org/jira/browse/YARN-11592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1443#comment-1443
]
ASF GitHub Bot commented on YARN-11592:
---
hadoop-yetus commented on PR #6189:
URL: https://github.com/apache/hadoop/pull/6189#issuecomment-1771576107
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 26s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 59s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 51s | | trunk passed |
| +1 :green_heart: | compile | 4m 29s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 4m 1s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 54s | | trunk passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 45s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 3m 54s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 3m 54s | | the patch passed |
| +1 :green_heart: | compile | 3m 53s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 3m 53s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 8s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6189/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 164 unchanged
- 0 fixed = 165 total (was 164) |
| +1 :green_heart: | mvnsite | 2m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 31s | | the patch passed with JDK
Ubuntu-11.0.20.1+1-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 2m 27s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 4m 40s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 57s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 4m 50s | | hadoop-yarn-common in the patch
passed. |
| -1 :x: | unit | 0m 36s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6189/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-client in the patch failed. |
| +1 :green_heart: | unit | 0m 55s | |
hadoop-yarn-server-globalpolicygenerator in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 137m 2s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6189/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6189 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 4db31f3a3fa2 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64
[jira] [Commented] (YARN-11593) [Federation] Improve command line help information.
[ https://issues.apache.org/jira/browse/YARN-11593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1390#comment-1390 ] ASF GitHub Bot commented on YARN-11593: --- slfan1989 commented on PR #6199: URL: https://github.com/apache/hadoop/pull/6199#issuecomment-1771406631 @goiri Can you help review this PR again? Thank you very much! I improved the code to make it more readable. > [Federation] Improve command line help information. > --- > > Key: YARN-11593 > URL: https://issues.apache.org/jira/browse/YARN-11593 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > > This jira will improve the help information of routeradmin so that users can > better understand and use commands. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9013) [GPG] fix order of steps cleaning Registry entries in ApplicationCleaner
[
https://issues.apache.org/jira/browse/YARN-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1388#comment-1388
]
ASF GitHub Bot commented on YARN-9013:
--
slfan1989 commented on code in PR #6147:
URL: https://github.com/apache/hadoop/pull/6147#discussion_r1365878423
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/test/java/org/apache/hadoop/yarn/server/globalpolicygenerator/applicationcleaner/TestDefaultApplicationCleaner.java:
##
@@ -159,7 +163,39 @@ public class TestableDefaultApplicationCleaner
extends DefaultApplicationCleaner {
@Override
public Set getAppsFromRouter() throws YarnRuntimeException {
+ if (appIdToAddConcurrently != null) {
+SubClusterId scId = SubClusterId.newInstance("MySubClusterId");
+try {
+ ApplicationHomeSubCluster appHomeSubCluster =
+ ApplicationHomeSubCluster.newInstance(appIdToAddConcurrently,
scId);
+ AddApplicationHomeSubClusterRequest request =
+
AddApplicationHomeSubClusterRequest.newInstance(appHomeSubCluster);
+ stateStore.addApplicationHomeSubCluster(request);
+} catch (YarnException e) {
+ throw new YarnRuntimeException(e);
+}
+registryClient.writeAMRMTokenForUAM(appIdToAddConcurrently,
scId.toString(),
+new Token<>());
+ }
return routerAppIds;
}
}
+
+ @Test
+ public void testConcurrentNewApp() throws YarnException {
+appIdToAddConcurrently = ApplicationId.newInstance(1, 1);
+
+appCleaner.run();
+
+// The concurrently added app should be still there
+GetApplicationsHomeSubClusterRequest appHomeSubClusterRequest =
+ GetApplicationsHomeSubClusterRequest.newInstance();
+GetApplicationsHomeSubClusterResponse applicationsHomeSubCluster =
+stateStore.getApplicationsHomeSubCluster(appHomeSubClusterRequest);
+int size = applicationsHomeSubCluster.getAppsHomeSubClusters().size();
+Assert.assertEquals(1, size);
Review Comment:
I will improve it.
> [GPG] fix order of steps cleaning Registry entries in ApplicationCleaner
>
>
> Key: YARN-9013
> URL: https://issues.apache.org/jira/browse/YARN-9013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Major
> Labels: pull-request-available
> Attachments: YARN-9013-YARN-7402.v1.patch,
> YARN-9013-YARN-7402.v2.patch
>
>
> ApplicationCleaner today deletes the entry for all finished (non-running)
> application in YarnRegistry using this logic:
> # GPG gets the list of running applications from Router.
> # GPG gets the full list of applications in registry
> # GPG deletes in registry every app in 2 that’s not in 1
> The problem is that jobs that started between 1 and 2 meets the criteria in
> 3, and thus get deleted by mistake. The fix/right order should be 2->1->3,
> rather than 1->2->3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9013) [GPG] fix order of steps cleaning Registry entries in ApplicationCleaner
[
https://issues.apache.org/jira/browse/YARN-9013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1386#comment-1386
]
ASF GitHub Bot commented on YARN-9013:
--
slfan1989 commented on code in PR #6147:
URL: https://github.com/apache/hadoop/pull/6147#discussion_r1365877007
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java:
##
@@ -4367,7 +4367,7 @@ public static boolean isAclEnabled(Configuration conf) {
* the default value is 0s.
*/
public static final long
DEFAULT_ROUTER_USER_CLIENT_THREAD_POOL_KEEP_ALIVE_TIME =
- TimeUnit.SECONDS.toMillis(0); // 0s
+ TimeUnit.SECONDS.toMillis(30); // 0s
Review Comment:
Thank you for discovering this issue! It was my oversight, and I introduced
this problem while conducting integration testing. I will fix it.
> [GPG] fix order of steps cleaning Registry entries in ApplicationCleaner
>
>
> Key: YARN-9013
> URL: https://issues.apache.org/jira/browse/YARN-9013
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Botong Huang
>Assignee: Botong Huang
>Priority: Major
> Labels: pull-request-available
> Attachments: YARN-9013-YARN-7402.v1.patch,
> YARN-9013-YARN-7402.v2.patch
>
>
> ApplicationCleaner today deletes the entry for all finished (non-running)
> application in YarnRegistry using this logic:
> # GPG gets the list of running applications from Router.
> # GPG gets the full list of applications in registry
> # GPG deletes in registry every app in 2 that’s not in 1
> The problem is that jobs that started between 1 and 2 meets the criteria in
> 3, and thus get deleted by mistake. The fix/right order should be 2->1->3,
> rather than 1->2->3.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11592) Add timeout to GPGUtils#invokeRMWebService.
[ https://issues.apache.org/jira/browse/YARN-11592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1378#comment-1378 ] ASF GitHub Bot commented on YARN-11592: --- slfan1989 commented on code in PR #6189: URL: https://github.com/apache/hadoop/pull/6189#discussion_r1365869956 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/GPGUtils.java: ## @@ -128,4 +130,16 @@ public static Map createUniformWeights( } return weights; } + + @VisibleForTesting Review Comment: We are adding a configuration parameter for timeout, which is helpful for REST interfaces. In cases where a call to a ResourceManager (RM) does not return results as expected, it prevents situations where the system might get stuck. > Add timeout to GPGUtils#invokeRMWebService. > --- > > Key: YARN-11592 > URL: https://issues.apache.org/jira/browse/YARN-11592 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > > GPGUtils#invokeRMWebService utilizes a Client, and we should set a timeout > for the Client. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11593) [Federation] Improve command line help information.
[
https://issues.apache.org/jira/browse/YARN-11593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1374#comment-1374
]
ASF GitHub Bot commented on YARN-11593:
---
slfan1989 commented on code in PR #6199:
URL: https://github.com/apache/hadoop/pull/6199#discussion_r1365863997
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RouterCLI.java:
##
@@ -77,25 +79,70 @@ public class RouterCLI extends Configured implements Tool {
private static final Logger LOG = LoggerFactory.getLogger(RouterCLI.class);
- protected final static Map ADMIN_USAGE =
- ImmutableMap.builder()
- // Command1: deregisterSubCluster
-.put("-deregisterSubCluster", new UsageInfo(
-"[-sc|--subClusterId [subCluster Id]]",
-"Deregister SubCluster, If the interval between the heartbeat time of
the subCluster " +
-"and the current time exceeds the timeout period, " +
-"set the state of the subCluster to SC_LOST."))
- // Command2: policy
-.put("-policy", new UsageInfo(
-"[-s|--save [queue;router weight;amrm weight;headroomalpha]] " +
- "[-bs|--batch-save [--format xml] [-f|--input-file fileName]]" +
- "[-l|--list [--pageSize][--currentPage][--queue][--queues]]",
-"We provide a set of commands for Policy:" +
-" Include list policies, save policies, batch save policies. " +
-" (Note: The policy type will be directly read from the" +
-" yarn.federation.policy-manager in the local yarn-site.xml.)" +
-" eg. (routeradmin -policy [-s|--save]
root.a;SC-1:0.7,SC-2:0.3;SC-1:0.7,SC-2:0.3;1.0)"))
-.build();
+ protected final static UsageInfo SUBCLUSTER_ID = new
UsageInfo("<-sc|--subClusterId>",
+ "'-sc' option allows you to specify the sub-cluster to operate on, " +
+ "while the '--subClusterId' option is the long format of -sc and serves
the same purpose.");
+ protected final static String DSEXAMPLE_1 = "yarn routeradmin
-deregisterSubCluster -sc SC-1";
+ protected final static String DSEXAMPLE_2 = "yarn routeradmin
-deregisterSubCluster --subClusterId SC-1";
+
+ protected final static RouterExampleUsageInfos DSEXAMPLEUSAGEINFOS_1 = new
RouterExampleUsageInfos(
+ Arrays.asList(DSEXAMPLE_1, DSEXAMPLE_2),
+ Arrays.asList("At this point we can use the following command:"));
+ protected final static RouterUsageInfos DEREGISTER_SUBCLUSTER =
+ new RouterUsageInfos(Arrays.asList(SUBCLUSTER_ID),
+ Arrays.asList("deregister subCluster, " +
Review Comment:
Thanks for your feedback! I will improve this part of the code to make it
more readable.
> [Federation] Improve command line help information.
> ---
>
> Key: YARN-11593
> URL: https://issues.apache.org/jira/browse/YARN-11593
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
>
> This jira will improve the help information of routeradmin so that users can
> better understand and use commands.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11593) [Federation] Improve command line help information.
[
https://issues.apache.org/jira/browse/YARN-11593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1375#comment-1375
]
ASF GitHub Bot commented on YARN-11593:
---
slfan1989 commented on code in PR #6199:
URL: https://github.com/apache/hadoop/pull/6199#discussion_r1365864346
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/RouterCLI.java:
##
@@ -159,43 +214,59 @@ public RouterCLI(Configuration conf) {
}
private static void buildHelpMsg(String cmd, StringBuilder builder) {
-UsageInfo usageInfo = ADMIN_USAGE.get(cmd);
-if (usageInfo == null) {
+RouterUsageInfos routerUsageInfo = ADMIN_USAGE.get(cmd);
+
+if (routerUsageInfo == null) {
return;
}
+builder.append("[").append(cmd).append("]\n");
-if (usageInfo.args != null) {
- String space = (usageInfo.args == "") ? "" : " ";
- builder.append(" ")
- .append(cmd)
- .append(space)
- .append(usageInfo.args)
- .append(": ")
- .append(usageInfo.help);
-} else {
- builder.append(" ")
- .append(cmd)
- .append(": ")
- .append(usageInfo.help);
+if (!routerUsageInfo.helpInfos.isEmpty()) {
+ builder.append("\t Description: \n");
+ for (String helpInfo : routerUsageInfo.helpInfos) {
+builder.append("\t\t").append(helpInfo).append("\n\n");
+ }
}
- }
- private static void buildIndividualUsageMsg(String cmd, StringBuilder
builder) {
-UsageInfo usageInfo = ADMIN_USAGE.get(cmd);
-if (usageInfo == null) {
- return;
+if (!routerUsageInfo.usageInfos.isEmpty()) {
+ builder.append("\t UsageInfos: \n");
+ for (UsageInfo usageInfo : routerUsageInfo.usageInfos) {
+builder.append("\t\t").append(usageInfo.args)
+.append(": ")
+.append("\n\t\t")
+.append(usageInfo.help).append("\n\n");
+ }
}
-if (usageInfo.args == null) {
- builder.append("Usage: routeradmin [")
- .append(cmd)
- .append("]\n");
-} else {
- String space = (usageInfo.args == "") ? "" : " ";
- builder.append("Usage: routeradmin [")
- .append(cmd)
- .append(space)
- .append(usageInfo.args)
- .append("]\n");
+
+if (MapUtils.isNotEmpty(routerUsageInfo.examples)) {
+ builder.append("\t Examples: \n");
+ int count = 1;
+ for (Map.Entry example :
+ routerUsageInfo.examples.entrySet()) {
+String key = example.getKey();
+builder.append("\t\t").append("Cmd:").append(count).append(".
").append(key).append(": \n\n");
+RouterExampleUsageInfos values = example.getValue();
+if (values == null) {
+ return;
+}
+// Print Command Description
+if (CollectionUtils.isNotEmpty(values.desc)) {
+ builder.append("\t\t").append("Cmd Requirement Description:\n");
+ for (String value : values.desc) {
+builder.append("\t\t").append(value).append("\n");
+ }
+}
+builder.append("\n");
+// Print Command example
+if (CollectionUtils.isNotEmpty(values.examples)) {
Review Comment:
I will add unit tests.
> [Federation] Improve command line help information.
> ---
>
> Key: YARN-11593
> URL: https://issues.apache.org/jira/browse/YARN-11593
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
>
> This jira will improve the help information of routeradmin so that users can
> better understand and use commands.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
