[jira] [Commented] (YARN-11670) Add CallerContext in NodeManager
[ https://issues.apache.org/jira/browse/YARN-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832044#comment-17832044 ] ASF GitHub Bot commented on YARN-11670: --- yangjiandan commented on PR #6688: URL: https://github.com/apache/hadoop/pull/6688#issuecomment-2026710223 @dineshchitlangia Thank you very much for your review. In my test environment, when running MR jobs, some of the HDFS-audit logs are as follows: ` 2024-03-28 22:31:09,716 INFO FSNamesystem.audit: allowed=true ugi=hadoop (auth:SIMPLE)ip=/172.16.70.128 cmd=getfileinfo src=/tmp/hadoop-yarn/staging/hadoop/.staging/job_1711679421425_0001/job.splitmetainfo dst=nullperm=null proto=rpc callerContext=nodemanager_hadoop01 2024-03-28 22:31:09,717 INFO FSNamesystem.audit: allowed=true ugi=hadoop (auth:SIMPLE)ip=/172.16.70.128 cmd=getfileinfo src=/tmp/hadoop-yarn/staging/hadoop/.staging/job_1711679421425_0001/job.splitmetainfo dst=nullperm=null proto=rpc callerContext=nodemanager_hadoop01 2024-03-28 22:31:09,717 INFO FSNamesystem.audit: allowed=true ugi=hadoop (auth:SIMPLE)ip=/172.16.70.128 cmd=open src=/tmp/hadoop-yarn/staging/hadoop/.staging/job_1711679421425_0001/job.splitmetainfo dst=nullperm=null proto=rpc callerContext=nodemanager_hadoop01 2024-03-28 22:31:09,726 INFO FSNamesystem.audit: allowed=true ugi=hadoop (auth:SIMPLE)ip=/172.16.70.128 cmd=getfileinfo src=/tmp/hadoop-yarn/staging/hadoop/.staging/job_1711679421425_0001/job.jar dst=nullperm=null proto=rpc callerContext=nodemanager_hadoop01 2024-03-28 22:31:09,727 INFO FSNamesystem.audit: allowed=true ugi=hadoop (auth:SIMPLE)ip=/172.16.70.128 cmd=getfileinfo src=/tmp/hadoop-yarn/staging/hadoop/.staging/job_1711679421425_0001/job.jar dst=nullperm=null proto=rpc callerContext=nodemanager_hadoop01 2024-03-28 22:31:09,728 INFO FSNamesystem.audit: allowed=true ugi=hadoop (auth:SIMPLE)ip=/172.16.70.128 cmd=open src=/tmp/hadoop-yarn/staging/hadoop/.staging/job_1711679421425_0001/job.jar dst=nullperm=null proto=rpc callerContext=nodemanager_hadoop01 ` > Add CallerContext in NodeManager > > > Key: YARN-11670 > URL: https://issues.apache.org/jira/browse/YARN-11670 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Jiandan Yang >Priority: Major > Labels: pull-request-available > > Currently, MR and Spark have added caller context, enabling tracing of > HDFS/ResourceManager operators from Spark apps and MapReduce apps. However, > operators from NodeManagers cannot be identified in the audit log. For > example, HDFS operations issued from NodeManagers during resource > localization cannot be identified. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11670) Add CallerContext in NodeManager
[
https://issues.apache.org/jira/browse/YARN-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832032#comment-17832032
]
ASF GitHub Bot commented on YARN-11670:
---
hadoop-yetus commented on PR #6688:
URL: https://github.com/apache/hadoop/pull/6688#issuecomment-2026618787
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 48s | | trunk passed |
| +1 :green_heart: | compile | 0m 49s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 47s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 15s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6688/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 1 new + 18 unchanged - 0 fixed = 19 total (was 18) |
| +1 :green_heart: | mvnsite | 0m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 0m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 22m 18s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 106m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6688/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6688 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 52f0adffb427 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 38d1079fd597dfeba756c7d7197943ab77285bac |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
|
[jira] [Commented] (YARN-11670) Add CallerContext in NodeManager
[ https://issues.apache.org/jira/browse/YARN-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832017#comment-17832017 ] ASF GitHub Bot commented on YARN-11670: --- yangjiandan opened a new pull request, #6688: URL: https://github.com/apache/hadoop/pull/6688 ### Description of PR Currently, MR and Spark have added caller context, enabling tracing of HDFS/ResourceManager operators from Spark apps and MapReduce apps. However, operators from NodeManagers cannot be identified in the audit log. For example, HDFS operations issued from NodeManagers during resource localization cannot be identified. ### How was this patch tested? 1. current ut 2. system test in my local env ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Add CallerContext in NodeManager > > > Key: YARN-11670 > URL: https://issues.apache.org/jira/browse/YARN-11670 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Jiandan Yang >Priority: Major > > Currently, MR and Spark have added caller context, enabling tracing of > HDFS/ResourceManager operators from Spark apps and MapReduce apps. However, > operators from NodeManagers cannot be identified in the audit log. For > example, HDFS operations issued from NodeManagers during resource > localization cannot be identified. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11670) Add CallerContext in NodeManager
[ https://issues.apache.org/jira/browse/YARN-11670?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11670: -- Labels: pull-request-available (was: ) > Add CallerContext in NodeManager > > > Key: YARN-11670 > URL: https://issues.apache.org/jira/browse/YARN-11670 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Reporter: Jiandan Yang >Priority: Major > Labels: pull-request-available > > Currently, MR and Spark have added caller context, enabling tracing of > HDFS/ResourceManager operators from Spark apps and MapReduce apps. However, > operators from NodeManagers cannot be identified in the audit log. For > example, HDFS operations issued from NodeManagers during resource > localization cannot be identified. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11668) Potential concurrent modification exception for node attributes of node manager
[ https://issues.apache.org/jira/browse/YARN-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11668. --- Fix Version/s: 3.4.1 3.5.0 Hadoop Flags: Reviewed Target Version/s: 3.4.1 Assignee: Junfan Zhang Resolution: Fixed > Potential concurrent modification exception for node attributes of node > manager > --- > > Key: YARN-11668 > URL: https://issues.apache.org/jira/browse/YARN-11668 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.1, 3.5.0 > > Attachments: img_v3_029c_55ac6b50-64aa-4cbe-81a0-5f8d22c623fg.jpg > > > The RM crash when encoutering the following the stacktrace in the attachment. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11668) Potential concurrent modification exception for node attributes of node manager
[ https://issues.apache.org/jira/browse/YARN-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-11668: -- Affects Version/s: 3.4.0 > Potential concurrent modification exception for node attributes of node > manager > --- > > Key: YARN-11668 > URL: https://issues.apache.org/jira/browse/YARN-11668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.4.0 >Reporter: Junfan Zhang >Assignee: Junfan Zhang >Priority: Major > Labels: pull-request-available > Fix For: 3.4.1, 3.5.0 > > Attachments: img_v3_029c_55ac6b50-64aa-4cbe-81a0-5f8d22c623fg.jpg > > > The RM crash when encoutering the following the stacktrace in the attachment. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11668) Potential concurrent modification exception for node attributes of node manager
[ https://issues.apache.org/jira/browse/YARN-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832013#comment-17832013 ] ASF GitHub Bot commented on YARN-11668: --- slfan1989 commented on PR #6681: URL: https://github.com/apache/hadoop/pull/6681#issuecomment-2026455866 @zuston Thanks for the contribution! merged into trunk. @dineshchitlangia Thanks for the review! > Potential concurrent modification exception for node attributes of node > manager > --- > > Key: YARN-11668 > URL: https://issues.apache.org/jira/browse/YARN-11668 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Junfan Zhang >Priority: Major > Labels: pull-request-available > Attachments: img_v3_029c_55ac6b50-64aa-4cbe-81a0-5f8d22c623fg.jpg > > > The RM crash when encoutering the following the stacktrace in the attachment. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11670) Add CallerContext in NodeManager
Jiandan Yang created YARN-11670: Summary: Add CallerContext in NodeManager Key: YARN-11670 URL: https://issues.apache.org/jira/browse/YARN-11670 Project: Hadoop YARN Issue Type: Improvement Components: nodemanager Reporter: Jiandan Yang Currently, MR and Spark have added caller context, enabling tracing of HDFS/ResourceManager operators from Spark apps and MapReduce apps. However, operators from NodeManagers cannot be identified in the audit log. For example, HDFS operations issued from NodeManagers during resource localization cannot be identified. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11668) Potential concurrent modification exception for node attributes of node manager
[ https://issues.apache.org/jira/browse/YARN-11668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17832012#comment-17832012 ] ASF GitHub Bot commented on YARN-11668: --- slfan1989 merged PR #6681: URL: https://github.com/apache/hadoop/pull/6681 > Potential concurrent modification exception for node attributes of node > manager > --- > > Key: YARN-11668 > URL: https://issues.apache.org/jira/browse/YARN-11668 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Junfan Zhang >Priority: Major > Labels: pull-request-available > Attachments: img_v3_029c_55ac6b50-64aa-4cbe-81a0-5f8d22c623fg.jpg > > > The RM crash when encoutering the following the stacktrace in the attachment. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11669) cgroups v2 support for YARN
Ferenc Erdelyi created YARN-11669: - Summary: cgroups v2 support for YARN Key: YARN-11669 URL: https://issues.apache.org/jira/browse/YARN-11669 Project: Hadoop YARN Issue Type: New Feature Components: yarn Reporter: Ferenc Erdelyi The cgroups v2 is becoming the default for OSs, like RHEL9. Support for YARN has to be implemented. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11664) Remove HDFS Binaries/Jars Dependency From YARN
[
https://issues.apache.org/jira/browse/YARN-11664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831901#comment-17831901
]
ASF GitHub Bot commented on YARN-11664:
---
steveloughran commented on PR #6631:
URL: https://github.com/apache/hadoop/pull/6631#issuecomment-2025705847
waiting to see what hdfs people say; mentioned internally.
now, there is a way to do this with a smaller diff, specifically, move the
IOPair class into hadoop common *but keep with the same package name*.
something to seriously consider. would reduce the risk of any code elsewhere
making explicit use of the class then breaking.
> Remove HDFS Binaries/Jars Dependency From YARN
> --
>
> Key: YARN-11664
> URL: https://issues.apache.org/jira/browse/YARN-11664
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
>
> In principle Hadoop Yarn is independent of HDFS. It can work with any
> filesystem. Currently there exists some code dependency for Yarn with HDFS.
> This dependency requires Yarn to bring in some of the HDFS binaries/jars to
> its class path. The idea behind this jira is to remove this dependency so
> that Yarn can run without HDFS binaries/jars
> *Scope*
> 1. Non test classes are considered
> 2. Some test classes which comes as transitive dependency are considered
> *Out of scope*
> 1. All test classes in Yarn module is not considered
>
>
> A quick search in Yarn module revealed following HDFS dependencies
> 1. Constants
> {code:java}
> import
> org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier;
> import org.apache.hadoop.hdfs.DFSConfigKeys;{code}
>
>
> 2. Exception
> {code:java}
> import org.apache.hadoop.hdfs.protocol.DSQuotaExceededException;{code}
>
> 3. Utility
> {code:java}
> import org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair;{code}
>
> Both Yarn and HDFS depends on *hadoop-common* module,
> * Constants variables and Utility classes can be moved to *hadoop-common*
> * Instead of DSQuotaExceededException, Use the parent exception
> ClusterStoragrCapacityExceeded
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11582) Improve WebUI diagnosticMessage to show AM Container resource request size
[
https://issues.apache.org/jira/browse/YARN-11582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831854#comment-17831854
]
ASF GitHub Bot commented on YARN-11582:
---
hiwangzhihui commented on code in PR #6139:
URL: https://github.com/apache/hadoop/pull/6139#discussion_r1542224470
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimitsByPartition.java:
##
@@ -1027,4 +1027,79 @@ public void testAMLimitByAllResources() throws Exception
{
rm.close();
}
+
+ @Test(timeout = 12)
+ public void testDiagnosticWhenAMActivated() throws Exception {
+/*
+ * Test Case:
+ * Verify AM resource limit per partition level and per queue level. So
+ * we use 2 queues to verify this case.
+ * Queue a1 supports labels (x,y). Configure am-resource-limit as 0.2 (x)
+ * Queue c1 supports default label. Configure am-resource-limit as 0.2
+ *
+ * Queue a1 for label X can only support 2Gb AM resource.
+ * Queue c1 (empty label) can support 2Gb AM resource.
+ *
+ * Verify at least one AM is launched, and AM resources should not go more
+ * than 2GB in each queue.
+ */
+
+simpleNodeLabelMappingToManager();
+CapacitySchedulerConfiguration config = (CapacitySchedulerConfiguration)
+ TestUtils.getConfigurationWithQueueLabels(conf);
+
+// After getting queue conf, configure AM resource percent for Queue a1
+// as 0.2 (Label X) and for Queue c1 as 0.2 (Empty Label).
+config.setMaximumAMResourcePercentPerPartition(A1, "x", 0.2f);
+config.setMaximumApplicationMasterResourcePerQueuePercent(C1, 0.2f);
+
+// Now inject node label manager with this updated config.
+MockRM rm = new MockRM(config) {
+ @Override
+ public RMNodeLabelsManager createNodeLabelManager() {
+return mgr;
+ }
+};
+
+rm.getRMContext().setNodeLabelManager(mgr);
+rm.start();
+rm.registerNode("h1:1234", 10 * GB); // label = x
+rm.registerNode("h2:1234", 10 * GB); // label = y
+rm.registerNode("h3:1234", 10 * GB); // label =
+
+// Submit app1 with 1Gb AM resource to Queue a1 for label X
+MockRMAppSubmissionData data1 =
+ MockRMAppSubmissionData.Builder.createWithMemory(GB, rm)
+ .withAppName("app")
+ .withUser("user")
+ .withAcls(null)
+ .withQueue("a1")
+ .withAmLabel("x")
+ .build();
+RMApp app1 = MockRMAppSubmitter.submit(rm, data1);
+
+// Submit app2 with 1Gb AM resource to Queue a1 for label X
+MockRMAppSubmissionData data2 =
+ MockRMAppSubmissionData.Builder.createWithMemory(GB, rm)
+ .withAppName("app")
+ .withUser("user")
+ .withAcls(null)
+ .withQueue("a1")
+ .withAmLabel("x")
+ .build();
+RMApp app2 = MockRMAppSubmitter.submit(rm, data2);
+
+CapacityScheduler cs = (CapacityScheduler) rm.getResourceScheduler();
+LeafQueue leafQueue = (LeafQueue) cs.getQueue("a1");
+Assert.assertNotNull(leafQueue);
+
+// Only one AM will be activated here, and second AM will be still pending.
+Assert.assertEquals(2, leafQueue.getNumActiveApplications());
+String activatedDiagnostics="AM Resource Request = ";
+Assert.assertTrue("still doesn't show AMResource When Activated",
app1.getDiagnostics()
+ .toString().contains(activatedDiagnostics));
Review Comment:
Add a test checks for Am resource prompts would be better
> Improve WebUI diagnosticMessage to show AM Container resource request size
> --
>
> Key: YARN-11582
> URL: https://issues.apache.org/jira/browse/YARN-11582
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: applications, resourcemanager
>Affects Versions: 3.3.4
>Reporter: xiaojunxiang
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2023-10-02-00-05-34-337.png,
> image-2024-03-28-22-11-37-903.png, success_ShowAMInfo.jpg
>
>
> When Yarn resources are insufficient, the newly submitted job AM may be in
> the state of "Application is Activated, waiting for resources to be assigned
> for AM". This is obviously because Yarn doesn't have enough resources to
> allocate another AM Container, so we want to know how large the AM Container
> is currently allocated. Unfortunately, the current diagnosticMessage on the
> Web page does not show this data. Therefore, it is necessary to add the
> resource size of the AM Container in the diagnosticMessage, which will be
> very useful for us to troubleshoise the production faults on line.
--
This message was sent by
[jira] [Comment Edited] (YARN-11582) Improve WebUI diagnosticMessage to show AM Container resource request size
[ https://issues.apache.org/jira/browse/YARN-11582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831827#comment-17831827 ] wangzhihui edited comment on YARN-11582 at 3/28/24 2:15 PM: hi, [~slfan1989] . This [PR|https://github.com/apache/hadoop/pull/6139] has added valid Test content and passed the latest Jenkins check; please help merge it. Thanks! was (Author: JIRAUSER302479): hi, [~slfan1989] This [PR|https://github.com/apache/hadoop/pull/6139] has added valid Test content and passed the latest Jenkins check; please help merge it. Thanks! > Improve WebUI diagnosticMessage to show AM Container resource request size > -- > > Key: YARN-11582 > URL: https://issues.apache.org/jira/browse/YARN-11582 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications, resourcemanager >Affects Versions: 3.3.4 >Reporter: xiaojunxiang >Priority: Minor > Labels: pull-request-available > Attachments: image-2023-10-02-00-05-34-337.png, > image-2024-03-28-22-11-37-903.png, success_ShowAMInfo.jpg > > > When Yarn resources are insufficient, the newly submitted job AM may be in > the state of "Application is Activated, waiting for resources to be assigned > for AM". This is obviously because Yarn doesn't have enough resources to > allocate another AM Container, so we want to know how large the AM Container > is currently allocated. Unfortunately, the current diagnosticMessage on the > Web page does not show this data. Therefore, it is necessary to add the > resource size of the AM Container in the diagnosticMessage, which will be > very useful for us to troubleshoise the production faults on line. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11582) Improve WebUI diagnosticMessage to show AM Container resource request size
[ https://issues.apache.org/jira/browse/YARN-11582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831827#comment-17831827 ] wangzhihui commented on YARN-11582: --- hi, [~slfan1989] This [PR|https://github.com/apache/hadoop/pull/6139] has added valid Test content and passed the latest Jenkins check; please help merge it. Thanks! > Improve WebUI diagnosticMessage to show AM Container resource request size > -- > > Key: YARN-11582 > URL: https://issues.apache.org/jira/browse/YARN-11582 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications, resourcemanager >Affects Versions: 3.3.4 >Reporter: xiaojunxiang >Priority: Minor > Labels: pull-request-available > Attachments: image-2023-10-02-00-05-34-337.png, > image-2024-03-28-22-11-37-903.png, success_ShowAMInfo.jpg > > > When Yarn resources are insufficient, the newly submitted job AM may be in > the state of "Application is Activated, waiting for resources to be assigned > for AM". This is obviously because Yarn doesn't have enough resources to > allocate another AM Container, so we want to know how large the AM Container > is currently allocated. Unfortunately, the current diagnosticMessage on the > Web page does not show this data. Therefore, it is necessary to add the > resource size of the AM Container in the diagnosticMessage, which will be > very useful for us to troubleshoise the production faults on line. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11582) Improve WebUI diagnosticMessage to show AM Container resource request size
[ https://issues.apache.org/jira/browse/YARN-11582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangzhihui updated YARN-11582: -- Attachment: image-2024-03-28-22-11-37-903.png > Improve WebUI diagnosticMessage to show AM Container resource request size > -- > > Key: YARN-11582 > URL: https://issues.apache.org/jira/browse/YARN-11582 > Project: Hadoop YARN > Issue Type: Improvement > Components: applications, resourcemanager >Affects Versions: 3.3.4 >Reporter: xiaojunxiang >Priority: Minor > Labels: pull-request-available > Attachments: image-2023-10-02-00-05-34-337.png, > image-2024-03-28-22-11-37-903.png, success_ShowAMInfo.jpg > > > When Yarn resources are insufficient, the newly submitted job AM may be in > the state of "Application is Activated, waiting for resources to be assigned > for AM". This is obviously because Yarn doesn't have enough resources to > allocate another AM Container, so we want to know how large the AM Container > is currently allocated. Unfortunately, the current diagnosticMessage on the > Web page does not show this data. Therefore, it is necessary to add the > resource size of the AM Container in the diagnosticMessage, which will be > very useful for us to troubleshoise the production faults on line. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11582) Improve WebUI diagnosticMessage to show AM Container resource request size
[
https://issues.apache.org/jira/browse/YARN-11582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831676#comment-17831676
]
ASF GitHub Bot commented on YARN-11582:
---
hadoop-yetus commented on PR #6139:
URL: https://github.com/apache/hadoop/pull/6139#issuecomment-2024708052
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 21s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 24s | | trunk passed |
| +1 :green_heart: | compile | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 6s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 89m 20s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 24s | | The patch does not
generate ASF License warnings. |
| | | 172m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6139/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6139 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 64f3cc57e612 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / dca8ab0eade23a70756077c5e60ce865237cf340 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6139/13/testReport/ |
| Max. process+thread count | 948 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6139/13/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus
