[jira] [Updated] (YARN-11533) CapacityScheduler CapacityConfigType changed in legacy queue allocation mode
[
https://issues.apache.org/jira/browse/YARN-11533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YARN-11533:
--
Labels: pull-request-available (was: )
> CapacityScheduler CapacityConfigType changed in legacy queue allocation mode
>
>
> Key: YARN-11533
> URL: https://issues.apache.org/jira/browse/YARN-11533
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.4.0
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
>
> YARN-11000 changed the CapacityConfigType determination method in legacy
> queue mode, which has some undesired effects, like marking root to be in
> percentage mode, when the rest of the queue structure is in absolute.
> Config:
> {code:java}
>
>
> yarn.scheduler.capacity.root.maximum-capacity
> [memory=40960,vcores=32]
>
>
> yarn.scheduler.capacity.root.queues
> default
>
>
> yarn.scheduler.capacity.root.capacity
> [memory=40960,vcores=32]
>
>
>
> yarn.scheduler.capacity.schedule-asynchronously.enable
> true
>
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=40960,vcores=32]
>
>
>
> yarn.scheduler.capacity.root.default.maximum-am-resource-percent
> 0.2
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=40960,vcores=32]
>
>
> {code}
> Old response:
> {code:java}
> {
> "scheduler": {
> "schedulerInfo": {
> "type": "capacityScheduler",
> "capacity": 100.0,
> "usedCapacity": 0.0,
> "maxCapacity": 100.0,
> "weight": -1.0,
> "normalizedWeight": 0.0,
> "queueName": "root",
> "queuePath": "root",
> "maxParallelApps": 2147483647,
> "isAbsoluteResource": true,
> "queues": {...}
> "queuePriority": 0,
> "orderingPolicyInfo": "utilization",
> "mode": "absolute",
> "queueType": "parent",
> "creationMethod": "static",
> "autoCreationEligibility": "off",
> "autoQueueTemplateProperties": {},
> "autoQueueParentTemplateProperties": {},
> "autoQueueLeafTemplateProperties": {}
> }
> }
> }
> {code}
> New response:
> {code:java}
> {
> "scheduler": {
> "schedulerInfo": {
> "type": "capacityScheduler",
> "capacity": 100.0,
> "usedCapacity": 0.0,
> "maxCapacity": 100.0,
> "weight": -1.0,
> "normalizedWeight": 0.0,
> "queueName": "root",
> "queuePath": "root",
> "maxParallelApps": 2147483647,
> "isAbsoluteResource": false,
> "queues": {...}
> "queuePriority": 0,
> "orderingPolicyInfo": "utilization",
> "mode": "percentage",
> "queueType": "parent",
> "creationMethod": "static",
> "autoCreationEligibility": "off",
> "autoQueueTemplateProperties": {},
> "autoQueueParentTemplateProperties": {},
> "autoQueueLeafTemplateProperties": {}
> }
> }
> }
> {code}
> This is misleading and has some side-effects on various clients.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11533) CapacityScheduler CapacityConfigType changed in legacy queue allocation mode
[
https://issues.apache.org/jira/browse/YARN-11533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744003#comment-17744003
]
ASF GitHub Bot commented on YARN-11533:
---
hadoop-yetus commented on PR #5852:
URL: https://github.com/apache/hadoop/pull/5852#issuecomment-1639025948
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 56m 14s | | trunk passed |
| +1 :green_heart: | compile | 1m 4s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 0m 49s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 44s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5852/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 1 new + 25 unchanged - 0 fixed = 26 total (was 25) |
| +1 :green_heart: | mvnsite | 0m 52s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 56s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 104m 13s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 254m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5852/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5852 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4bbb2cd60778 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5490f9b6f28fa41d5964256f4e72cd1e4555242d |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Priva
[jira] [Commented] (YARN-11531) [Federation] Code cleanup for NodeManager#amrmproxy
[
https://issues.apache.org/jira/browse/YARN-11531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743993#comment-17743993
]
ASF GitHub Bot commented on YARN-11531:
---
goiri commented on code in PR #5841:
URL: https://github.com/apache/hadoop/pull/5841#discussion_r1265950558
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/AMRMProxyApplicationContextImpl.java:
##
@@ -53,7 +53,7 @@ public class AMRMProxyApplicationContextImpl implements
* @param nmContext NM context
* @param conf configuration
* @param applicationAttemptId attempt id
- * @param user user name of the application
+ * @param user username of the application
Review Comment:
I would leave this as is.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestFederationInterceptor.java:
##
@@ -268,7 +267,7 @@ private void releaseContainersAndAssert(List
containers)
// aggregated and returned back to us, and we can check if total request
size
// and returned size are the same
List containersForReleasedContainerIds =
-new ArrayList();
+new ArrayList<>();
Review Comment:
Indentation looks funny.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/FederationInterceptor.java:
##
@@ -1052,49 +1049,44 @@ protected void reAttachUAMAndMergeRegisterResponse(
final Token amrmToken = entry.getValue();
completionService
- .submit(new Callable() {
-@Override
-public RegisterApplicationMasterResponse call() throws Exception {
- RegisterApplicationMasterResponse response = null;
- try {
-// Create a config loaded with federation on and subclusterId
-// for each UAM
-YarnConfiguration config = new YarnConfiguration(getConf());
-FederationProxyProviderUtil.updateConfForFederation(config,
-subClusterId.getId());
-
-ApplicationSubmissionContext originalSubmissionContext =
-federationFacade.getApplicationSubmissionContext(appId);
-
-uamPool.reAttachUAM(subClusterId.getId(), config, appId,
-amRegistrationResponse.getQueue(),
-getApplicationContext().getUser(),
homeSubClusterId.getId(),
-amrmToken, subClusterId.toString(),
originalSubmissionContext);
-
-secondaryRelayers.put(subClusterId.getId(),
-uamPool.getAMRMClientRelayer(subClusterId.getId()));
-
-response = uamPool.registerApplicationMaster(
-subClusterId.getId(), amRegistrationRequest);
-
-// Set sub-cluster to be timed out initially
-lastSCResponseTime.put(subClusterId,
-clock.getTime() - subClusterTimeOut);
-
-if (response != null
-&& response.getContainersFromPreviousAttempts() != null) {
- cacheAllocatedContainers(
- response.getContainersFromPreviousAttempts(),
- subClusterId);
-}
-LOG.info("UAM {} reattached for {}", subClusterId, appId);
- } catch (Throwable e) {
-LOG.error(
-"Reattaching UAM " + subClusterId + " failed for " + appId,
-e);
+ .submit(() -> {
+RegisterApplicationMasterResponse response = null;
+try {
+ // Create a config loaded with federation on and subclusterId
+ // for each UAM
+ YarnConfiguration config = new YarnConfiguration(getConf());
+ FederationProxyProviderUtil.updateConfForFederation(config,
+ subClusterId.getId());
+
+ ApplicationSubmissionContext originalSubmissionContext =
+ federationFacade.getApplicationSubmissionContext(appId);
+
+ uamPool.reAttachUAM(subClusterId.getId(), config, appId,
+ amRegistrationResponse.getQueue(),
+ getApplicationContext().getUser(), homeSubClusterId.getId(),
+ amrmToken, subClusterId.toString(),
originalSubmissionContext);
+
+ secondaryRelayers.put(subClusterId.getId(),
+ uamPool.getAMRMClientRelayer(subClusterId.getId()));
+
+ response = uamPool.registerApplicationMaster(
+ subClusterId.getId(), amRegistrationRequest);
+
+ // Set sub-cluster to be timed out initially
+ lastSCResponseTime.put(
[jira] [Commented] (YARN-7707) [GPG] Policy generator framework
[
https://issues.apache.org/jira/browse/YARN-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743991#comment-17743991
]
ASF GitHub Bot commented on YARN-7707:
--
goiri commented on code in PR #5810:
URL: https://github.com/apache/hadoop/pull/5810#discussion_r1265947134
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml:
##
@@ -5426,4 +5426,36 @@
100ms
+
+
+ The interval at which the policy generator runs, default is one hour
+
+yarn.federation.gpg.policy.generator.interval-ms
Review Comment:
Not sure how we handle this with backports but as the metric is general
durations with units, it shouldn't include ms.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/GPGPolicyFacade.java:
##
@@ -0,0 +1,222 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.globalpolicygenerator;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.policies.FederationPolicyUtils;
+import
org.apache.hadoop.yarn.server.federation.policies.dao.WeightedPolicyInfo;
+import
org.apache.hadoop.yarn.server.federation.policies.manager.WeightedLocalityPolicyManager;
+import
org.apache.hadoop.yarn.server.federation.policies.router.FederationRouterPolicy;
+import
org.apache.hadoop.yarn.server.federation.policies.amrmproxy.FederationAMRMProxyPolicy;
+import
org.apache.hadoop.yarn.server.federation.policies.exceptions.FederationPolicyInitializationException;
+import
org.apache.hadoop.yarn.server.federation.policies.manager.FederationPolicyManager;
+import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterPolicyConfiguration;
+import
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * A utility class for the GPG Policy Generator to read and write policies
+ * into the FederationStateStore. Policy specific logic is abstracted away in
+ * this class, so the PolicyGenerator can avoid dealing with policy
+ * construction, reinitialization, and serialization.
+ *
+ * There are only two exposed methods:
+ *
+ * {@link #getPolicyManager(String)}
+ * Gets the PolicyManager via queue name. Null if there is no policy
+ * configured for the specified queue. The PolicyManager can be used to
+ * extract the {@link FederationRouterPolicy} and
+ * {@link FederationAMRMProxyPolicy}, as well as any policy specific parameters
+ *
+ * {@link #setPolicyManager(FederationPolicyManager)}
+ * Sets the PolicyManager. If the policy configuration is the same, no change
+ * occurs. Otherwise, the internal cache is updated and the new configuration
+ * is written into the FederationStateStore
+ *
+ * This class assumes that the GPG is the only service
+ * writing policies. Thus, the only FederationStateStore reads occur the first
+ * time a queue policy is retrieved - after that, the GPG only writes to the
+ * FederationStateStore.
+ *
+ * The class uses a PolicyManager cache and a SubClusterPolicyConfiguration
+ * cache. The primary use for these caches are to serve reads, and to
+ * identify when the PolicyGenerator has actually changed the policy
+ * so unnecessary FederationStateStore policy writes can be avoided.
+ */
+
+public class GPGPolicyFacade {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(GPGPolicyFacade.class);
+
+ private FederationStateStoreFacade stateStore;
+
+ private Map policyManagerMap;
+ private Map policyConfMap;
+
+ private boolean readOnly;
+
+ public GPGPolicyFacade(FederationStateStoreFacade stateStore,
+ Configuration conf) {
+this.stateStore = stateStore;
+this.policyManagerMap = new HashMap<>();
+this.policyConfMap = new HashMap<>();
+this.readOnly =
+conf.getBoolean(Yar
[jira] [Commented] (YARN-11491) [Federation] Use ZookeeperFederationStateStore as the DefaultStateStore
[
https://issues.apache.org/jira/browse/YARN-11491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743981#comment-17743981
]
ASF GitHub Bot commented on YARN-11491:
---
slfan1989 commented on code in PR #5637:
URL: https://github.com/apache/hadoop/pull/5637#discussion_r1265919772
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java:
##
@@ -4001,7 +4001,7 @@ public static boolean isAclEnabled(Configuration conf) {
FEDERATION_PREFIX + "state-store.class";
public static final String DEFAULT_FEDERATION_STATESTORE_CLIENT_CLASS =
-
"org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore";
+
"org.apache.hadoop.yarn.server.federation.store.impl.ZookeeperFederationStateStore";
Review Comment:
Thank you for helping to review the code! I will check the configuration in
xml.
> [Federation] Use ZookeeperFederationStateStore as the DefaultStateStore
> ---
>
> Key: YARN-11491
> URL: https://issues.apache.org/jira/browse/YARN-11491
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
>
> We currently use MemoryStateStore as the default StateStore, in the
> production environment, we should use ZookeeperFederationStateStore as the
> default StateStore.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7707) [GPG] Policy generator framework
[ https://issues.apache.org/jira/browse/YARN-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743978#comment-17743978 ] ASF GitHub Bot commented on YARN-7707: -- slfan1989 commented on PR #5747: URL: https://github.com/apache/hadoop/pull/5747#issuecomment-1638899425 @goiri Can you help to merge this pr into the trunk branch? Thank you very much! I will continue to follow up bakcport [YARN-8934](https://issues.apache.org/jira/browse/YARN-8934). The PR (#5810) has the same code, and it compiles successfully. > [GPG] Policy generator framework > > > Key: YARN-7707 > URL: https://issues.apache.org/jira/browse/YARN-7707 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Carlo Curino >Assignee: Young Chen >Priority: Major > Labels: federation, gpg, pull-request-available > Attachments: YARN-7707-YARN-7402.01.patch, > YARN-7707-YARN-7402.02.patch, YARN-7707-YARN-7402.03.patch, > YARN-7707-YARN-7402.04.patch, YARN-7707-YARN-7402.05.patch, > YARN-7707-YARN-7402.06.patch, YARN-7707-YARN-7402.07.patch, > YARN-7707-YARN-7402.08.patch, YARN-7707-YARN-7402.09.patch, > YARN-7707-YARN-7402.10.patch, YARN-7707-YARN-7402.11.patch > > > This JIRA tracks the development of a generic framework for querying > sub-clusters for metrics, running policies, and updating them in the > FederationStateStore. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9877) Intermittent TIME_OUT of LogAggregationReport
[
https://issues.apache.org/jira/browse/YARN-9877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743948#comment-17743948
]
ASF GitHub Bot commented on YARN-9877:
--
hadoop-yetus commented on PR #5784:
URL: https://github.com/apache/hadoop/pull/5784#issuecomment-1638750770
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 32s | | trunk passed |
| +1 :green_heart: | compile | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 59s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 5s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 0m 54s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 54s | | the patch passed |
| +1 :green_heart: | compile | 0m 48s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 45s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 51s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 32s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 100m 17s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 233m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5784/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5784 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ac26654a68a3 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fd10ab7097598543282bd35ea015aebc4268f5c9 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5784/3/testReport/ |
| Max. process+thread count | 993 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5784/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
[jira] [Commented] (YARN-11525) [Federation] Router CLI Supports Save the SubClusterPolicyConfiguration Of Queues.
[
https://issues.apache.org/jira/browse/YARN-11525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743943#comment-17743943
]
ASF GitHub Bot commented on YARN-11525:
---
hadoop-yetus commented on PR #5816:
URL: https://github.com/apache/hadoop/pull/5816#issuecomment-1638735062
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 18m 49s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 36m 34s | | trunk passed |
| +1 :green_heart: | compile | 7m 58s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 7m 19s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 2m 4s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 42s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 6m 34s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 46s | |
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 36m 8s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 36m 33s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 45s | | the patch passed |
| +1 :green_heart: | compile | 7m 8s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | cc | 7m 8s | | the patch passed |
| +1 :green_heart: | javac | 7m 8s | | the patch passed |
| +1 :green_heart: | compile | 7m 13s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | cc | 7m 13s | | the patch passed |
| +1 :green_heart: | javac | 7m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 47s | | the patch passed |
| +1 :green_heart: | mvnsite | 5m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 16s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 5m 23s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 41s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 36m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 13s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 44s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 42s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 105m 54s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 28m 51s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 0m 53s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | unit | 0m 35s | | hadoop-yarn-site in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 9s | | The patch does not
generate ASF License warnings. |
| | | 383m 54s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAP
[jira] [Updated] (YARN-11533) CapacityScheduler CapacityConfigType changed in legacy queue allocation mode
[
https://issues.apache.org/jira/browse/YARN-11533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Teke updated YARN-11533:
-
Parent: YARN-10888
Issue Type: Sub-task (was: Bug)
> CapacityScheduler CapacityConfigType changed in legacy queue allocation mode
>
>
> Key: YARN-11533
> URL: https://issues.apache.org/jira/browse/YARN-11533
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.4.0
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
>
> YARN-11000 changed the CapacityConfigType determination method in legacy
> queue mode, which has some undesired effects, like marking root to be in
> percentage mode, when the rest of the queue structure is in absolute.
> Config:
> {code:java}
>
>
> yarn.scheduler.capacity.root.maximum-capacity
> [memory=40960,vcores=32]
>
>
> yarn.scheduler.capacity.root.queues
> default
>
>
> yarn.scheduler.capacity.root.capacity
> [memory=40960,vcores=32]
>
>
>
> yarn.scheduler.capacity.schedule-asynchronously.enable
> true
>
>
>
> yarn.scheduler.capacity.root.default.maximum-capacity
> [memory=40960,vcores=32]
>
>
>
> yarn.scheduler.capacity.root.default.maximum-am-resource-percent
> 0.2
>
>
> yarn.scheduler.capacity.root.default.capacity
> [memory=40960,vcores=32]
>
>
> {code}
> Old response:
> {code:java}
> {
> "scheduler": {
> "schedulerInfo": {
> "type": "capacityScheduler",
> "capacity": 100.0,
> "usedCapacity": 0.0,
> "maxCapacity": 100.0,
> "weight": -1.0,
> "normalizedWeight": 0.0,
> "queueName": "root",
> "queuePath": "root",
> "maxParallelApps": 2147483647,
> "isAbsoluteResource": true,
> "queues": {...}
> "queuePriority": 0,
> "orderingPolicyInfo": "utilization",
> "mode": "absolute",
> "queueType": "parent",
> "creationMethod": "static",
> "autoCreationEligibility": "off",
> "autoQueueTemplateProperties": {},
> "autoQueueParentTemplateProperties": {},
> "autoQueueLeafTemplateProperties": {}
> }
> }
> }
> {code}
> New response:
> {code:java}
> {
> "scheduler": {
> "schedulerInfo": {
> "type": "capacityScheduler",
> "capacity": 100.0,
> "usedCapacity": 0.0,
> "maxCapacity": 100.0,
> "weight": -1.0,
> "normalizedWeight": 0.0,
> "queueName": "root",
> "queuePath": "root",
> "maxParallelApps": 2147483647,
> "isAbsoluteResource": false,
> "queues": {...}
> "queuePriority": 0,
> "orderingPolicyInfo": "utilization",
> "mode": "percentage",
> "queueType": "parent",
> "creationMethod": "static",
> "autoCreationEligibility": "off",
> "autoQueueTemplateProperties": {},
> "autoQueueParentTemplateProperties": {},
> "autoQueueLeafTemplateProperties": {}
> }
> }
> }
> {code}
> This is misleading and has some side-effects on various clients.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11533) CapacityScheduler CapacityConfigType changed in legacy queue allocation mode
Benjamin Teke created YARN-11533:
Summary: CapacityScheduler CapacityConfigType changed in legacy
queue allocation mode
Key: YARN-11533
URL: https://issues.apache.org/jira/browse/YARN-11533
Project: Hadoop YARN
Issue Type: Bug
Components: capacity scheduler
Affects Versions: 3.4.0
Reporter: Benjamin Teke
Assignee: Benjamin Teke
YARN-11000 changed the CapacityConfigType determination method in legacy queue
mode, which has some undesired effects, like marking root to be in percentage
mode, when the rest of the queue structure is in absolute.
Config:
{code:java}
yarn.scheduler.capacity.root.maximum-capacity
[memory=40960,vcores=32]
yarn.scheduler.capacity.root.queues
default
yarn.scheduler.capacity.root.capacity
[memory=40960,vcores=32]
yarn.scheduler.capacity.schedule-asynchronously.enable
true
yarn.scheduler.capacity.root.default.maximum-capacity
[memory=40960,vcores=32]
yarn.scheduler.capacity.root.default.maximum-am-resource-percent
0.2
yarn.scheduler.capacity.root.default.capacity
[memory=40960,vcores=32]
{code}
Old response:
{code:java}
{
"scheduler": {
"schedulerInfo": {
"type": "capacityScheduler",
"capacity": 100.0,
"usedCapacity": 0.0,
"maxCapacity": 100.0,
"weight": -1.0,
"normalizedWeight": 0.0,
"queueName": "root",
"queuePath": "root",
"maxParallelApps": 2147483647,
"isAbsoluteResource": true,
"queues": {...}
"queuePriority": 0,
"orderingPolicyInfo": "utilization",
"mode": "absolute",
"queueType": "parent",
"creationMethod": "static",
"autoCreationEligibility": "off",
"autoQueueTemplateProperties": {},
"autoQueueParentTemplateProperties": {},
"autoQueueLeafTemplateProperties": {}
}
}
}
{code}
New response:
{code:java}
{
"scheduler": {
"schedulerInfo": {
"type": "capacityScheduler",
"capacity": 100.0,
"usedCapacity": 0.0,
"maxCapacity": 100.0,
"weight": -1.0,
"normalizedWeight": 0.0,
"queueName": "root",
"queuePath": "root",
"maxParallelApps": 2147483647,
"isAbsoluteResource": false,
"queues": {...}
"queuePriority": 0,
"orderingPolicyInfo": "utilization",
"mode": "percentage",
"queueType": "parent",
"creationMethod": "static",
"autoCreationEligibility": "off",
"autoQueueTemplateProperties": {},
"autoQueueParentTemplateProperties": {},
"autoQueueLeafTemplateProperties": {}
}
}
}
{code}
This is misleading and has some side-effects on various clients.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] (YARN-10301) "DIGEST-MD5: digest response format violation. Mismatched response." when network partition occurs
[ https://issues.apache.org/jira/browse/YARN-10301 ]
walhl.liu deleted comment on YARN-10301:
--
was (Author: walhl):
Generally, RM issue NM master key, NM take back master key and save to memory
when registering RM. RM will carry master when invoke nm rpc api to stop
container. nm will compare master key, nm will throw DIGEST-MD5 error if
master is not consistent. If NM register RM timely, NM will compare old
master key with new master key carried by rm, so nm will encounter in
consistent.
> "DIGEST-MD5: digest response format violation. Mismatched response." when
> network partition occurs
> --
>
> Key: YARN-10301
> URL: https://issues.apache.org/jira/browse/YARN-10301
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: YCozy
>Priority: Major
>
> We observed the "Mismatched response." error in RM's log when a NM gets
> network-partitioned after RM failover. Here's how it happens:
>
> Initially, we have a sleeper YARN service running in a cluster with two RMs
> (an active RM1 and a standby RM2) and one NM. At some point, we perform a RM
> failover from RM1 to RM2.
> RM1's log:
> {noformat}
> 2020-06-01 16:29:20,387 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned
> to standby state{noformat}
> RM2's log:
> {noformat}
> 2020-06-01 16:29:27,818 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned
> to active state{noformat}
>
> After the RM failover, the NM encounters a network partition and fails to
> register with RM2. In other words, there's no "NodeManager from node ***
> registered" in RM2's log.
>
> This does not affect the sleeper YARN service. The sleeper service
> successfully recovers after the RM failover. We can see in RM2's log:
> {noformat}
> 2020-06-01 16:30:06,703 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> appattempt_6_0001_01 State change from LAUNCHED to RUNNING on event =
> REGISTERED{noformat}
>
> Then, we stop the sleeper service. In RM2's log, we can see that:
> {noformat}
> 2020-06-01 16:30:12,157 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> application_6_0001 unregistered successfully.
> ...
> 2020-06-01 16:31:09,861 INFO org.apache.hadoop.yarn.service.webapp.ApiServer:
> Successfully stopped service sleeper1{noformat}
> And in AM's log, we can see that:
> {noformat}
> 2020-06-01 16:30:12,651 [shutdown-hook-0] INFO service.ServiceMaster -
> SHUTDOWN_MSG:{noformat}
>
> Some time later, we observe the "Mismatched response" in RM2's log:
> {noformat}
> 2020-06-01 16:43:20,699 WARN org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:376)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623)
> at org.apache.hadoop.ipc.Client$Connection.access$2400(Client.java:414)
>
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:827)
>
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:823)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:823)
>
> at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:414)
>
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1667)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1483)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1436)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy102.stopContainers(Unknown Source)
>
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.stopContainers(ContainerManagementProtocolPBClientImpl.java:147)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invok
[jira] [Commented] (YARN-10301) "DIGEST-MD5: digest response format violation. Mismatched response." when network partition occurs
[
https://issues.apache.org/jira/browse/YARN-10301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743834#comment-17743834
]
walhl.liu commented on YARN-10301:
--
Generally, RM issue NM master key, NM take back master key and save to memory
when registering RM. RM will carry master when invoke nm rpc api to stop
container. nm will compare master key, nm will throw DIGEST-MD5 error if
master is not consistent. If NM register RM timely, NM will compare old
master key with new master key carried by rm, so nm will encounter in
consistent.
> "DIGEST-MD5: digest response format violation. Mismatched response." when
> network partition occurs
> --
>
> Key: YARN-10301
> URL: https://issues.apache.org/jira/browse/YARN-10301
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: YCozy
>Priority: Major
>
> We observed the "Mismatched response." error in RM's log when a NM gets
> network-partitioned after RM failover. Here's how it happens:
>
> Initially, we have a sleeper YARN service running in a cluster with two RMs
> (an active RM1 and a standby RM2) and one NM. At some point, we perform a RM
> failover from RM1 to RM2.
> RM1's log:
> {noformat}
> 2020-06-01 16:29:20,387 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned
> to standby state{noformat}
> RM2's log:
> {noformat}
> 2020-06-01 16:29:27,818 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned
> to active state{noformat}
>
> After the RM failover, the NM encounters a network partition and fails to
> register with RM2. In other words, there's no "NodeManager from node ***
> registered" in RM2's log.
>
> This does not affect the sleeper YARN service. The sleeper service
> successfully recovers after the RM failover. We can see in RM2's log:
> {noformat}
> 2020-06-01 16:30:06,703 INFO
> org.apache.hadoop.yarn.server.resourcemanager.rmapp.attempt.RMAppAttemptImpl:
> appattempt_6_0001_01 State change from LAUNCHED to RUNNING on event =
> REGISTERED{noformat}
>
> Then, we stop the sleeper service. In RM2's log, we can see that:
> {noformat}
> 2020-06-01 16:30:12,157 INFO
> org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService:
> application_6_0001 unregistered successfully.
> ...
> 2020-06-01 16:31:09,861 INFO org.apache.hadoop.yarn.service.webapp.ApiServer:
> Successfully stopped service sleeper1{noformat}
> And in AM's log, we can see that:
> {noformat}
> 2020-06-01 16:30:12,651 [shutdown-hook-0] INFO service.ServiceMaster -
> SHUTDOWN_MSG:{noformat}
>
> Some time later, we observe the "Mismatched response" in RM2's log:
> {noformat}
> 2020-06-01 16:43:20,699 WARN org.apache.hadoop.ipc.Client: Exception
> encountered while connecting to the server
> org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException):
> DIGEST-MD5: digest response format violation. Mismatched response.
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:376)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623)
> at org.apache.hadoop.ipc.Client$Connection.access$2400(Client.java:414)
>
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:827)
>
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:823)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at javax.security.auth.Subject.doAs(Subject.java:422)
>
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1845)
> at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:823)
>
> at org.apache.hadoop.ipc.Client$Connection.access$3800(Client.java:414)
>
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1667)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1483)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:1436)
>
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:233)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:118)
> at com.sun.proxy.$Proxy102.stopContainers(Unknown Source)
>
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.stopContainers(ContainerManagementProtocolPBClientImpl.java:147)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.in
[jira] [Commented] (YARN-11525) [Federation] Router CLI Supports Save the SubClusterPolicyConfiguration Of Queues.
[
https://issues.apache.org/jira/browse/YARN-11525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743763#comment-17743763
]
ASF GitHub Bot commented on YARN-11525:
---
hadoop-yetus commented on PR #5816:
URL: https://github.com/apache/hadoop/pull/5816#issuecomment-1637924903
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 9s | | trunk passed |
| +1 :green_heart: | compile | 7m 48s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 7m 14s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 2m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 54s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 6m 34s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 46s | |
branch/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 34m 19s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 47s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 55s | | the patch passed |
| +1 :green_heart: | compile | 7m 0s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | cc | 6m 59s | | the patch passed |
| +1 :green_heart: | javac | 6m 59s | | the patch passed |
| +1 :green_heart: | compile | 7m 14s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | cc | 7m 14s | | the patch passed |
| +1 :green_heart: | javac | 7m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 51s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5816/18/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 65 unchanged -
0 fixed = 66 total (was 65) |
| +1 :green_heart: | mvnsite | 6m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 6m 3s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 5m 56s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 40s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site has no data from spotbugs |
| +1 :green_heart: | shadedclient | 34m 28s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 17s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 56s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 41s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 101m 3s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 28m 51s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 0m 52s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | unit | 0m 45s | | hadoop-ya
[jira] [Commented] (YARN-7707) [GPG] Policy generator framework
[
https://issues.apache.org/jira/browse/YARN-7707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743681#comment-17743681
]
ASF GitHub Bot commented on YARN-7707:
--
hadoop-yetus commented on PR #5810:
URL: https://github.com/apache/hadoop/pull/5810#issuecomment-1637626654
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 21s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 31m 25s | | trunk passed |
| +1 :green_heart: | compile | 7m 46s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 7m 18s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 2m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 3s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 51s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 6m 50s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 27s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 12s | | the patch passed |
| +1 :green_heart: | compile | 7m 3s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 7m 3s | | the patch passed |
| +1 :green_heart: | compile | 7m 12s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 7m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 56s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 33s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 3m 26s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 7m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 18s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 56s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 42s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 1m 2s | |
hadoop-yarn-server-globalpolicygenerator in the patch passed. |
| +1 :green_heart: | asflicense | 1m 6s | | The patch does not
generate ASF License warnings. |
| | | 210m 19s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5810/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5810 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint
jsonlint |
| uname | Linux 1d08ae8947e4 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0d8944f95a43258ca4afa479e9d497e08694db87 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
[jira] [Commented] (YARN-11531) [Federation] Code cleanup for NodeManager#amrmproxy
[
https://issues.apache.org/jira/browse/YARN-11531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17743660#comment-17743660
]
ASF GitHub Bot commented on YARN-11531:
---
hadoop-yetus commented on PR #5841:
URL: https://github.com/apache/hadoop/pull/5841#issuecomment-1637556563
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 45s | | trunk passed |
| +1 :green_heart: | compile | 1m 31s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 0m 44s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 51s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 37m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 1m 25s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 1m 25s | | the patch passed |
| +1 :green_heart: | compile | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 1m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 28s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5841/2/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
The patch generated 4 new + 29 unchanged - 1 fixed = 33 total (was 30) |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 25m 2s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 162m 21s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5841/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5841 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4a89b6aff3d5 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3d2b4971dc735d7a26b389671cc42e244f95b61e |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5841/2/
