[jira] [Commented] (YARN-11425) [Federation] Router Supports SubClusterCleaner
[
https://issues.apache.org/jira/browse/YARN-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681191#comment-17681191
]
ASF GitHub Bot commented on YARN-11425:
---
slfan1989 commented on code in PR #5326:
URL: https://github.com/apache/hadoop/pull/5326#discussion_r1088614242
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/cleaner/TestSubClusterCleaner.java:
##
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.cleaner;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import
org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Map;
+
+public class TestSubClusterCleaner {
+
+
+ // Router Constants
+
+ private Configuration conf;
+ private MemoryFederationStateStore stateStore;
+ private FederationStateStoreFacade facade;
+ private SubClusterCleaner cleaner;
+
+ @Before
+ public void setup() throws YarnException {
+conf = new YarnConfiguration();
+conf.setLong(YarnConfiguration.ROUTER_SUBCLUSTER_EXPIRATION_TIME, 1000);
+conf.setInt(YarnConfiguration.FEDERATION_CACHE_TIME_TO_LIVE_SECS, 0);
+
+stateStore = new MemoryFederationStateStore();
+stateStore.init(conf);
+
+facade = FederationStateStoreFacade.getInstance();
+facade.reinitialize(stateStore, conf);
+
+cleaner = new SubClusterCleaner(conf);
+for (int i = 0; i < 4; i++){
+ // Create sub cluster id and info
+ SubClusterId subClusterId = SubClusterId.newInstance("SC-" + i);
+ SubClusterInfo subClusterInfo = SubClusterInfo.newInstance(subClusterId,
+ "127.0.0.1:1", "127.0.0.1:2", "127.0.0.1:3", "127.0.0.1:4",
+ SubClusterState.SC_RUNNING, System.currentTimeMillis(), "");
+ // Register the subCluster
+ stateStore.registerSubCluster(
+ SubClusterRegisterRequest.newInstance(subClusterInfo));
+}
+ }
+
+ @Test
+ public void testSubClusterRegisterHeartBeatTime() throws YarnException,
InterruptedException {
+Thread.sleep(1000);
Review Comment:
I will continue to improve this unit test and supplement the description
information of the unit test.
> [Federation] Router Supports SubClusterCleaner
> --
>
> Key: YARN-11425
> URL: https://issues.apache.org/jira/browse/YARN-11425
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
>
> In YARN-Federation mode, once a SubCluster is registered, the SubCluster is
> always in the RUNNING state, even if the SubCluster has no heartbeat for a
> long time.
> We will let the Router automatically check the heartbeat time of the
> SubCluster, and once the heartbeat exceeds 30mins, we will set the SubCluster
> to the SC_LOST state.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11425) [Federation] Router Supports SubClusterCleaner
[
https://issues.apache.org/jira/browse/YARN-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681190#comment-17681190
]
ASF GitHub Bot commented on YARN-11425:
---
slfan1989 commented on code in PR #5326:
URL: https://github.com/apache/hadoop/pull/5326#discussion_r1088613766
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/cleaner/TestSubClusterCleaner.java:
##
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.cleaner;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import
org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Map;
+
+public class TestSubClusterCleaner {
+
+
+ // Router Constants
+
+ private Configuration conf;
+ private MemoryFederationStateStore stateStore;
+ private FederationStateStoreFacade facade;
+ private SubClusterCleaner cleaner;
+
+ @Before
+ public void setup() throws YarnException {
+conf = new YarnConfiguration();
+conf.setLong(YarnConfiguration.ROUTER_SUBCLUSTER_EXPIRATION_TIME, 1000);
+conf.setInt(YarnConfiguration.FEDERATION_CACHE_TIME_TO_LIVE_SECS, 0);
+
+stateStore = new MemoryFederationStateStore();
+stateStore.init(conf);
+
+facade = FederationStateStoreFacade.getInstance();
+facade.reinitialize(stateStore, conf);
+
+cleaner = new SubClusterCleaner(conf);
+for (int i = 0; i < 4; i++){
+ // Create sub cluster id and info
+ SubClusterId subClusterId = SubClusterId.newInstance("SC-" + i);
+ SubClusterInfo subClusterInfo = SubClusterInfo.newInstance(subClusterId,
+ "127.0.0.1:1", "127.0.0.1:2", "127.0.0.1:3", "127.0.0.1:4",
+ SubClusterState.SC_RUNNING, System.currentTimeMillis(), "");
+ // Register the subCluster
+ stateStore.registerSubCluster(
+ SubClusterRegisterRequest.newInstance(subClusterInfo));
+}
+ }
+
+ @Test
+ public void testSubClusterRegisterHeartBeatTime() throws YarnException,
InterruptedException {
+Thread.sleep(1000);
Review Comment:
Thank you very much for helping to review the code!
For this unit test, we build a cluster with 4 subClusters whose initial
state is `RUNNING`.
TestSubClusterCleaner#setup#Line 58
```
for (int i = 0; i < 4; i++){
// Create sub cluster id and info
SubClusterId subClusterId = SubClusterId.newInstance("SC-" + i);
SubClusterInfo subClusterInfo =
SubClusterInfo.newInstance(subClusterId,
"127.0.0.1:1", "127.0.0.1:2", "127.0.0.1:3", "127.0.0.1:4",
SubClusterState.SC_RUNNING, System.currentTimeMillis(), "");
// Register the subCluster
stateStore.registerSubCluster(
SubClusterRegisterRequest.newInstance(subClusterInfo));
}
```
We set a timeout of 1 second
TestSubClusterCleaner#setup#Line 48
```
conf = new YarnConfiguration();
conf.setLong(YarnConfiguration.ROUTER_SUBCLUSTER_EXPIRATION_TIME, 1000);
```
In this unit test, we first sleep for `1s`, so that all subclusters will
timeout, we execute `SubClusterCleaner#run`, it will help set all subclusters
to `SC_LOST`.
> [Federation] Router Supports SubClusterCleaner
> --
>
> Key: YARN-11425
> URL: https://issues.apache.org/jira/browse/YARN-11425
>
[jira] [Commented] (YARN-11218) [Federation] Add getActivities, getBulkActivities REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681189#comment-17681189 ] ASF GitHub Bot commented on YARN-11218: --- slfan1989 commented on PR #5284: URL: https://github.com/apache/hadoop/pull/5284#issuecomment-1406054688 @goiri Thank you very much for helping to review the code! > [Federation] Add getActivities, getBulkActivities REST APIs for Router > -- > > Key: YARN-11218 > URL: https://issues.apache.org/jira/browse/YARN-11218 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0, 3.3.4 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11000) Replace queue resource calculation logic in updateClusterResource
[
https://issues.apache.org/jira/browse/YARN-11000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681142#comment-17681142
]
ASF GitHub Bot commented on YARN-11000:
---
hadoop-yetus commented on PR #3618:
URL: https://github.com/apache/hadoop/pull/3618#issuecomment-1405735022
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 10 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 0s | | trunk passed |
| +1 :green_heart: | compile | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 3s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 6s | | trunk passed |
| -1 :x: | javadoc | 0m 59s |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3618/6/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 0m 45s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 28s | | trunk passed |
| -1 :x: | shadedclient | 26m 8s | | branch has errors when building
and testing our client artifacts. |
| -0 :warning: | patch | 26m 30s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 1m 13s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 13s | | the patch passed |
| -1 :x: | compile | 0m 38s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3618/6/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | javac | 0m 38s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3618/6/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 10s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3618/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 29 new + 775 unchanged - 2 fixed = 804 total (was 777) |
| +1 :green_heart: | mvnsite | 1m 6s | | the patch passed |
| -1 :x: | javadoc | 0m 51s |
[jira] [Commented] (YARN-11000) Replace queue resource calculation logic in updateClusterResource
[
https://issues.apache.org/jira/browse/YARN-11000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681130#comment-17681130
]
ASF GitHub Bot commented on YARN-11000:
---
hadoop-yetus commented on PR #3618:
URL: https://github.com/apache/hadoop/pull/3618#issuecomment-1405695665
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 10 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 55m 35s | | trunk passed |
| +1 :green_heart: | compile | 1m 51s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 35s | | trunk passed |
| -1 :x: | javadoc | 1m 14s |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3618/5/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 5s | | trunk passed |
| -1 :x: | shadedclient | 15m 40s | | branch has errors when building
and testing our client artifacts. |
| -0 :warning: | patch | 16m 13s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 24s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3618/5/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 0m 22s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3618/5/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | javac | 0m 22s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3618/5/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | compile | 0m 23s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3618/5/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | javac | 0m 23s |
[jira] [Resolved] (YARN-11218) [Federation] Add getActivities, getBulkActivities REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved YARN-11218. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > [Federation] Add getActivities, getBulkActivities REST APIs for Router > -- > > Key: YARN-11218 > URL: https://issues.apache.org/jira/browse/YARN-11218 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0, 3.3.4 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11218) [Federation] Add getActivities, getBulkActivities REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681093#comment-17681093 ] ASF GitHub Bot commented on YARN-11218: --- goiri merged PR #5284: URL: https://github.com/apache/hadoop/pull/5284 > [Federation] Add getActivities, getBulkActivities REST APIs for Router > -- > > Key: YARN-11218 > URL: https://issues.apache.org/jira/browse/YARN-11218 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0, 3.3.4 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11425) [Federation] Router Supports SubClusterCleaner
[
https://issues.apache.org/jira/browse/YARN-11425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681092#comment-17681092
]
ASF GitHub Bot commented on YARN-11425:
---
goiri commented on code in PR #5326:
URL: https://github.com/apache/hadoop/pull/5326#discussion_r1088242768
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/cleaner/TestSubClusterCleaner.java:
##
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.cleaner;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import
org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Map;
+
+public class TestSubClusterCleaner {
+
+
+ // Router Constants
+
+ private Configuration conf;
+ private MemoryFederationStateStore stateStore;
+ private FederationStateStoreFacade facade;
+ private SubClusterCleaner cleaner;
+
+ @Before
+ public void setup() throws YarnException {
+conf = new YarnConfiguration();
+conf.setLong(YarnConfiguration.ROUTER_SUBCLUSTER_EXPIRATION_TIME, 1000);
+conf.setInt(YarnConfiguration.FEDERATION_CACHE_TIME_TO_LIVE_SECS, 0);
+
+stateStore = new MemoryFederationStateStore();
+stateStore.init(conf);
+
+facade = FederationStateStoreFacade.getInstance();
+facade.reinitialize(stateStore, conf);
+
+cleaner = new SubClusterCleaner(conf);
+for (int i = 0; i < 4; i++){
+ // Create sub cluster id and info
+ SubClusterId subClusterId = SubClusterId.newInstance("SC-" + i);
+ SubClusterInfo subClusterInfo = SubClusterInfo.newInstance(subClusterId,
+ "127.0.0.1:1", "127.0.0.1:2", "127.0.0.1:3", "127.0.0.1:4",
+ SubClusterState.SC_RUNNING, System.currentTimeMillis(), "");
+ // Register the subCluster
+ stateStore.registerSubCluster(
+ SubClusterRegisterRequest.newInstance(subClusterInfo));
+}
+ }
+
+ @Test
+ public void testSubClusterRegisterHeartBeatTime() throws YarnException,
InterruptedException {
+Thread.sleep(1000);
Review Comment:
Why do we need to wait here?
> [Federation] Router Supports SubClusterCleaner
> --
>
> Key: YARN-11425
> URL: https://issues.apache.org/jira/browse/YARN-11425
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
>
> In YARN-Federation mode, once a SubCluster is registered, the SubCluster is
> always in the RUNNING state, even if the SubCluster has no heartbeat for a
> long time.
> We will let the Router automatically check the heartbeat time of the
> SubCluster, and once the heartbeat exceeds 30mins, we will set the SubCluster
> to the SC_LOST state.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10965) Centralize queue resource calculation based on CapacityVectors
[ https://issues.apache.org/jira/browse/YARN-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681078#comment-17681078 ] ASF GitHub Bot commented on YARN-10965: --- szilard-nemeth closed pull request #3470: YARN-10965. Centralize queue resource calculation based on CapacityVectors URL: https://github.com/apache/hadoop/pull/3470 > Centralize queue resource calculation based on CapacityVectors > -- > > Key: YARN-10965 > URL: https://issues.apache.org/jira/browse/YARN-10965 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > > With the introduction of YARN-10930 it is possible to unify queue resource > calculation. In order to narrow down the scope of this patch, the base system > is implemented here, without refactoring the existing resource calculation in > updateClusterResource (which will be done in YARN-11000). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10965) Centralize queue resource calculation based on CapacityVectors
[ https://issues.apache.org/jira/browse/YARN-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681077#comment-17681077 ] ASF GitHub Bot commented on YARN-10965: --- szilard-nemeth commented on PR #3470: URL: https://github.com/apache/hadoop/pull/3470#issuecomment-1405447433 Thanks @9uapaw for this huge contribution. Latest patch LGTM, committed to trunk. Thanks @brumi1024 , @shuzirra for the reviews. > Centralize queue resource calculation based on CapacityVectors > -- > > Key: YARN-10965 > URL: https://issues.apache.org/jira/browse/YARN-10965 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > > With the introduction of YARN-10930 it is possible to unify queue resource > calculation. In order to narrow down the scope of this patch, the base system > is implemented here, without refactoring the existing resource calculation in > updateClusterResource (which will be done in YARN-11000). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10965) Centralize queue resource calculation based on CapacityVectors
[ https://issues.apache.org/jira/browse/YARN-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth resolved YARN-10965. --- Hadoop Flags: Reviewed Resolution: Fixed > Centralize queue resource calculation based on CapacityVectors > -- > > Key: YARN-10965 > URL: https://issues.apache.org/jira/browse/YARN-10965 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > > With the introduction of YARN-10930 it is possible to unify queue resource > calculation. In order to narrow down the scope of this patch, the base system > is implemented here, without refactoring the existing resource calculation in > updateClusterResource (which will be done in YARN-11000). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10965) Centralize queue resource calculation based on CapacityVectors
[ https://issues.apache.org/jira/browse/YARN-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10965: -- Fix Version/s: 3.4.0 > Centralize queue resource calculation based on CapacityVectors > -- > > Key: YARN-10965 > URL: https://issues.apache.org/jira/browse/YARN-10965 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 6.5h > Remaining Estimate: 0h > > With the introduction of YARN-10930 it is possible to unify queue resource > calculation. In order to narrow down the scope of this patch, the base system > is implemented here, without refactoring the existing resource calculation in > updateClusterResource (which will be done in YARN-11000). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10965) Centralize queue resource calculation based on CapacityVectors
[
https://issues.apache.org/jira/browse/YARN-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681072#comment-17681072
]
ASF GitHub Bot commented on YARN-10965:
---
hadoop-yetus commented on PR #3470:
URL: https://github.com/apache/hadoop/pull/3470#issuecomment-1405414803
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 8 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 20s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 16s | | trunk passed |
| +1 :green_heart: | compile | 10m 20s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 8m 29s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 18s | | trunk passed |
| -1 :x: | javadoc | 1m 1s |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3470/19/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 34s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 56s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 43s | | the patch passed |
| +1 :green_heart: | compile | 9m 4s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| -1 :x: | javac | 9m 4s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3470/19/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
|
hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 2 new + 719
unchanged - 3 fixed = 721 total (was 722) |
| +1 :green_heart: | compile | 8m 26s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 8m 26s | |
hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08
with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 0 new + 637
unchanged - 1 fixed = 637 total (was 638) |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3470/19/artifact/out/blanks-eol.txt)
| The patch has 1 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 39s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3470/19/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 25 new + 150 unchanged
- 0 fixed = 175 total (was 150) |
| +1 :green_heart: | mvnsite | 2m 6s | | the patch passed |
| -1 :x: | javadoc | 0m 56s |
[jira] [Commented] (YARN-11416) FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
[ https://issues.apache.org/jira/browse/YARN-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17681020#comment-17681020 ] ASF GitHub Bot commented on YARN-11416: --- brumi1024 commented on PR #5320: URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1405165777 > > I added some comments where I saw further possibilities for using the convenience get methods beside the fixed ones. I also saw that there are some config settings in the FsQueueConverter class where the CapacitySchedulerConfiguration’s setter methods could be used too, so that way we would have only one place for setting these values. > > For example here we can use the CapacitySchedulerConfiguration’s setMaximumCapacity() method: > > https://github.com/apache/hadoop/blob/a38eb1f19318e3dd34d9312afe1096db46652f49/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java#L167 > > > > What are your thoughts on that @brumi1024, @susheel-gupta? > > I think the setters can be handled in another separate jira. What do you suggest @p-szucs @brumi1024 ? I agree with @p-szucs, it strongly connects to this PR, so it be best to do it in one go. > FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder > --- > > Key: YARN-11416 > URL: https://issues.apache.org/jira/browse/YARN-11416 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Benjamin Teke >Assignee: Susheel Gupta >Priority: Major > Labels: pull-request-available > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSQueueConverter > and its builder stores the variable capacitySchedulerConfig as a simple > Configuration object instead of CapacitySchedulerConfiguration. This is > misleading, as capacitySchedulerConfig suggests that it is indeed a > CapacitySchedulerConfiguration and it loses access to the convenience methods > to check for various properties. Because of this every time a property getter > is changed FS2CS should be checked if it reimplemented the same, otherwise > there might be behaviour differences or even bugs. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10379) Refactor ContainerExecutor exit code Exception handling
[ https://issues.apache.org/jira/browse/YARN-10379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferenc Erdelyi reassigned YARN-10379: - Assignee: Ferenc Erdelyi (was: Benjamin Teke) > Refactor ContainerExecutor exit code Exception handling > --- > > Key: YARN-10379 > URL: https://issues.apache.org/jira/browse/YARN-10379 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Reporter: Benjamin Teke >Assignee: Ferenc Erdelyi >Priority: Minor > > **Currently every time a shell command is executed and returns with a > non-zero exitcode an exception gets thrown. But along the call tree this > exception gets catched, after some info/warn logging and other processing > steps rethrown, possibly packaged to another exception. For example: > * from PrivilegedOperationExecutor.executePrivilegedOperation - > ExitCodeException catch (as IOException), PrivilegedOperationException thrown > * then in LinuxContainerExecutor.startLocalizer - > PrivilegedOperationException catch, exitCode collection, logging, IOException > rethrown > * then in ResourceLocalizationService.run - generic Exception catch, but > there is a TODO for separate ExitCodeException handling, however that > information is only present here in an error message string > This flow could be simplified and unified in the different executors. For > example use one specific exception till the last possible step, catch it only > where it is necessary and keep the exitcode as it could be used later in the > process. This change could help with maintainability and readability. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11416) FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
[ https://issues.apache.org/jira/browse/YARN-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680999#comment-17680999 ] ASF GitHub Bot commented on YARN-11416: --- susheel-gupta commented on PR #5320: URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1405059886 > I added some comments where I saw further possibilities for using the convenience get methods beside the fixed ones. I also saw that there are some config settings in the FsQueueConverter class where the CapacitySchedulerConfiguration’s setter methods could be used too, so that way we would have only one place for setting these values. > > For example here we can use the CapacitySchedulerConfiguration’s setMaximumCapacity() method: > > https://github.com/apache/hadoop/blob/a38eb1f19318e3dd34d9312afe1096db46652f49/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java#L167 > > What are your thoughts on that @brumi1024, @susheel-gupta? I think the setters can be handled in another separate jira. What do you suggest @p-szucs @brumi1024 ? > FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder > --- > > Key: YARN-11416 > URL: https://issues.apache.org/jira/browse/YARN-11416 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Benjamin Teke >Assignee: Susheel Gupta >Priority: Major > Labels: pull-request-available > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSQueueConverter > and its builder stores the variable capacitySchedulerConfig as a simple > Configuration object instead of CapacitySchedulerConfiguration. This is > misleading, as capacitySchedulerConfig suggests that it is indeed a > CapacitySchedulerConfiguration and it loses access to the convenience methods > to check for various properties. Because of this every time a property getter > is changed FS2CS should be checked if it reimplemented the same, otherwise > there might be behaviour differences or even bugs. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10965) Centralize queue resource calculation based on CapacityVectors
[
https://issues.apache.org/jira/browse/YARN-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680998#comment-17680998
]
ASF GitHub Bot commented on YARN-10965:
---
szilard-nemeth commented on PR #3470:
URL: https://github.com/apache/hadoop/pull/3470#issuecomment-1405041912
Added commit that fixes checkstyle issues.
Except some of those fall into these types:
- Variable 'configuredMaxCapacityVectors' must be private and have accessor
methods. [VisibilityModifier]
- Redundant 'public' modifier. [RedundantModifier]
- QueueUpdateWarning.java:48:public QueueUpdateWarning ofQueue(String
queue) {:46: 'queue' hides a field. [HiddenField]
LGTM +1, waiting for Jenkins
> Centralize queue resource calculation based on CapacityVectors
> --
>
> Key: YARN-10965
> URL: https://issues.apache.org/jira/browse/YARN-10965
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> With the introduction of YARN-10930 it is possible to unify queue resource
> calculation. In order to narrow down the scope of this patch, the base system
> is implemented here, without refactoring the existing resource calculation in
> updateClusterResource (which will be done in YARN-11000).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10965) Centralize queue resource calculation based on CapacityVectors
[
https://issues.apache.org/jira/browse/YARN-10965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680997#comment-17680997
]
ASF GitHub Bot commented on YARN-10965:
---
szilard-nemeth commented on code in PR #3470:
URL: https://github.com/apache/hadoop/pull/3470#discussion_r1087885157
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueManager.java:
##
@@ -100,6 +101,7 @@ public CapacitySchedulerQueueManager(Configuration conf,
this.queueStateManager = new QueueStateManager<>();
this.appPriorityACLManager = appPriorityACLManager;
this.configuredNodeLabels = new ConfiguredNodeLabels();
+this.queueCapacityHandler = new
CapacitySchedulerQueueCapacityHandler(labelManager);
Review Comment:
Thanks, got it
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerQueueCapacityHandler.java:
##
@@ -0,0 +1,219 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.hadoop.yarn.api.records.Resource;
+import
org.apache.hadoop.yarn.server.resourcemanager.nodelabels.RMNodeLabelsManager;
+import org.apache.hadoop.yarn.server.resourcemanager.scheduler.ResourceLimits;
+import
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.QueueCapacityVector.ResourceUnitCapacityType;
+import org.apache.hadoop.yarn.util.resource.ResourceCalculator;
+import org.apache.hadoop.yarn.util.resource.ResourceUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.LinkedHashSet;
+import java.util.Map;
+import java.util.Set;
+
+import static
org.apache.hadoop.yarn.api.records.ResourceInformation.MEMORY_URI;
+import static
org.apache.hadoop.yarn.api.records.ResourceInformation.VCORES_URI;
+import static
org.apache.hadoop.yarn.nodelabels.CommonNodeLabelsManager.NO_LABEL;
+
+/**
+ * Controls how capacity and resource values are set and calculated for a
queue.
+ * Effective minimum and maximum resource values are set for each label and
resource separately.
+ */
+public class CapacitySchedulerQueueCapacityHandler {
Review Comment:
Thanks, got it
> Centralize queue resource calculation based on CapacityVectors
> --
>
> Key: YARN-10965
> URL: https://issues.apache.org/jira/browse/YARN-10965
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Reporter: Andras Gyori
>Assignee: Andras Gyori
>Priority: Major
> Labels: pull-request-available
> Time Spent: 6.5h
> Remaining Estimate: 0h
>
> With the introduction of YARN-10930 it is possible to unify queue resource
> calculation. In order to narrow down the scope of this patch, the base system
> is implemented here, without refactoring the existing resource calculation in
> updateClusterResource (which will be done in YARN-11000).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11000) Replace queue resource calculation logic in updateClusterResource
[
https://issues.apache.org/jira/browse/YARN-11000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680995#comment-17680995
]
ASF GitHub Bot commented on YARN-11000:
---
hadoop-yetus commented on PR #3618:
URL: https://github.com/apache/hadoop/pull/3618#issuecomment-1405017142
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 17 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 35m 10s | | trunk passed |
| +1 :green_heart: | compile | 11m 10s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 9m 37s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 1s | | trunk passed |
| -1 :x: | javadoc | 0m 54s |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3618/4/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 1m 28s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 4m 28s | | trunk passed |
| +1 :green_heart: | shadedclient | 28m 39s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 29m 3s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 56s | | the patch passed |
| +1 :green_heart: | compile | 11m 36s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 11m 36s | |
hadoop-yarn-project_hadoop-yarn-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 0 new + 720
unchanged - 1 fixed = 720 total (was 721) |
| +1 :green_heart: | compile | 11m 0s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 11m 0s | |
hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08
with JDK Private Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 0 new + 635
unchanged - 1 fixed = 635 total (was 636) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 51s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3618/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 117 new + 836
unchanged - 2 fixed = 953 total (was 838) |
| +1 :green_heart: | mvnsite | 2m 16s | | the patch passed |
| -1 :x: | javadoc | 1m 0s |
[/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-3618/4/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 5m 11s | | the patch passed |
| +1 :green_heart: | shadedclient | 29m 21s | | patch has no errors
when building and testing our client artifacts. |
[jira] [Commented] (YARN-11416) FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
[ https://issues.apache.org/jira/browse/YARN-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680979#comment-17680979 ] ASF GitHub Bot commented on YARN-11416: --- p-szucs commented on PR #5320: URL: https://github.com/apache/hadoop/pull/5320#issuecomment-1404951874 I added some comments where I saw further possibilities for using the convenience get methods beside the fixed ones. I also saw that there are some config settings in the FsQueueConverter class where the CapacitySchedulerConfiguration’s setter methods could be used too, so that way we would have only one place for setting these values. For example here we can use the CapacitySchedulerConfiguration’s setMaximumCapacity() method: https://github.com/apache/hadoop/blob/a38eb1f19318e3dd34d9312afe1096db46652f49/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/FSQueueConverter.java#L167 What are your thoughts on that @brumi1024, @susheel-gupta? > FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder > --- > > Key: YARN-11416 > URL: https://issues.apache.org/jira/browse/YARN-11416 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Benjamin Teke >Assignee: Susheel Gupta >Priority: Major > Labels: pull-request-available > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSQueueConverter > and its builder stores the variable capacitySchedulerConfig as a simple > Configuration object instead of CapacitySchedulerConfiguration. This is > misleading, as capacitySchedulerConfig suggests that it is indeed a > CapacitySchedulerConfiguration and it loses access to the convenience methods > to check for various properties. Because of this every time a property getter > is changed FS2CS should be checked if it reimplemented the same, otherwise > there might be behaviour differences or even bugs. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11416) FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
[
https://issues.apache.org/jira/browse/YARN-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680971#comment-17680971
]
ASF GitHub Bot commented on YARN-11416:
---
p-szucs commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1087785053
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##
@@ -508,29 +521,29 @@ public void testReservationSystemNotSupported() {
}
private void assertNoValueForQueues(Set queues, String postfix,
- Configuration config) {
+ CapacitySchedulerConfiguration capacitySchedulerConfig) {
for (String queue : queues) {
String key = PREFIX + queue + postfix;
assertNull("Key " + key + " has value, but it should be null",
- config.get(key));
+ capacitySchedulerConfig.get(key));
}
}
private void assertValueForQueues(Set queues, String postfix,
- Configuration config, String expectedValue) {
+ CapacitySchedulerConfiguration capacitySchedulerConfig, String
expectedValue) {
for (String queue : queues) {
String key = PREFIX + queue + postfix;
assertEquals("Key " + key + " has different value",
- expectedValue, config.get(key));
+ expectedValue, capacitySchedulerConfig.get(key));
Review Comment:
The same applies here.
> FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
> ---
>
> Key: YARN-11416
> URL: https://issues.apache.org/jira/browse/YARN-11416
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Benjamin Teke
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSQueueConverter
> and its builder stores the variable capacitySchedulerConfig as a simple
> Configuration object instead of CapacitySchedulerConfiguration. This is
> misleading, as capacitySchedulerConfig suggests that it is indeed a
> CapacitySchedulerConfiguration and it loses access to the convenience methods
> to check for various properties. Because of this every time a property getter
> is changed FS2CS should be checked if it reimplemented the same, otherwise
> there might be behaviour differences or even bugs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11416) FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
[
https://issues.apache.org/jira/browse/YARN-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680970#comment-17680970
]
ASF GitHub Bot commented on YARN-11416:
---
p-szucs commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1087784312
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##
@@ -508,29 +521,29 @@ public void testReservationSystemNotSupported() {
}
private void assertNoValueForQueues(Set queues, String postfix,
- Configuration config) {
+ CapacitySchedulerConfiguration capacitySchedulerConfig) {
for (String queue : queues) {
String key = PREFIX + queue + postfix;
assertNull("Key " + key + " has value, but it should be null",
- config.get(key));
+ capacitySchedulerConfig.get(key));
Review Comment:
I think we should also check where we are calling this method, because in
some cases the simple get should be changed to the corresponding convenience
method here as well.
> FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
> ---
>
> Key: YARN-11416
> URL: https://issues.apache.org/jira/browse/YARN-11416
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Benjamin Teke
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSQueueConverter
> and its builder stores the variable capacitySchedulerConfig as a simple
> Configuration object instead of CapacitySchedulerConfiguration. This is
> misleading, as capacitySchedulerConfig suggests that it is indeed a
> CapacitySchedulerConfiguration and it loses access to the convenience methods
> to check for various properties. Because of this every time a property getter
> is changed FS2CS should be checked if it reimplemented the same, otherwise
> there might be behaviour differences or even bugs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11416) FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
[
https://issues.apache.org/jira/browse/YARN-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680969#comment-17680969
]
ASF GitHub Bot commented on YARN-11416:
---
p-szucs commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1087778061
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSQueueConverter.java:
##
@@ -440,28 +454,27 @@ public void testQueueOrderingPolicy() throws Exception {
rootQueue = fs.getQueueManager().getRootQueue();
converter.convertQueueHierarchy(rootQueue);
-
// root
assertEquals("root ordering policy", null,
-csConfig.get(PREFIX + "root.ordering-policy"));
+capacitySchedulerConfig.get(PREFIX + "root.ordering-policy"));
Review Comment:
Could you please use the getAppOrderingPolicy() method to get the configs in
this test case?
> FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
> ---
>
> Key: YARN-11416
> URL: https://issues.apache.org/jira/browse/YARN-11416
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Benjamin Teke
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSQueueConverter
> and its builder stores the variable capacitySchedulerConfig as a simple
> Configuration object instead of CapacitySchedulerConfiguration. This is
> misleading, as capacitySchedulerConfig suggests that it is indeed a
> CapacitySchedulerConfiguration and it loses access to the convenience methods
> to check for various properties. Because of this every time a property getter
> is changed FS2CS should be checked if it reimplemented the same, otherwise
> there might be behaviour differences or even bugs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11416) FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
[
https://issues.apache.org/jira/browse/YARN-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680968#comment-17680968
]
ASF GitHub Bot commented on YARN-11416:
---
p-szucs commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1087770675
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##
@@ -214,31 +216,31 @@ public void testDefaultMaxAMShareDisabled() throws
Exception {
converter.convert(params);
-Configuration conf = converter.getCapacitySchedulerConfig();
+CapacitySchedulerConfiguration conf =
converter.getCapacitySchedulerConfig();
// -1.0 means disabled ==> 1.0 in CS
assertEquals("Default max-am-resource-percent", "1.0",
conf.get(CapacitySchedulerConfiguration.
MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT));
-// root.admins.bob -1.0 equals to the default -1.0
-assertNull("root.admins.bob maximum-am-resource-percent should be null",
-conf.get(PREFIX + "root.admins.bob.maximum-am-resource-percent"));
+// root.admins.bob is unset,so falling back to the global value
+assertEquals("root.admins.bob maximum-am-resource-percent", 1.0f,
+
conf.getMaximumApplicationMasterResourcePerQueuePercent("root.admins.bob"),
0.0f);
// root.admins.alice 0.15 != -1.0
-assertEquals("root.admins.alice max-am-resource-percent", "0.15",
-conf.get(PREFIX + "root.admins.alice.maximum-am-resource-percent"));
+assertEquals("root.admins.alice max-am-resource-percent", 0.15f,
+
conf.getMaximumApplicationMasterResourcePerQueuePercent("root.admins.alice"),
0.0f);
-// root.users.joe is unset, inherits -1.0
-assertNull("root.users.joe maximum-am-resource-percent should be null",
-conf.get(PREFIX + "root.users.joe.maximum-am-resource-percent"));
+// root.users.joe is unset,so falling back to the global value
+assertEquals("root.users.joe maximum-am-resource-percent", 1.0f,
+
conf.getMaximumApplicationMasterResourcePerQueuePercent("root.users.joe"),
0.0f);
}
@Test
public void testConvertACLs() throws Exception {
converter.convert(config);
-Configuration conf = converter.getCapacitySchedulerConfig();
+CapacitySchedulerConfiguration conf =
converter.getCapacitySchedulerConfig();
// root
assertEquals("root submit ACL", "alice,bob,joe,john hadoop_users",
Review Comment:
There is a getAcl() method in the config class which you can use in this
test case.
> FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
> ---
>
> Key: YARN-11416
> URL: https://issues.apache.org/jira/browse/YARN-11416
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Benjamin Teke
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSQueueConverter
> and its builder stores the variable capacitySchedulerConfig as a simple
> Configuration object instead of CapacitySchedulerConfiguration. This is
> misleading, as capacitySchedulerConfig suggests that it is indeed a
> CapacitySchedulerConfiguration and it loses access to the convenience methods
> to check for various properties. Because of this every time a property getter
> is changed FS2CS should be checked if it reimplemented the same, otherwise
> there might be behaviour differences or even bugs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11416) FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
[
https://issues.apache.org/jira/browse/YARN-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680964#comment-17680964
]
ASF GitHub Bot commented on YARN-11416:
---
p-szucs commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1087764208
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##
@@ -214,31 +216,31 @@ public void testDefaultMaxAMShareDisabled() throws
Exception {
converter.convert(params);
-Configuration conf = converter.getCapacitySchedulerConfig();
+CapacitySchedulerConfiguration conf =
converter.getCapacitySchedulerConfig();
// -1.0 means disabled ==> 1.0 in CS
assertEquals("Default max-am-resource-percent", "1.0",
conf.get(CapacitySchedulerConfiguration.
MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT));
Review Comment:
The same as my previous comment.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##
@@ -214,31 +216,31 @@ public void testDefaultMaxAMShareDisabled() throws
Exception {
converter.convert(params);
-Configuration conf = converter.getCapacitySchedulerConfig();
+CapacitySchedulerConfiguration conf =
converter.getCapacitySchedulerConfig();
// -1.0 means disabled ==> 1.0 in CS
assertEquals("Default max-am-resource-percent", "1.0",
conf.get(CapacitySchedulerConfiguration.
MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT));
Review Comment:
The same here as my previous comment.
> FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
> ---
>
> Key: YARN-11416
> URL: https://issues.apache.org/jira/browse/YARN-11416
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Benjamin Teke
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSQueueConverter
> and its builder stores the variable capacitySchedulerConfig as a simple
> Configuration object instead of CapacitySchedulerConfiguration. This is
> misleading, as capacitySchedulerConfig suggests that it is indeed a
> CapacitySchedulerConfiguration and it loses access to the convenience methods
> to check for various properties. Because of this every time a property getter
> is changed FS2CS should be checked if it reimplemented the same, otherwise
> there might be behaviour differences or even bugs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11416) FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
[
https://issues.apache.org/jira/browse/YARN-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680965#comment-17680965
]
ASF GitHub Bot commented on YARN-11416:
---
p-szucs commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1087762346
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##
@@ -172,37 +170,41 @@ private void createConverter() {
public void testDefaultMaxAMShare() throws Exception {
converter.convert(config);
-Configuration conf = converter.getCapacitySchedulerConfig();
+CapacitySchedulerConfiguration conf =
converter.getCapacitySchedulerConfig();
String maxAmShare =
conf.get(CapacitySchedulerConfiguration.
MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT);
Review Comment:
There is a getMaximumApplicationMasterResourcePercent() method in the
CapacitySchedulerConfiuration class which returns this value. Could you please
use that method here?
> FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
> ---
>
> Key: YARN-11416
> URL: https://issues.apache.org/jira/browse/YARN-11416
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Benjamin Teke
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSQueueConverter
> and its builder stores the variable capacitySchedulerConfig as a simple
> Configuration object instead of CapacitySchedulerConfiguration. This is
> misleading, as capacitySchedulerConfig suggests that it is indeed a
> CapacitySchedulerConfiguration and it loses access to the convenience methods
> to check for various properties. Because of this every time a property getter
> is changed FS2CS should be checked if it reimplemented the same, otherwise
> there might be behaviour differences or even bugs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11416) FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
[
https://issues.apache.org/jira/browse/YARN-11416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17680963#comment-17680963
]
ASF GitHub Bot commented on YARN-11416:
---
p-szucs commented on code in PR #5320:
URL: https://github.com/apache/hadoop/pull/5320#discussion_r1087762346
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/converter/TestFSConfigToCSConfigConverter.java:
##
@@ -172,37 +170,41 @@ private void createConverter() {
public void testDefaultMaxAMShare() throws Exception {
converter.convert(config);
-Configuration conf = converter.getCapacitySchedulerConfig();
+CapacitySchedulerConfiguration conf =
converter.getCapacitySchedulerConfig();
String maxAmShare =
conf.get(CapacitySchedulerConfiguration.
MAXIMUM_APPLICATION_MASTERS_RESOURCE_PERCENT);
Review Comment:
There is a getMaximumApplicationMasterResourcePercent() method in the
CapacitySchedulerConfiuration class which returns this value. Could you please
use that here?
> FS2CS should use CapacitySchedulerConfiguration in FSQueueConverterBuilder
> ---
>
> Key: YARN-11416
> URL: https://issues.apache.org/jira/browse/YARN-11416
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Benjamin Teke
>Assignee: Susheel Gupta
>Priority: Major
> Labels: pull-request-available
>
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.converter.FSQueueConverter
> and its builder stores the variable capacitySchedulerConfig as a simple
> Configuration object instead of CapacitySchedulerConfiguration. This is
> misleading, as capacitySchedulerConfig suggests that it is indeed a
> CapacitySchedulerConfiguration and it loses access to the convenience methods
> to check for various properties. Because of this every time a property getter
> is changed FS2CS should be checked if it reimplemented the same, otherwise
> there might be behaviour differences or even bugs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
