[jira] [Commented] (YARN-11664) Remove HDFS Binaries/Jars Dependency From YARN
[
https://issues.apache.org/jira/browse/YARN-11664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831179#comment-17831179
]
ASF GitHub Bot commented on YARN-11664:
---
shameersss1 commented on PR #6631:
URL: https://github.com/apache/hadoop/pull/6631#issuecomment-2022015512
> looks ok to me, but hdfs mail list should be invited to comment.
>
> made some minor suggestions
I have send an email to hdfs mailing list asking their opinion as well.
> Remove HDFS Binaries/Jars Dependency From YARN
> --
>
> Key: YARN-11664
> URL: https://issues.apache.org/jira/browse/YARN-11664
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
>
> In principle Hadoop Yarn is independent of HDFS. It can work with any
> filesystem. Currently there exists some code dependency for Yarn with HDFS.
> This dependency requires Yarn to bring in some of the HDFS binaries/jars to
> its class path. The idea behind this jira is to remove this dependency so
> that Yarn can run without HDFS binaries/jars
> *Scope*
> 1. Non test classes are considered
> 2. Some test classes which comes as transitive dependency are considered
> *Out of scope*
> 1. All test classes in Yarn module is not considered
>
>
> A quick search in Yarn module revealed following HDFS dependencies
> 1. Constants
> {code:java}
> import
> org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier;
> import org.apache.hadoop.hdfs.DFSConfigKeys;{code}
>
>
> 2. Exception
> {code:java}
> import org.apache.hadoop.hdfs.protocol.DSQuotaExceededException;{code}
>
> 3. Utility
> {code:java}
> import org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair;{code}
>
> Both Yarn and HDFS depends on *hadoop-common* module,
> * Constants variables and Utility classes can be moved to *hadoop-common*
> * Instead of DSQuotaExceededException, Use the parent exception
> ClusterStoragrCapacityExceeded
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11664) Remove HDFS Binaries/Jars Dependency From YARN
[
https://issues.apache.org/jira/browse/YARN-11664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831169#comment-17831169
]
ASF GitHub Bot commented on YARN-11664:
---
shameersss1 commented on code in PR #6631:
URL: https://github.com/apache/hadoop/pull/6631#discussion_r1540493331
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOStreamPair.java:
##
@@ -15,15 +15,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.hadoop.hdfs.protocol.datatransfer;
+package org.apache.hadoop.io;
import java.io.Closeable;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import org.apache.hadoop.classification.InterfaceAudience;
-import org.apache.hadoop.io.IOUtils;
/**
* A little struct class to wrap an InputStream and an OutputStream.
Review Comment:
ack
> Remove HDFS Binaries/Jars Dependency From YARN
> --
>
> Key: YARN-11664
> URL: https://issues.apache.org/jira/browse/YARN-11664
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
>
> In principle Hadoop Yarn is independent of HDFS. It can work with any
> filesystem. Currently there exists some code dependency for Yarn with HDFS.
> This dependency requires Yarn to bring in some of the HDFS binaries/jars to
> its class path. The idea behind this jira is to remove this dependency so
> that Yarn can run without HDFS binaries/jars
> *Scope*
> 1. Non test classes are considered
> 2. Some test classes which comes as transitive dependency are considered
> *Out of scope*
> 1. All test classes in Yarn module is not considered
>
>
> A quick search in Yarn module revealed following HDFS dependencies
> 1. Constants
> {code:java}
> import
> org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier;
> import org.apache.hadoop.hdfs.DFSConfigKeys;{code}
>
>
> 2. Exception
> {code:java}
> import org.apache.hadoop.hdfs.protocol.DSQuotaExceededException;{code}
>
> 3. Utility
> {code:java}
> import org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair;{code}
>
> Both Yarn and HDFS depends on *hadoop-common* module,
> * Constants variables and Utility classes can be moved to *hadoop-common*
> * Instead of DSQuotaExceededException, Use the parent exception
> ClusterStoragrCapacityExceeded
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11664) Remove HDFS Binaries/Jars Dependency From YARN
[
https://issues.apache.org/jira/browse/YARN-11664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831159#comment-17831159
]
ASF GitHub Bot commented on YARN-11664:
---
shameersss1 commented on code in PR #6631:
URL: https://github.com/apache/hadoop/pull/6631#discussion_r1540456198
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/Constants.java:
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop;
+
+import org.apache.hadoop.io.Text;
+
+/**
+ * This class contains constants for configuration keys and default values.
+ */
+public final class Constants {
Review Comment:
ack
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/Constants.java:
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop;
+
+import org.apache.hadoop.io.Text;
+
+/**
+ * This class contains constants for configuration keys and default values.
+ */
+public final class Constants {
+
+ public static final Text HDFS_DELEGATION_KIND =
Review Comment:
ack
> Remove HDFS Binaries/Jars Dependency From YARN
> --
>
> Key: YARN-11664
> URL: https://issues.apache.org/jira/browse/YARN-11664
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
>
> In principle Hadoop Yarn is independent of HDFS. It can work with any
> filesystem. Currently there exists some code dependency for Yarn with HDFS.
> This dependency requires Yarn to bring in some of the HDFS binaries/jars to
> its class path. The idea behind this jira is to remove this dependency so
> that Yarn can run without HDFS binaries/jars
> *Scope*
> 1. Non test classes are considered
> 2. Some test classes which comes as transitive dependency are considered
> *Out of scope*
> 1. All test classes in Yarn module is not considered
>
>
> A quick search in Yarn module revealed following HDFS dependencies
> 1. Constants
> {code:java}
> import
> org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier;
> import org.apache.hadoop.hdfs.DFSConfigKeys;{code}
>
>
> 2. Exception
> {code:java}
> import org.apache.hadoop.hdfs.protocol.DSQuotaExceededException;{code}
>
> 3. Utility
> {code:java}
> import org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair;{code}
>
> Both Yarn and HDFS depends on *hadoop-common* module,
> * Constants variables and Utility classes can be moved to *hadoop-common*
> * Instead of DSQuotaExceededException, Use the parent exception
> ClusterStoragrCapacityExceeded
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11663) [Federation] Add Cache Entity Nums Limit.
[
https://issues.apache.org/jira/browse/YARN-11663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831102#comment-17831102
]
ASF GitHub Bot commented on YARN-11663:
---
slfan1989 commented on code in PR #6662:
URL: https://github.com/apache/hadoop/pull/6662#discussion_r1540208851
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java:
##
@@ -4031,6 +4031,11 @@ public static boolean isAclEnabled(Configuration conf) {
// 5 minutes
public static final int DEFAULT_FEDERATION_CACHE_TIME_TO_LIVE_SECS = 5 * 60;
+ public static final String FEDERATION_CACHE_ENTITY_NUMS =
+ FEDERATION_PREFIX + "cache-entity.nums";
+ // default 1000
Review Comment:
Thanks for reviewing the code! I will improve the code.
> [Federation] Add Cache Entity Nums Limit.
> -
>
> Key: YARN-11663
> URL: https://issues.apache.org/jira/browse/YARN-11663
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation, yarn
>Affects Versions: 3.4.0
>Reporter: Yuan Luo
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-03-14-18-12-28-426.png,
> image-2024-03-14-18-12-49-950.png, image-2024-03-15-10-50-32-860.png
>
>
> !image-2024-03-14-18-12-28-426.png!
> !image-2024-03-14-18-12-49-950.png!
> hi [~slfan1989] After apply this feature to our prod env, I found the memory
> of the router keeps growing over time. This is because after jobs finished,
> we won't access the expired key to trigger cleanup mechanism. Is it better to
> add cache maximum number limit？
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-11664) Remove HDFS Binaries/Jars Dependency From YARN
[
https://issues.apache.org/jira/browse/YARN-11664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai reassigned YARN-11664:
---
Assignee: Syed Shameerur Rahman
> Remove HDFS Binaries/Jars Dependency From YARN
> --
>
> Key: YARN-11664
> URL: https://issues.apache.org/jira/browse/YARN-11664
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
>
> In principle Hadoop Yarn is independent of HDFS. It can work with any
> filesystem. Currently there exists some code dependency for Yarn with HDFS.
> This dependency requires Yarn to bring in some of the HDFS binaries/jars to
> its class path. The idea behind this jira is to remove this dependency so
> that Yarn can run without HDFS binaries/jars
> *Scope*
> 1. Non test classes are considered
> 2. Some test classes which comes as transitive dependency are considered
> *Out of scope*
> 1. All test classes in Yarn module is not considered
>
>
> A quick search in Yarn module revealed following HDFS dependencies
> 1. Constants
> {code:java}
> import
> org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier;
> import org.apache.hadoop.hdfs.DFSConfigKeys;{code}
>
>
> 2. Exception
> {code:java}
> import org.apache.hadoop.hdfs.protocol.DSQuotaExceededException;{code}
>
> 3. Utility
> {code:java}
> import org.apache.hadoop.hdfs.protocol.datatransfer.IOStreamPair;{code}
>
> Both Yarn and HDFS depends on *hadoop-common* module,
> * Constants variables and Utility classes can be moved to *hadoop-common*
> * Instead of DSQuotaExceededException, Use the parent exception
> ClusterStoragrCapacityExceeded
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11664) Remove HDFS Binaries/Jars Dependency From YARN
[
https://issues.apache.org/jira/browse/YARN-11664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17831047#comment-17831047
]
ASF GitHub Bot commented on YARN-11664:
---
steveloughran commented on code in PR #6631:
URL: https://github.com/apache/hadoop/pull/6631#discussion_r1539932538
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOStreamPair.java:
##
@@ -15,15 +15,14 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.hadoop.hdfs.protocol.datatransfer;
+package org.apache.hadoop.io;
import java.io.Closeable;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import org.apache.hadoop.classification.InterfaceAudience;
-import org.apache.hadoop.io.IOUtils;
/**
* A little struct class to wrap an InputStream and an OutputStream.
Review Comment:
add that they both get closed in the close
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/Constants.java:
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop;
+
+import org.apache.hadoop.io.Text;
+
+/**
+ * This class contains constants for configuration keys and default values.
+ */
+public final class Constants {
+
+ public static final Text HDFS_DELEGATION_KIND =
Review Comment:
javadocs with {@value} entries here and below
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/Constants.java:
##
@@ -0,0 +1,35 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop;
+
+import org.apache.hadoop.io.Text;
+
+/**
+ * This class contains constants for configuration keys and default values.
+ */
+public final class Constants {
Review Comment:
not in this package. should go somewhere under filesystem, maybe a class like
`org.apache.hadoop.fs.HdfsCommonConstants` for all hdfs related constants
which need to go into hadoop-common
> Remove HDFS Binaries/Jars Dependency From YARN
> --
>
> Key: YARN-11664
> URL: https://issues.apache.org/jira/browse/YARN-11664
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Reporter: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
>
> In principle Hadoop Yarn is independent of HDFS. It can work with any
> filesystem. Currently there exists some code dependency for Yarn with HDFS.
> This dependency requires Yarn to bring in some of the HDFS binaries/jars to
> its class path. The idea behind this jira is to remove this dependency so
> that Yarn can run without HDFS binaries/jars
> *Scope*
> 1. Non test classes are considered
> 2. Some test classes which comes as transitive dependency are considered
> *Out of scope*
> 1. All test classes in Yarn module is not considered
>
>
> A quick search in Yarn module revealed following HDFS dependencies
> 1. Constants
> {code:java}
> import
> org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenIdentifier;
>
[jira] [Commented] (YARN-11663) [Federation] Add Cache Entity Nums Limit.
[
https://issues.apache.org/jira/browse/YARN-11663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830999#comment-17830999
]
ASF GitHub Bot commented on YARN-11663:
---
dineshchitlangia commented on code in PR #6662:
URL: https://github.com/apache/hadoop/pull/6662#discussion_r1539567214
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java:
##
@@ -4031,6 +4031,11 @@ public static boolean isAclEnabled(Configuration conf) {
// 5 minutes
public static final int DEFAULT_FEDERATION_CACHE_TIME_TO_LIVE_SECS = 5 * 60;
+ public static final String FEDERATION_CACHE_ENTITY_NUMS =
+ FEDERATION_PREFIX + "cache-entity.nums";
+ // default 1000
Review Comment:
NIT - This comment can be removed as the variable name is appropriate.
> [Federation] Add Cache Entity Nums Limit.
> -
>
> Key: YARN-11663
> URL: https://issues.apache.org/jira/browse/YARN-11663
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation, yarn
>Affects Versions: 3.4.0
>Reporter: Yuan Luo
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-03-14-18-12-28-426.png,
> image-2024-03-14-18-12-49-950.png, image-2024-03-15-10-50-32-860.png
>
>
> !image-2024-03-14-18-12-28-426.png!
> !image-2024-03-14-18-12-49-950.png!
> hi [~slfan1989] After apply this feature to our prod env, I found the memory
> of the router keeps growing over time. This is because after jobs finished,
> we won't access the expired key to trigger cleanup mechanism. Is it better to
> add cache maximum number limit？
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11387) [GPG] YARN GPG mistakenly deleted applicationid
[
https://issues.apache.org/jira/browse/YARN-11387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830971#comment-17830971
]
ASF GitHub Bot commented on YARN-11387:
---
slfan1989 commented on code in PR #6660:
URL: https://github.com/apache/hadoop/pull/6660#discussion_r1539407981
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-globalpolicygenerator/src/main/java/org/apache/hadoop/yarn/server/globalpolicygenerator/applicationcleaner/DefaultApplicationCleaner.java:
##
@@ -46,47 +45,38 @@ public void run() {
LOG.info("Application cleaner run at time {}", now);
FederationStateStoreFacade facade = getGPGContext().getStateStoreFacade();
Review Comment:
Step 1: Retrieve all applications stored in the StateStore, which represents
all applications submitted to the Router.
Step 2: Use the Router's REST API to fetch all running tasks. This API will
invoke applications from all active SubClusters.
Step 3: Compare the results of Step1 and Step2 to identify applications that
exist in Step1 but not in Step2. Delete these applications.
There is a potential issue with this approach. If a particular SubCluster is
undergoing maintenance, such as RM restart, Step2 will not be able to fetch the
complete list of running applications. As a result, during the comparison in
Step3, there is a risk of mistakenly deleting applications that are still
running.
We have three SubClusters: subClusterA, subClusterB, and subClusterC, with
an equal allocation ratio of 1:1:1.
We submit six applications through routerA.
app1 and app2 are allocated to subClusterA
app3 and app4 to subClusterB
app5 and app6 to subClusterC.
Among these, app1, app3, and app5 have completed their execution, and we
expect to retain app2, app4, and app6 in the StateStore.
In the normal scenario:
Comparing the steps mentioned above:
Step 1: We will retrieve six applications [app1, app2, app3, app4, app5,
app6] from the StateStore.
Step 2: We will fetch three applications [app2, app4, app6] from the
Router's REST interface.
Step 3: By comparing Step 1 and Step 2, we can identify that applications
[app1, app3, app5] should be deleted.
In the exceptional scenario:
Comparing the steps mentioned above:
Step 1: We will retrieve six applications [app1, app2, app3, app4, app5,
app6] from the StateStore.
Step 2: We will fetch the list of running applications from the Router's
REST interface. However, due to maintenance in subClusterB and subClusterC, we
can only obtain the applications running in subClusterA [app2].
Step 3: By comparing Step 1 and Step 3, we can identify that applications
[app1, app3, app4, app5, app6] should be deleted.
In this case, we had an error deletion.
> [GPG] YARN GPG mistakenly deleted applicationid
> ---
>
> Key: YARN-11387
> URL: https://issues.apache.org/jira/browse/YARN-11387
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.2.1, 3.4.0
>Reporter: zhangjunj
>Assignee: Shilun Fan
>Priority: Major
> Labels: federation, gpg, pull-request-available
> Attachments: YARN-11387-YARN-11387.v1.patch,
> yarn-gpg-mistakenly-deleted-applicationid.png
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In [YARN-7599|https://issues.apache.org/jira/browse/YARN-7599], the
> Federation can delete expired applicationid, but YARN GPG uses getRouter()
> method to obtain application information for multiple clusters. If there are
> too many applicationids that more than 200,000 , it will not be possible to
> pull all the applicationid information at one time, resulting in the
> possibility of accidental deletion. The following error is reported for spark
> component.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11663) [Federation] Add Cache Entity Nums Limit.
[ https://issues.apache.org/jira/browse/YARN-11663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830970#comment-17830970 ] ASF GitHub Bot commented on YARN-11663: --- slfan1989 commented on PR #6662: URL: https://github.com/apache/hadoop/pull/6662#issuecomment-2020639828 @goiri Can you help review this PR? Thank you very much! > [Federation] Add Cache Entity Nums Limit. > - > > Key: YARN-11663 > URL: https://issues.apache.org/jira/browse/YARN-11663 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, yarn >Affects Versions: 3.4.0 >Reporter: Yuan Luo >Priority: Major > Labels: pull-request-available > Attachments: image-2024-03-14-18-12-28-426.png, > image-2024-03-14-18-12-49-950.png, image-2024-03-15-10-50-32-860.png > > > !image-2024-03-14-18-12-28-426.png! > !image-2024-03-14-18-12-49-950.png! > hi [~slfan1989] After apply this feature to our prod env, I found the memory > of the router keeps growing over time. This is because after jobs finished, > we won't access the expired key to trigger cleanup mechanism. Is it better to > add cache maximum number limit？ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11261) Upgrade JUnit from 4 to 5 in hadoop-yarn-server-web-proxy
[
https://issues.apache.org/jira/browse/YARN-11261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830929#comment-17830929
]
ASF GitHub Bot commented on YARN-11261:
---
hadoop-yetus commented on PR #6652:
URL: https://github.com/apache/hadoop/pull/6652#issuecomment-2020426986
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 22s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 19m 52s | | trunk passed |
| +1 :green_heart: | compile | 8m 39s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 7m 53s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 2m 3s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 16s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 1s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 2m 35s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 8m 22s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 8m 22s | | the patch passed |
| +1 :green_heart: | compile | 8m 4s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 8m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 3s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6652/3/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 7 new + 140 unchanged - 0 fixed = 147 total (was
140) |
| +1 :green_heart: | mvnsite | 1m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 7s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| -1 :x: | spotbugs | 1m 33s |
[/new-spotbugs-hadoop-common-project_hadoop-common.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6652/3/artifact/out/new-spotbugs-hadoop-common-project_hadoop-common.html)
| hadoop-common-project/hadoop-common generated 1 new + 0 unchanged - 0 fixed
= 1 total (was 0) |
| +1 :green_heart: | shadedclient | 20m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 42s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 88m 3s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6652/3/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 237m 43s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-common-project/hadoop-common |
| | Inconsistent synchronization of
org.apache.hadoop.conf.Configuration.properties; locked 87% of time
Unsynchronized access at Configuration.java:87% of time Unsynchronized access
at Configuration.java:[line 771] |
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerNewQueueAutoCreation
|
| Subsystem | Report/Notes |
|--:|:-|
