[jira] [Created] (YARN-11562) [Federation] GPG Support Query Policies In Web.
Shilun Fan created YARN-11562: - Summary: [Federation] GPG Support Query Policies In Web. Key: YARN-11562 URL: https://issues.apache.org/jira/browse/YARN-11562 Project: Hadoop YARN Issue Type: Sub-task Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan Attachments: image-2023-09-05-22-57-32-641.png GPG should provide the functionality to query queue policies through the web interface. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11435) [Router] FederationStateStoreFacade is not reinitialized with Router conf
[ https://issues.apache.org/jira/browse/YARN-11435?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11435. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Target Version/s: 3.4.0 Assignee: Shilun Fan (was: Aparajita Choudhary) Resolution: Fixed > [Router] FederationStateStoreFacade is not reinitialized with Router conf > - > > Key: YARN-11435 > URL: https://issues.apache.org/jira/browse/YARN-11435 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation, router, yarn >Reporter: Aparajita Choudhary >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > FederationStateStoreFacade is not initialized or reinitialized with Router > conf when used from Router's FederationClientInterceptor. Ends up connecting > to default memory state store. > h4. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6537) Running RM tests against the Router
[ https://issues.apache.org/jira/browse/YARN-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-6537. -- Hadoop Flags: Reviewed Target Version/s: 3.4.0 Resolution: Fixed > Running RM tests against the Router > --- > > Key: YARN-6537 > URL: https://issues.apache.org/jira/browse/YARN-6537 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Shilun Fan >Priority: Minor > Labels: pull-request-available > > We should run all the YARN RM tests that involve ApplicationClientProtocol, > AdminProtocol and RestAPI against the Router. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11537) [Federation] Router CLI Supports List SubClusterPolicyConfiguration Of Queues.
[ https://issues.apache.org/jira/browse/YARN-11537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11537. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > [Federation] Router CLI Supports List SubClusterPolicyConfiguration Of Queues. > -- > > Key: YARN-11537 > URL: https://issues.apache.org/jira/browse/YARN-11537 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > In this jira, we will support list SubClusterPolicyConfiguration information > on the command line. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11561) [Federation] GPG Supports Format PolicyStateStore.
Shilun Fan created YARN-11561: - Summary: [Federation] GPG Supports Format PolicyStateStore. Key: YARN-11561 URL: https://issues.apache.org/jira/browse/YARN-11561 Project: Hadoop YARN Issue Type: Sub-task Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11570) Add YARN_GLOBALPOLICYGENERATOR_HEAPSIZE to yarn-env for GPG
Shilun Fan created YARN-11570:
-
Summary: Add YARN_GLOBALPOLICYGENERATOR_HEAPSIZE to yarn-env for
GPG
Key: YARN-11570
URL: https://issues.apache.org/jira/browse/YARN-11570
Project: Hadoop YARN
Issue Type: Sub-task
Components: federation
Affects Versions: 3.4.0
Reporter: Shilun Fan
Assignee: Shilun Fan
Yarn services NM, RM, Router etc have YARN_{SERVICENAME}_HEAPSIZE variable
defined, we should have similar parameter for GPG Service also.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11571) [GPG] YARN GPG Supports Cross-Origin.
Shilun Fan created YARN-11571: - Summary: [GPG] YARN GPG Supports Cross-Origin. Key: YARN-11571 URL: https://issues.apache.org/jira/browse/YARN-11571 Project: Hadoop YARN Issue Type: Sub-task Components: federation Reporter: Shilun Fan Assignee: Shilun Fan 1.The Rest interface of GPG should support Cross-Origin like RM and NM. 2.The Yarn-UI2 module requires the GPG to support Cross-Origin. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11434) [Router] UGI conf doesn't read user overridden configurations on Router startup
[ https://issues.apache.org/jira/browse/YARN-11434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11434. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Target Version/s: 3.4.0 Assignee: Shilun Fan (was: Aparajita Choudhary) Resolution: Fixed > [Router] UGI conf doesn't read user overridden configurations on Router > startup > --- > > Key: YARN-11434 > URL: https://issues.apache.org/jira/browse/YARN-11434 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation, router, yarn >Reporter: Aparajita Choudhary >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Similar to [YARN-9668|https://issues.apache.org/jira/browse/YARN-9668], > configs overridden thru e.g. -D or -conf are not passed to the UGI conf on > Router startup. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-8980) Mapreduce application container start fail after AM restart.
[ https://issues.apache.org/jira/browse/YARN-8980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-8980. -- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Mapreduce application container start fail after AM restart. > - > > Key: YARN-8980 > URL: https://issues.apache.org/jira/browse/YARN-8980 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bibin Chundatt >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > UAM to subclusters are always launched with keepContainers. > On AM restart scenarios , UAM register again with RM . UAM receive running > containers with NMToken. NMToken received by UAM in > getPreviousAttemptContainersNMToken is never used by mapreduce application. > Federation Interceptor should take care of such scenarios too. Merge NMToken > received at registration to allocate response. > Container allocation response on same node will have NMToken empty. > issue credits : [~Nallasivan] > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11552) timeline endpoint: /clusters/{clusterid}/apps/{appid}/entity-types Error when using hdfs store
[
https://issues.apache.org/jira/browse/YARN-11552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan resolved YARN-11552.
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Assignee: Jiandan Yang
Resolution: Fixed
> timeline endpoint: /clusters/{clusterid}/apps/{appid}/entity-types Error when
> using hdfs store
> --
>
> Key: YARN-11552
> URL: https://issues.apache.org/jira/browse/YARN-11552
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Jiandan Yang
>Assignee: Jiandan Yang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> *Problem description*
> I start hadoop cluster and timeservice based on hdfs store ，it throw
> execption when accessing
> [http://timelineserver:8088/ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo=hadoop=1692343022285],
> and detailed logs includes exception:
> {code:java}
> 2023-08-18 17:40:51 2023-08-18 09:40:51 INFO TimelineReaderWebServices:3372
> - Processed URL
> /ws/v2/timeline/clusters/yjd/apps/application_1692341890246_0001/entity-types?flowname=QuasiMonteCarlo=hadoop=1=yjd=application_1692341890246_0001
> (Took 56 ms.)
> 2023-08-18 17:40:51 2023-08-18 09:40:51 WARN GenericExceptionHandler:96 -
> INTERNAL_SERVER_ERROR
> 2023-08-18 17:40:51 javax.ws.rs.WebApplicationException:
> java.io.FileNotFoundException: File
> /tmp/hadoop-hadoop/timeline_service_data/entities/yjd/hadoop/QuasiMonteCarlo/1/application_1692341890246_0001
> does not exist.
> 2023-08-18 17:40:51 at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.handleException(TimelineReaderWebServices.java:200)
> 2023-08-18 17:40:51 at
> org.apache.hadoop.yarn.server.timelineservice.reader.TimelineReaderWebServices.getEntityTypes(TimelineReaderWebServices.java:3368)
> 2023-08-18 17:40:51 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2023-08-18 17:40:51 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2023-08-18 17:40:51 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2023-08-18 17:40:51 at java.lang.reflect.Method.invoke(Method.java:498)
> 2023-08-18 17:40:51 at
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
> 2023-08-18 17:40:51 at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
> 2023-08-18 17:40:51 at
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
> 2023-08-18 17:40:51 at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
> 2023-08-18 17:40:51 at
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
> 2023-08-18 17:40:51 at
> javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
> 2023-08-18 17:40:51 at
> org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656)
> 2023-08-18 17:40:51 at
>
[jira] [Resolved] (YARN-11564) Fix wrong config in yarn-default.xml
[ https://issues.apache.org/jira/browse/YARN-11564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11564. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix wrong config in yarn-default.xml > > > Key: YARN-11564 > URL: https://issues.apache.org/jira/browse/YARN-11564 > Project: Hadoop YARN > Issue Type: Improvement > Components: router >Affects Versions: 3.3.6 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Fix For: 3.4.0 > > > yarn.router.subcluster.cleaner.interval.time is duplicated in yarn-default.xml -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11597) NPE when getting the static files in SLSWebApp
[
https://issues.apache.org/jira/browse/YARN-11597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan resolved YARN-11597.
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
Resolution: Fixed
> NPE when getting the static files in SLSWebApp
> ---
>
> Key: YARN-11597
> URL: https://issues.apache.org/jira/browse/YARN-11597
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler-load-simulator
>Affects Versions: 3.3.6
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments: 20231023-171754.jpeg
>
>
> When using the SLS, the web api of {{http://localhost:10001/simulate}} is
> broken, because the static file loading failed due to 404.
> This is caused by the static handler is not initialized. NPE stacktrace is
> attached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11593) [Federation] Improve command line help information.
[ https://issues.apache.org/jira/browse/YARN-11593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11593. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Target Version/s: 3.4.0 Resolution: Fixed > [Federation] Improve command line help information. > --- > > Key: YARN-11593 > URL: https://issues.apache.org/jira/browse/YARN-11593 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > This jira will improve the help information of routeradmin so that users can > better understand and use commands. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10900) Yarn nodes missing from router web app
[
https://issues.apache.org/jira/browse/YARN-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan resolved YARN-10900.
---
Fix Version/s: 3.4.0
Resolution: Not A Bug
> Yarn nodes missing from router web app
> --
>
> Key: YARN-10900
> URL: https://issues.apache.org/jira/browse/YARN-10900
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation, router
>Affects Versions: 3.2.1
>Reporter: Babble Shack
>Priority: Major
> Fix For: 3.4.0
>
>
> {color:#172b4d}Hi,
> I am trying to configure YARN Federation mode.I seem to be able to schedule
> to all nodes in my federation across each of my subclusters.
>
> However my federation router shows both of my subclusters, but nodes from
> only a single cluster.{color}
>
>
> {color:#ff}[!https://preview.redd.it/hdjwtn43ptj71.png?width=1437=png=webp=2d55343688c0de7a6f3da629e334cd318219c392!|https://preview.redd.it/hdjwtn43ptj71.png?width=1437=png=webp=2d55343688c0de7a6f3da629e334cd318219c392]{color}
> {color:#172b4d}Federation Page – Showing both clusters and both nodes{color}
> {color:#172b4d}
> This page is showing both of my clusters, configured with a single <8 CPU,
> 7GB> node.{color}
> {color:#172b4d}However the "Nodes" and "About" pages are invalid.{color}
>
>
> [!https://preview.redd.it/lawgst2yotj71.png?width=1373=png=webp=d06663904538bc993418c6184c3686cc2b02ea6e!|https://preview.redd.it/lawgst2yotj71.png?width=1373=png=webp=d06663904538bc993418c6184c3686cc2b02ea6e]
> {color:#172b4d}Nodes Page – showing nodes from only one cluster{color}
>
>
> [!https://preview.redd.it/dtuqblquotj71.png?width=482=png=webp=df740ff49df8b8de5015bccc936c9accee65aee0!|https://preview.redd.it/dtuqblquotj71.png?width=482=png=webp=df740ff49df8b8de5015bccc936c9accee65aee0]
> {color:#172b4d}About Page – showing nodes from only one cluster{color}
> {color:#172b4d}Each node is configured as follows:{color}
> {color:#172b4d}*Minimum Memory Allocation:* 512MB{color}
> {color:#172b4d}*Minimum CPU Allocation:* 1{color}
> {color:#172b4d}*Maximum Memory Allocation:* 7168{color}
> {color:#172b4d}*Maximum CPU Allocation:* 7{color}
> {color:#172b4d}Federation configuration can be found at this
> [link|https://drive.google.com/file/d/16xc2V7CvJLVQgsaDEOHhIaDrz5dnKxB_/view?usp=sharing]{color}
> {color:#172b4d}Has anyone had an issue like this before, does anyone have any
> solutions?{color}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10776) Make ConfiguredRMFailoverProxyProvider select ResourceManager or Router randomly
[ https://issues.apache.org/jira/browse/YARN-10776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-10776. --- Fix Version/s: 3.4.0 Resolution: Duplicate > Make ConfiguredRMFailoverProxyProvider select ResourceManager or Router > randomly > > > Key: YARN-10776 > URL: https://issues.apache.org/jira/browse/YARN-10776 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > I think we need to select resourcemanager or router randomly, especially for > router in yarn federation. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11609) Improve the time unit for FederationRMAdminInterceptor#heartbeatExpirationMillis
[ https://issues.apache.org/jira/browse/YARN-11609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11609. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Improve the time unit for > FederationRMAdminInterceptor#heartbeatExpirationMillis > > > Key: YARN-11609 > URL: https://issues.apache.org/jira/browse/YARN-11609 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Reporter: WangYuanben >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11610) [Federation] Add WeightedHomePolicyManager
Shilun Fan created YARN-11610: - Summary: [Federation] Add WeightedHomePolicyManager Key: YARN-11610 URL: https://issues.apache.org/jira/browse/YARN-11610 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan We will add a new PolicyManager. This PolicyManager can select a router based on the weight configured by the user, and then all container requests will be in the home-subcluster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11599) Incorrect log4j properties file in SLS sample conf
[
https://issues.apache.org/jira/browse/YARN-11599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan resolved YARN-11599.
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Target Version/s: 3.4.0
Resolution: Fixed
> Incorrect log4j properties file in SLS sample conf
> --
>
> Key: YARN-11599
> URL: https://issues.apache.org/jira/browse/YARN-11599
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler-load-simulator
>Reporter: Junfan Zhang
>Assignee: Junfan Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-sls/src/main/sample-conf/log4j.properties
>
> log4j.appender.test=org.apache.log4j.ConsoleAppender
> log4j.appender.test.Target=System.out
> log4j.appender.test.layout=org.apache.log4j.PatternLayout
> log4j.appender.test.layout.ConversionPattern=%d\{ABSOLUTE} %5p %c\{1}:%L -
> %m%n
> log4j.logger=NONE, test
>
> This is invalid in current log4j version, if applied this, the test
> performance will be slow!
>
> I think the warn level is enough and required. The level < warn will effect
> performance
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11548) [Federation] Router Supports Format FederationStateStore.
[ https://issues.apache.org/jira/browse/YARN-11548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11548. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > [Federation] Router Supports Format FederationStateStore. > - > > Key: YARN-11548 > URL: https://issues.apache.org/jira/browse/YARN-11548 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Similar to RM, Router should support format FederationStateStore. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11011) Make YARN Router throw Exception to client clearly
[ https://issues.apache.org/jira/browse/YARN-11011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11011. --- Hadoop Flags: Reviewed Target Version/s: 3.4.0 Resolution: Fixed > Make YARN Router throw Exception to client clearly > -- > > Key: YARN-11011 > URL: https://issues.apache.org/jira/browse/YARN-11011 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.3.1 >Reporter: Yuan Luo >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: router-error.png > > Time Spent: 20m > Remaining Estimate: 0h > > When router submits job to yarn, the job is rejected by yarn for some reason. > Yarn router just throw exception to log and return not enough info to client. > The router always throws the following non-obvious error to client: > Exception in thread "main" > org.apache.hadoop.yarn.server.federation.policies.exceptions.FederationPolicyException: > No active SubCluster available to submit the request. > To make it easy for the user to locate the cause of a job submission failure, > we need to make Router throw the true cause of the error to the client > clearly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11594) [Federation] Improve Yarn Federation documentation
[ https://issues.apache.org/jira/browse/YARN-11594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11594. --- Hadoop Flags: Reviewed Resolution: Fixed > [Federation] Improve Yarn Federation documentation > -- > > Key: YARN-11594 > URL: https://issues.apache.org/jira/browse/YARN-11594 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > > In this JIRA we will continue to improve the Yarn Federation documentation. > 1. Add configuration information of database connection pool. > 2. Fixed some MD document format errors. > 3. Modify some default parameter configuration errors. > 4. Add instructions for configuring Router/GPG memory & JVM OPT. > 5. Add instructions for configuring Router’s AuditLog. > 6. Add instructions for configuring the Inteceptor of Router. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11592) Add timeout to GPGUtils#invokeRMWebService.
[ https://issues.apache.org/jira/browse/YARN-11592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11592. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Add timeout to GPGUtils#invokeRMWebService. > --- > > Key: YARN-11592 > URL: https://issues.apache.org/jira/browse/YARN-11592 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > GPGUtils#invokeRMWebService utilizes a Client, and we should set a timeout > for the Client. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9586) [QA] Need more doc for yarn.federation.policy-manager-params when LoadBasedRouterPolicy is used
[ https://issues.apache.org/jira/browse/YARN-9586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-9586. -- Fix Version/s: 3.4.0 Target Version/s: 3.4.0 Resolution: Fixed > [QA] Need more doc for yarn.federation.policy-manager-params when > LoadBasedRouterPolicy is used > --- > > Key: YARN-9586 > URL: https://issues.apache.org/jira/browse/YARN-9586 > Project: Hadoop YARN > Issue Type: Wish > Components: federation >Reporter: Shen Yinjie >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > We picked LoadBasedRouterPolicy for YARN federation, but had no idea what to > set to yarn.federation.policy-manager-params. Is there a demo config or more > detailed description for this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11580) YARN Router Web supports displaying information for Non-Federation.
Shilun Fan created YARN-11580: - Summary: YARN Router Web supports displaying information for Non-Federation. Key: YARN-11580 URL: https://issues.apache.org/jira/browse/YARN-11580 Project: Hadoop YARN Issue Type: Improvement Components: federation, router Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan Typically, YARN Router serves multiple YARN sub-clusters in what we refer to as YARN Federation. In this scenario, there are two or more YARN sub-clusters, and Router, based on queue configuration weights, forwards client requests to a specific sub-cluster. However, YARN Router also offers the capability to directly forward client requests to a particular local cluster. This functionality has already been implemented, and you can refer to components such as DefaultRequestInterceptorREST, DefaultClientRequestInterceptor, and DefaultRMAdminRequestInterceptor. In this JIRA ticket, we aim to enhance the functionality of YARN Router Web to display information about a specific local cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11583) Improve Node Link for YARN Federation Web Page
Shilun Fan created YARN-11583: - Summary: Improve Node Link for YARN Federation Web Page Key: YARN-11583 URL: https://issues.apache.org/jira/browse/YARN-11583 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan When working on the YARN Federation Web Page, I noticed that the functionality for Node redirection is missing. In this JIRA, I will be enhancing this feature. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11576) Improve FederationInterceptorREST AuditLog
Shilun Fan created YARN-11576: - Summary: Improve FederationInterceptorREST AuditLog Key: YARN-11576 URL: https://issues.apache.org/jira/browse/YARN-11576 Project: Hadoop YARN Issue Type: Improvement Reporter: Shilun Fan Some methods in Federation Interceptor REST lack Audit Log recording. This JIRA will enhance this functionality. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11577) Improve FederationInterceptorREST Method Result
Shilun Fan created YARN-11577: - Summary: Improve FederationInterceptorREST Method Result Key: YARN-11577 URL: https://issues.apache.org/jira/browse/YARN-11577 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan In the FederationInterceptorREST, some methods currently return null or throw exceptions. We will ensure that the returned results are similar to those returned by RM, handling cases of null values or exceptions gracefully. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11579) Fix 'Physical Mem Used' and 'Physical VCores Used' are not displaying data
Shilun Fan created YARN-11579: - Summary: Fix 'Physical Mem Used' and 'Physical VCores Used' are not displaying data Key: YARN-11579 URL: https://issues.apache.org/jira/browse/YARN-11579 Project: Hadoop YARN Issue Type: Improvement Components: federation, router Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan During the YARN Federation integration testing process, we encountered an issue where the 'About' page of the Router does not display cluster resource utilization. This problem arises from the omission of calculations for the 'Physical Mem Used' and 'Physical VCores Used' metrics when merging metrics from sub-clusters. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11580) YARN Router Web supports displaying information for Non-Federation.
[ https://issues.apache.org/jira/browse/YARN-11580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11580. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > YARN Router Web supports displaying information for Non-Federation. > --- > > Key: YARN-11580 > URL: https://issues.apache.org/jira/browse/YARN-11580 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, router >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Typically, YARN Router serves multiple YARN sub-clusters in what we refer to > as YARN Federation. In this scenario, there are two or more YARN > sub-clusters, and Router, based on queue configuration weights, forwards > client requests to a specific sub-cluster. > However, YARN Router also offers the capability to directly forward client > requests to a particular local cluster. This functionality has already been > implemented, and you can refer to components such as > DefaultRequestInterceptorREST, DefaultClientRequestInterceptor, and > DefaultRMAdminRequestInterceptor. > In this JIRA ticket, we aim to enhance the functionality of YARN Router Web > to display information about a specific local cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11579) Fix 'Physical Mem Used' and 'Physical VCores Used' are not displaying data
[ https://issues.apache.org/jira/browse/YARN-11579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11579. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix 'Physical Mem Used' and 'Physical VCores Used' are not displaying data > -- > > Key: YARN-11579 > URL: https://issues.apache.org/jira/browse/YARN-11579 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, router >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > During the YARN Federation integration testing process, we encountered an > issue where the 'About' page of the Router does not display cluster resource > utilization. This problem arises from the omission of calculations for the > 'Physical Mem Used' and 'Physical VCores Used' metrics when merging metrics > from sub-clusters. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11593) [Federation] Improve command line help information.
Shilun Fan created YARN-11593: - Summary: [Federation] Improve command line help information. Key: YARN-11593 URL: https://issues.apache.org/jira/browse/YARN-11593 Project: Hadoop YARN Issue Type: Sub-task Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan This jira will improve the help information of routeradmin so that users can better understand and use commands. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11594) Improve Yarn Federation documentation
Shilun Fan created YARN-11594: - Summary: Improve Yarn Federation documentation Key: YARN-11594 URL: https://issues.apache.org/jira/browse/YARN-11594 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan In this JIRA we will continue to improve the Yarn Federation documentation. 1. Add configuration information of database connection pool. 2. Fixed some MD document format errors. 3. Modify some default parameter configuration errors. 4. Add instructions for configuring Router/GPG memory & JVM OPT. 5. Add instructions for configuring Router’s AuditLog. 6. Add instructions for configuring the Inteceptor of Router. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11595) Fix hadoop-yarn-client#java.lang.NoClassDefFoundError
Shilun Fan created YARN-11595:
-
Summary: Fix hadoop-yarn-client#java.lang.NoClassDefFoundError
Key: YARN-11595
URL: https://issues.apache.org/jira/browse/YARN-11595
Project: Hadoop YARN
Issue Type: Bug
Components: yarn-client
Affects Versions: 3.4.0
Reporter: Shilun Fan
Assignee: Shilun Fan
When completing the PR, I found the following issue in the compilation report:
{code:java}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1:test (default-test) on
project hadoop-yarn-client: Execution default-test of goal
org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1:test failed:
java.lang.NoClassDefFoundError: org/junit/jupiter/api/TestInfo:
org.junit.jupiter.api.TestInfo -
{code}
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6199/9/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11595) Fix hadoop-yarn-client#java.lang.NoClassDefFoundError
[
https://issues.apache.org/jira/browse/YARN-11595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan resolved YARN-11595.
---
Fix Version/s: 3.4.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Fix hadoop-yarn-client#java.lang.NoClassDefFoundError
> -
>
> Key: YARN-11595
> URL: https://issues.apache.org/jira/browse/YARN-11595
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn-client
>Affects Versions: 3.4.0
>Reporter: Shilun Fan
>Assignee: Shilun Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
>
> When completing the PR, I found the following issue in the compilation report:
> {code:java}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1:test (default-test)
> on project hadoop-yarn-client: Execution default-test of goal
> org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M1:test failed:
> java.lang.NoClassDefFoundError: org/junit/jupiter/api/TestInfo:
> org.junit.jupiter.api.TestInfo -
> {code}
> https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6199/9/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11469) Refactor FederationStateStoreFacade Cache Code
[ https://issues.apache.org/jira/browse/YARN-11469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11469. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Refactor FederationStateStoreFacade Cache Code > -- > > Key: YARN-11469 > URL: https://issues.apache.org/jira/browse/YARN-11469 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The Cache of FederationStateStoreFacade uses JCache, but considering that > JCache is not a general Cache implementation (the latest version was released > in 2014), this part of the code is refactored to support multiple Cache in > the future. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11571) [GPG] Add Information About YARN GPG in Federation.md
[ https://issues.apache.org/jira/browse/YARN-11571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11571. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > [GPG] Add Information About YARN GPG in Federation.md > - > > Key: YARN-11571 > URL: https://issues.apache.org/jira/browse/YARN-11571 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11583) Improve Node Link for YARN Federation Web Page
[ https://issues.apache.org/jira/browse/YARN-11583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11583. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Improve Node Link for YARN Federation Web Page > -- > > Key: YARN-11583 > URL: https://issues.apache.org/jira/browse/YARN-11583 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > When working on the YARN Federation Web Page, I noticed that the > functionality for Node redirection is missing. In this JIRA, I will be > enhancing this feature. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11576) Improve FederationInterceptorREST AuditLog
[ https://issues.apache.org/jira/browse/YARN-11576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11576. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Improve FederationInterceptorREST AuditLog > -- > > Key: YARN-11576 > URL: https://issues.apache.org/jira/browse/YARN-11576 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, router >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Some methods in Federation Interceptor REST lack Audit Log recording. This > JIRA will enhance this functionality. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11592) Add timeout to GPGUtils#invokeRMWebService.
Shilun Fan created YARN-11592: - Summary: Add timeout to GPGUtils#invokeRMWebService. Key: YARN-11592 URL: https://issues.apache.org/jira/browse/YARN-11592 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan GPGUtils#invokeRMWebService utilizes a Client, and we should set a timeout for the Client. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11554) Fix TestRMFailover#testWebAppProxyInStandAloneMode Failed
Shilun Fan created YARN-11554: - Summary: Fix TestRMFailover#testWebAppProxyInStandAloneMode Failed Key: YARN-11554 URL: https://issues.apache.org/jira/browse/YARN-11554 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan There is an error in TestRMFailover#testWebAppProxyInStandAloneMode, which is caused after YARN-11153 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10218) [GPG] Support HTTPS in GPG
[ https://issues.apache.org/jira/browse/YARN-10218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-10218. --- Fix Version/s: 3.4.0 Target Version/s: 3.4.0 Resolution: Fixed > [GPG] Support HTTPS in GPG > -- > > Key: YARN-10218 > URL: https://issues.apache.org/jira/browse/YARN-10218 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Bilwa S T >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > HTTPS support in Router is handled as part of Jira YARN-10120. Https Rest > calls from GPG to Router must be supported -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11154) Make router support proxy server.
[ https://issues.apache.org/jira/browse/YARN-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11154. --- Hadoop Flags: Reviewed Target Version/s: 3.4.0 Resolution: Fixed > Make router support proxy server. > - > > Key: YARN-11154 > URL: https://issues.apache.org/jira/browse/YARN-11154 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-11154.draft.patch > > > Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and > YARN-10775-design-doc.001.pdf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11613) [Federation] Router CLI Supports Delete SubClusterPolicyConfiguration Of Queues.
Shilun Fan created YARN-11613: - Summary: [Federation] Router CLI Supports Delete SubClusterPolicyConfiguration Of Queues. Key: YARN-11613 URL: https://issues.apache.org/jira/browse/YARN-11613 Project: Hadoop YARN Issue Type: Sub-task Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan Router's command line tool needs to support delete SubClusterPolicyConfiguration. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11409) Fix Typo of ResourceManager#webapp moudle
Shilun Fan created YARN-11409: - Summary: Fix Typo of ResourceManager#webapp moudle Key: YARN-11409 URL: https://issues.apache.org/jira/browse/YARN-11409 Project: Hadoop YARN Issue Type: Bug Components: resourcemanager Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan Fix For: 3.4.0 When finishing YARN-11218, I found some typo problems in RM's RMWebServices. I checked the java class of the webapp moudle and fixed the typo problems. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11413) Fix Junit Test ERROR Introduced By YARN-6412
Shilun Fan created YARN-11413: - Summary: Fix Junit Test ERROR Introduced By YARN-6412 Key: YARN-11413 URL: https://issues.apache.org/jira/browse/YARN-11413 Project: Hadoop YARN Issue Type: Bug Components: api Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan YARN-6412 (#5242) caused TestCommonConfigurationFields#testCompareXmlAgainstConfigurationClass to be wrong, I will fix it -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11409) Fix Typo of ResourceManager#webapp moudle
[ https://issues.apache.org/jira/browse/YARN-11409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11409. --- Target Version/s: 3.4.0 Resolution: Fixed > Fix Typo of ResourceManager#webapp moudle > - > > Key: YARN-11409 > URL: https://issues.apache.org/jira/browse/YARN-11409 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > When finishing YARN-11218, I found some typo problems in RM's RMWebServices. > I checked the java class of the webapp moudle and fixed the typo problems. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11394) Fix hadoop-yarn-server-resourcemanager module Java Doc Errors.
Shilun Fan created YARN-11394: - Summary: Fix hadoop-yarn-server-resourcemanager module Java Doc Errors. Key: YARN-11394 URL: https://issues.apache.org/jira/browse/YARN-11394 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan When I finished the pr of YARN-11350, I encountered java doc error, this jira will fix the java doc error in hadoop-yarn-server-resourcemanager. The java doc error report is as follows: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5131/22/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11385) Fix hadoop-yarn-server-common module Java Doc Errors
Shilun Fan created YARN-11385: - Summary: Fix hadoop-yarn-server-common module Java Doc Errors Key: YARN-11385 URL: https://issues.apache.org/jira/browse/YARN-11385 Project: Hadoop YARN Issue Type: Improvement Components: yarn Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan In the process of completing YARN-11350, the java doc error is displayed in the compilation report, and I will fix the java doc error of the hadoop-yarn-server-common module. The compilation report is as follows: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5131/15/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-common-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11423) [Federation] Router Supports CLI Commands
Shilun Fan created YARN-11423: - Summary: [Federation] Router Supports CLI Commands Key: YARN-11423 URL: https://issues.apache.org/jira/browse/YARN-11423 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan In YARN Federation mode, if we want to execute SubCluster commands, we need to go to SubCluster RM and execute commands. Router Should Provide Cli commands, So that We Can Execute Commands For Different SubClusters. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11424) [Federation] Router Supports DeregisterSubCluster.
Shilun Fan created YARN-11424: - Summary: [Federation] Router Supports DeregisterSubCluster. Key: YARN-11424 URL: https://issues.apache.org/jira/browse/YARN-11424 Project: Hadoop YARN Issue Type: Sub-task Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan Router provides commands for Deregister SubCluster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11425) [Federation] Router Supports SubClusterCleaner
Shilun Fan created YARN-11425: - Summary: [Federation] Router Supports SubClusterCleaner Key: YARN-11425 URL: https://issues.apache.org/jira/browse/YARN-11425 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan In YARN-Federation mode, once a SubCluster is registered, the SubCluster is always in the RUNNING state, even if the SubCluster has no heartbeat for a long time. We will let the Router automatically check the heartbeat time of the SubCluster, and once the heartbeat exceeds 30mins, we will set the SubCluster to the SC_LOST state. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11289) [Federation] Improve NM FederationInterceptor removeAppFromRegistry
[ https://issues.apache.org/jira/browse/YARN-11289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11289. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > [Federation] Improve NM FederationInterceptor removeAppFromRegistry > --- > > Key: YARN-11289 > URL: https://issues.apache.org/jira/browse/YARN-11289 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, nodemanager >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > [Federation] Improve NM FederationInterceptor removeAppFromRegistry > 1.FederationInterceptor#finishApplicationMaster needs to check > getFinalApplicationStatus, and clean up when the status is SUCCESS. > 2. When FederationInterceptor#shutdown, perform CleanUp on Application. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11446) [Federation] Add updateSchedulerConfiguration, getSchedulerConfiguration REST APIs for Router.
Shilun Fan created YARN-11446: - Summary: [Federation] Add updateSchedulerConfiguration, getSchedulerConfiguration REST APIs for Router. Key: YARN-11446 URL: https://issues.apache.org/jira/browse/YARN-11446 Project: Hadoop YARN Issue Type: Sub-task Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11445) [Federation] Add getClusterInfo, getClusterUserInfo REST APIs for Router.
Shilun Fan created YARN-11445: - Summary: [Federation] Add getClusterInfo, getClusterUserInfo REST APIs for Router. Key: YARN-11445 URL: https://issues.apache.org/jira/browse/YARN-11445 Project: Hadoop YARN Issue Type: Sub-task Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11442) Refactor FederationInterceptorREST Code
Shilun Fan created YARN-11442: - Summary: Refactor FederationInterceptorREST Code Key: YARN-11442 URL: https://issues.apache.org/jira/browse/YARN-11442 Project: Hadoop YARN Issue Type: Improvement Components: federation, router Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan Fix code issues to make code more readable 1. Add getOrCreateInterceptorForSubCluster method, the parameter is a parameter of type SubClusterInfo. 2. Delete FederationInterceptorREST#getActiveSubclusters, add getActiveSubClusters in FederationStateStoreFacade, you can get the return result of Collection. 3. Parallel query code, use Collection's parallelStream method instead of thread pool. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11439) Fix Typo of hadoop-yarn-ui README.md
[ https://issues.apache.org/jira/browse/YARN-11439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11439. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix Typo of hadoop-yarn-ui README.md > > > Key: YARN-11439 > URL: https://issues.apache.org/jira/browse/YARN-11439 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn-ui-v2 >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > When I read the README of hadoop-yarn-ui, I found a typo issue, I will fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11444) Improve YARN md documentation format
Shilun Fan created YARN-11444: - Summary: Improve YARN md documentation format Key: YARN-11444 URL: https://issues.apache.org/jira/browse/YARN-11444 Project: Hadoop YARN Issue Type: Improvement Components: yarn Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan 1. Improve the table format to make the readability better 2. Modify some typo errors 3. Modify the list number to display correctly -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11469) Refactor FederationStateStoreFacade Cache Code
Shilun Fan created YARN-11469: - Summary: Refactor FederationStateStoreFacade Cache Code Key: YARN-11469 URL: https://issues.apache.org/jira/browse/YARN-11469 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan The Cache of FederationStateStoreFacade uses JCache, but considering that JCache is not a general Cache implementation (the latest version was released in 2014), this part of the code is refactored to support multiple Cache in the future. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11471) FederationStateStoreFacade Cache Support Caffeine
Shilun Fan created YARN-11471: - Summary: FederationStateStoreFacade Cache Support Caffeine Key: YARN-11471 URL: https://issues.apache.org/jira/browse/YARN-11471 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan FederationStateStoreFacade Cache Support Caffeine -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11470) FederationStateStoreFacade Cache Support Guava Cache
Shilun Fan created YARN-11470: - Summary: FederationStateStoreFacade Cache Support Guava Cache Key: YARN-11470 URL: https://issues.apache.org/jira/browse/YARN-11470 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan FederationStateStoreFacade Cache Support Guava Cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11478) [Federation] SQLFederationStateStore Support Store ApplicationSubmitData
Shilun Fan created YARN-11478: - Summary: [Federation] SQLFederationStateStore Support Store ApplicationSubmitData Key: YARN-11478 URL: https://issues.apache.org/jira/browse/YARN-11478 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan Before completing YARN-8898, we need to store ApplicationSubmitData in FederationStateStore first, this jira will store ApplicationSubmitData in SQLFederationStateStore. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11477) [Federation] MemoryFederationStateStore Support Store ApplicationSubmitData
Shilun Fan created YARN-11477: - Summary: [Federation] MemoryFederationStateStore Support Store ApplicationSubmitData Key: YARN-11477 URL: https://issues.apache.org/jira/browse/YARN-11477 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan Before completing YARN-8898, We need to store ApplicationSubmitData in FederationStateStore first, this jira will store ApplicationSubmitData in MemoryFederationStateStore. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11479) [Federation] ZookeeperFederationStateStore Support Store ApplicationSubmitData
Shilun Fan created YARN-11479: - Summary: [Federation] ZookeeperFederationStateStore Support Store ApplicationSubmitData Key: YARN-11479 URL: https://issues.apache.org/jira/browse/YARN-11479 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan Before completing YARN-8898, we need to store ApplicationSubmitData in FederationStateStore first, this jira will store ApplicationSubmitData in ZookeeperFederationStateStore. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11474) The yarn queue list is displayed on the CLI
[ https://issues.apache.org/jira/browse/YARN-11474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11474. --- Fix Version/s: 3.4.0 (was: 3.3.4) Hadoop Flags: Reviewed Target Version/s: 3.4.0 (was: 3.3.4) Resolution: Fixed > The yarn queue list is displayed on the CLI > --- > > Key: YARN-11474 > URL: https://issues.apache.org/jira/browse/YARN-11474 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Affects Versions: 3.3.4 >Reporter: Lu Yuan >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > The yarn queue list is displayed on the CLI -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11485) [Federation] Router Supports Yarn Admin CLI Cmds.
Shilun Fan created YARN-11485: - Summary: [Federation] Router Supports Yarn Admin CLI Cmds. Key: YARN-11485 URL: https://issues.apache.org/jira/browse/YARN-11485 Project: Hadoop YARN Issue Type: Sub-task Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11484) [Federation] Router Supports Yarn Client CLI Cmds.
Shilun Fan created YARN-11484: - Summary: [Federation] Router Supports Yarn Client CLI Cmds. Key: YARN-11484 URL: https://issues.apache.org/jira/browse/YARN-11484 Project: Hadoop YARN Issue Type: Sub-task Components: federation Reporter: Shilun Fan Assignee: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11483) [Federation] Router AdminCLI Supports Clean Finish Apps.
Shilun Fan created YARN-11483: - Summary: [Federation] Router AdminCLI Supports Clean Finish Apps. Key: YARN-11483 URL: https://issues.apache.org/jira/browse/YARN-11483 Project: Hadoop YARN Issue Type: Sub-task Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11462) Fix Typo of hadoop-yarn-common
[ https://issues.apache.org/jira/browse/YARN-11462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11462. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Fix Typo of hadoop-yarn-common > -- > > Key: YARN-11462 > URL: https://issues.apache.org/jira/browse/YARN-11462 > Project: Hadoop YARN > Issue Type: Improvement > Components: yarn >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > In the process of reading the code, I found some typo issues, this jira will > fix these problems, I checked all the classes of hadoop-yarn-common. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11465) Improved YarnClient Log Format
[ https://issues.apache.org/jira/browse/YARN-11465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11465. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Improved YarnClient Log Format > -- > > Key: YARN-11465 > URL: https://issues.apache.org/jira/browse/YARN-11465 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Affects Versions: 3.4.0 >Reporter: Lu Yuan >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > When reading the YarnClient module code, I found some logging irregularities. > This PR fixes the problem -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11426) Improve YARN NodeLabel Memory Display
[ https://issues.apache.org/jira/browse/YARN-11426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11426. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Improve YARN NodeLabel Memory Display > - > > Key: YARN-11426 > URL: https://issues.apache.org/jira/browse/YARN-11426 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: before improvement.png, image-2023-02-01-17-55-59-037.png > > > NodeLabel displays the memory information of the cluster, directly displays > the numbers, does not display the unit information, and is not easy to > understand. > - before improvement > !before improvement.png! > * after improvement > !image-2023-02-01-17-55-59-037.png! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11462) Fix Typo of hadoop-yarn-common
Shilun Fan created YARN-11462: - Summary: Fix Typo of hadoop-yarn-common Key: YARN-11462 URL: https://issues.apache.org/jira/browse/YARN-11462 Project: Hadoop YARN Issue Type: Improvement Components: yarn Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan In the process of reading the code, I found some typo issues, this jira will fix these problems, I checked all the classes of hadoop-yarn-common. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10146) [Federation] Add missing REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-10146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-10146. --- Hadoop Flags: Reviewed Resolution: Fixed > [Federation] Add missing REST APIs for Router > - > > Key: YARN-10146 > URL: https://issues.apache.org/jira/browse/YARN-10146 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Reporter: Bilwa S T >Assignee: Shilun Fan >Priority: Major > > In FederationInterceptorREST following API are not implemented: > |getClusterInfo| > |getSchedulerInfo| > |dumpSchedulerLogs| > |getActivities| > |getAppActivities| > |getAppStatistics| > |getAppState| > |getNodeToLabels| > |getLabelsToNodes| > |replaceLabelsOnNodes| > |replaceLabelsOnNode| > |getClusterNodeLabels| > |addToClusterNodeLabels| > |removeFromCluserNodeLabels| > |getLabelsOnNode| > |getAppPriority| > |updateApplicationPriority| > |getAppQueue| > |updateAppQueue| > |postDelegationToken| > |postDelegationTokenExpiration| > |cancelDelegationToken| > |createNewReservation| > |submitReservation| > |updateReservation| > |deleteReservation| > |listReservation| > |getAppTimeout| > |getAppTimeouts| > |updateApplicationTimeout| > |getAppAttempts| > |checkUserAccessToQueue| > |getAppAttempt| > |getContainers| > |getContainer| -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11419) Remove redundant exception capture in NMClientAsyncImpl and improve readability in ContainerShellWebSocket, etc
[ https://issues.apache.org/jira/browse/YARN-11419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11419. --- Fix Version/s: 3.4.0 Resolution: Fixed > Remove redundant exception capture in NMClientAsyncImpl and improve > readability in ContainerShellWebSocket, etc > > > Key: YARN-11419 > URL: https://issues.apache.org/jira/browse/YARN-11419 > Project: Hadoop YARN > Issue Type: Improvement > Components: client >Reporter: jingxiong zhong >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Make the code more readable. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11436) [Federation] MemoryFederationStateStore Support Version.
Shilun Fan created YARN-11436: - Summary: [Federation] MemoryFederationStateStore Support Version. Key: YARN-11436 URL: https://issues.apache.org/jira/browse/YARN-11436 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan In YARN-5604, we define the method that FederationStateStore supports Version, and we will implement this method in MemoryFederationStateStore. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11323) [Federation] Improve Router Handler FinishApps
[ https://issues.apache.org/jira/browse/YARN-11323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11323. --- Fix Version/s: 3.4.0 Resolution: Fixed > [Federation] Improve Router Handler FinishApps > -- > > Key: YARN-11323 > URL: https://issues.apache.org/jira/browse/YARN-11323 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, router, yarn >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-6572) Refactoring Router services to use common util classes for pipeline creations
[ https://issues.apache.org/jira/browse/YARN-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-6572. -- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Target Version/s: 3.4.0 Resolution: Fixed > Refactoring Router services to use common util classes for pipeline creations > - > > Key: YARN-6572 > URL: https://issues.apache.org/jira/browse/YARN-6572 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-3657) Federation maintenance mechanisms (simple CLI and command propagation)
[ https://issues.apache.org/jira/browse/YARN-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-3657. -- Fix Version/s: 3.4.0 Resolution: Fixed > Federation maintenance mechanisms (simple CLI and command propagation) > -- > > Key: YARN-3657 > URL: https://issues.apache.org/jira/browse/YARN-3657 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Carlo Curino >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > The maintenance mechanisms provided by the RM are not sufficient in a > federated environment. In this JIRA we track few extensions > (more to come later) to allow basic maintenance mechanisms (and command > propagation) for the federated components. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11333) Federation: Improve Yarn Router Web Page
[ https://issues.apache.org/jira/browse/YARN-11333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11333. --- Fix Version/s: 3.4.0 Resolution: Fixed > Federation: Improve Yarn Router Web Page > > > Key: YARN-11333 > URL: https://issues.apache.org/jira/browse/YARN-11333 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation, router >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Fix For: 3.4.0 > > > Improve the Yarn Router page, involving pages such as About, Federation, > Node, Application, etc. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11438) [Federation] ZookeeperFederationStateStore Support Version.
Shilun Fan created YARN-11438: - Summary: [Federation] ZookeeperFederationStateStore Support Version. Key: YARN-11438 URL: https://issues.apache.org/jira/browse/YARN-11438 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan In YARN-5604, we define the method that FederationStateStore supports Version, and we will implement this method in ZookeeperFederationStateStore. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11437) [Federation] SQLFederationStateStore Support Version.
Shilun Fan created YARN-11437: - Summary: [Federation] SQLFederationStateStore Support Version. Key: YARN-11437 URL: https://issues.apache.org/jira/browse/YARN-11437 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan In YARN-5604, we define the method that FederationStateStore supports Version, and we will implement this method in MemoryFederationStateStore. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11439) Fix Typo of hadoop-yarn-ui README.md
Shilun Fan created YARN-11439: - Summary: Fix Typo of hadoop-yarn-ui README.md Key: YARN-11439 URL: https://issues.apache.org/jira/browse/YARN-11439 Project: Hadoop YARN Issue Type: Improvement Components: yarn-ui-v2 Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan When I read the README of hadoop-yarn-ui, I found a typo issue, I will fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11228) [Federation] Add getAppAttempts, getAppAttempt REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11228. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > [Federation] Add getAppAttempts, getAppAttempt REST APIs for Router > --- > > Key: YARN-11228 > URL: https://issues.apache.org/jira/browse/YARN-11228 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0, 3.3.4 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 3h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11426) Improve YARN NodeLabel Memory Display
Shilun Fan created YARN-11426: - Summary: Improve YARN NodeLabel Memory Display Key: YARN-11426 URL: https://issues.apache.org/jira/browse/YARN-11426 Project: Hadoop YARN Issue Type: Improvement Components: resourcemanager Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan NodeLabel displays the memory information of the cluster, directly displays the numbers, does not display the unit information, and is not easy to understand. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11502) Refactor AMRMProxy#FederationInterceptor#registerApplicationMaster
[ https://issues.apache.org/jira/browse/YARN-11502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11502. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Refactor AMRMProxy#FederationInterceptor#registerApplicationMaster > -- > > Key: YARN-11502 > URL: https://issues.apache.org/jira/browse/YARN-11502 > Project: Hadoop YARN > Issue Type: Improvement > Components: amrmproxy, federation, nodemanager >Affects Versions: 3.4.0 >Reporter: Shilun Fan >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Refactor the code of FederationInterceptor#registerApplicationMaster to > improve code readability and remove redundancy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11527) Upgrade node.js to 14.0.0 in YARN application catalog webapp
Shilun Fan created YARN-11527: - Summary: Upgrade node.js to 14.0.0 in YARN application catalog webapp Key: YARN-11527 URL: https://issues.apache.org/jira/browse/YARN-11527 Project: Hadoop YARN Issue Type: Improvement Components: yarn Affects Versions: 3.4.0 Environment: Upgrade node.js to 14.0.0 in YARN application catalog webapp Reporter: Shilun Fan Assignee: Shilun Fan -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11531) [Federation] Code cleanup for NodeManager#amrmproxy
Shilun Fan created YARN-11531: - Summary: [Federation] Code cleanup for NodeManager#amrmproxy Key: YARN-11531 URL: https://issues.apache.org/jira/browse/YARN-11531 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan In the process of reading the code, I found that the code of NodeManager#amrmproxy has some readability issues. This JIRA will fix the code readability issues. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11536) [Federation] Router CLI Supports Batch Save the SubClusterPolicyConfiguration Of Queues.
Shilun Fan created YARN-11536: - Summary: [Federation] Router CLI Supports Batch Save the SubClusterPolicyConfiguration Of Queues. Key: YARN-11536 URL: https://issues.apache.org/jira/browse/YARN-11536 Project: Hadoop YARN Issue Type: Sub-task Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan In this jira, we will support batch saving of SubCluster Policy Configuration of Queue. Users can provide an xml configuration file, and we will initialize SubCluster Policy Configuration according to the configuration file. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11537) [Federation] Router CLI Supports List SubClusterPolicyConfiguration Of Queues.
Shilun Fan created YARN-11537: - Summary: [Federation] Router CLI Supports List SubClusterPolicyConfiguration Of Queues. Key: YARN-11537 URL: https://issues.apache.org/jira/browse/YARN-11537 Project: Hadoop YARN Issue Type: Sub-task Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan In this jira, we will support list SubClusterPolicyConfiguration information on the command line. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11526) Add a unit test
[ https://issues.apache.org/jira/browse/YARN-11526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11526. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Add a unit test > --- > > Key: YARN-11526 > URL: https://issues.apache.org/jira/browse/YARN-11526 > Project: Hadoop YARN > Issue Type: Test > Components: client >Affects Versions: 3.4.0 >Reporter: Lu Yuan >Assignee: Lu Yuan >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > Add a unit test to YARN-11506 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11276) Add lru cache for RMWebServices.getApps
[ https://issues.apache.org/jira/browse/YARN-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11276. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Target Version/s: 3.4.0 Resolution: Fixed > Add lru cache for RMWebServices.getApps > --- > > Key: YARN-11276 > URL: https://issues.apache.org/jira/browse/YARN-11276 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: Xianming Lei >Assignee: Xianming Lei >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > In our YARN cluster, there are thousands of apps running at the same time, > the return result of getApps reaches about 10M, and many requests are the > same input parameters, we can add cache for RMWebServices.getApps to reduce > processing delay -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11500) Fix typos in hadoop-yarn-server-common#federation
Shilun Fan created YARN-11500: - Summary: Fix typos in hadoop-yarn-server-common#federation Key: YARN-11500 URL: https://issues.apache.org/jira/browse/YARN-11500 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan I found some typos while reading the code, this jira will fix these issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11502) Refactor AMRMProxy#FederationInterceptor#registerApplicationMaster
Shilun Fan created YARN-11502: - Summary: Refactor AMRMProxy#FederationInterceptor#registerApplicationMaster Key: YARN-11502 URL: https://issues.apache.org/jira/browse/YARN-11502 Project: Hadoop YARN Issue Type: Improvement Components: amrmproxy, federation, nodemanager Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan Refactor the code of FederationInterceptor#registerApplicationMaster to improve code readability and remove redundancy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11277) trigger deletion of log-dir by size for NonAggregatingLogHandler
[ https://issues.apache.org/jira/browse/YARN-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11277. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Target Version/s: 3.4.0 Resolution: Fixed > trigger deletion of log-dir by size for NonAggregatingLogHandler > > > Key: YARN-11277 > URL: https://issues.apache.org/jira/browse/YARN-11277 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 3.4.0 >Reporter: Xianming Lei >Assignee: Xianming Lei >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > In our yarn cluster, the log files of some containers are too large, which > causes the NodeManager to frequently switch to the unhealthy state. For logs > that are too large, we can consider deleting them directly without delaying > yarn.nodemanager.log.retain-seconds. > Cluster environment: > # 8k nodes+ > # 50w+ apps / day > Configuration: > # yarn.nodemanager.log.retain-seconds=3days > # yarn.log-aggregation-enable=false > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11497) Support removal of only selective node states in untracked removal flow
[ https://issues.apache.org/jira/browse/YARN-11497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan resolved YARN-11497. --- Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Target Version/s: 3.4.0 Resolution: Fixed > Support removal of only selective node states in untracked removal flow > --- > > Key: YARN-11497 > URL: https://issues.apache.org/jira/browse/YARN-11497 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Mudit Sharma >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > Currently inactive nodes are removed from the Yarn local memory irrespective > of which state they are in. This makes the node removal process not too much > configurable > After this patch: https://issues.apache.org/jira/browse/YARN-10854 > If autoscaling is enabled, lot many nodes go into DECOMMISSIONED state but > still other states like LOST, SHUTDOWN are very less and systems might want > them to be still visible on UI for better tracking > The proposal is to introduce a new config, which when set, will allow only > selective node states to be removed after going into untracked state. > > PR: [https://github.com/apache/hadoop/pull/5681] > Any thoughts/suggestions/feedbacks are welcome! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11508) [Minor] Improve UnmanagedAMPoolManager/UnmanagedApplicationManager Code
Shilun Fan created YARN-11508: - Summary: [Minor] Improve UnmanagedAMPoolManager/UnmanagedApplicationManager Code Key: YARN-11508 URL: https://issues.apache.org/jira/browse/YARN-11508 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan This Jira is mainly used to improve the readability of the code and modify the format of the log. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11509) The FederationInterceptor#launchUAM Added retry logic.
Shilun Fan created YARN-11509: - Summary: The FederationInterceptor#launchUAM Added retry logic. Key: YARN-11509 URL: https://issues.apache.org/jira/browse/YARN-11509 Project: Hadoop YARN Issue Type: Improvement Components: amrmproxy Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan There is a "todo" in the FederationInterceptor#registerAndAllocateWithNewSubClusters method. According to the "todo" description, the request needs to be retried to other subclusters, but changing the parameter requests in registerAndAllocateWithNewSubClusters is not a good operation. It is better to add retry logic here. We don't need to worry about losing requests because when the request cannot be satisfied, the AM of the task will continue to apply, and these requests will be properly transferred to other clusters for execution. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11510) [Federation] Fix NodeManager#TestFederationInterceptor Flaky Unit Test
Shilun Fan created YARN-11510:
-
Summary: [Federation] Fix NodeManager#TestFederationInterceptor
Flaky Unit Test
Key: YARN-11510
URL: https://issues.apache.org/jira/browse/YARN-11510
Project: Hadoop YARN
Issue Type: Improvement
Components: federation, nodemanager
Affects Versions: 3.4.0
Reporter: Shilun Fan
Assignee: Shilun Fan
I am performing overall integration testing for YARN Federation, and I found
that running {{NodeManager#TestFederationInterceptor}} sometimes results in
Flaky Task. This JIRA will fix this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11504) [Federation] YARN Federation Supports Non-HA mode.
Shilun Fan created YARN-11504: - Summary: [Federation] YARN Federation Supports Non-HA mode. Key: YARN-11504 URL: https://issues.apache.org/jira/browse/YARN-11504 Project: Hadoop YARN Issue Type: Improvement Components: federation, nodemanager Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan I am testing the functionality of YARN Federation and have set up the simplest test environment for this purpose. I found that if I set up a non-HA mode test environment, MR/Spark jobs cannot run normally. This JIRA will fix this issue. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11505) [Federation] Add Steps To Set up a Test Cluster.
Shilun Fan created YARN-11505: - Summary: [Federation] Add Steps To Set up a Test Cluster. Key: YARN-11505 URL: https://issues.apache.org/jira/browse/YARN-11505 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan According to the current YARN Federation documentation, users have difficulties setting up a test cluster (due to too many configurations that are needed). This JIRA will improve the documentation so that users can quickly set up a YARN Federation test cluster by following the steps in the documentation. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
[jira] [Created] (YARN-11547) [Federation] Router Supports Remove individual application records from FederationStateStore.
Shilun Fan created YARN-11547: - Summary: [Federation] Router Supports Remove individual application records from FederationStateStore. Key: YARN-11547 URL: https://issues.apache.org/jira/browse/YARN-11547 Project: Hadoop YARN Issue Type: Sub-task Components: federation Affects Versions: 3.4.0 Reporter: Shilun Fan Assignee: Shilun Fan Similar to RM, Router should support clearing individual application from FederationStateStore. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
