[jira] [Commented] (YARN-11297) Improve Yarn Router Reservation Submission Code
[
https://issues.apache.org/jira/browse/YARN-11297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1760#comment-1760
]
ASF GitHub Bot commented on YARN-11297:
---
hadoop-yetus commented on PR #4863:
URL: https://github.com/apache/hadoop/pull/4863#issuecomment-1238846049
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 7s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 56s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 31s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 0m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 40s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 97m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4863/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4863 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux bd48acd0acaf 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0d041a85a5586a2cfbe48993516466fde62f64fb |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4863/1/testReport/ |
| Max. process+thread count | 1857 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4863/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.ap
[jira] [Updated] (YARN-11297) Improve Yarn Router Reservation Submission Code
[ https://issues.apache.org/jira/browse/YARN-11297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11297: -- Labels: pull-request-available (was: ) > Improve Yarn Router Reservation Submission Code > --- > > Key: YARN-11297 > URL: https://issues.apache.org/jira/browse/YARN-11297 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > > The same reservation may be submitted repeatedly. At this time, we should use > the reserved results first. If the reserved results are not available, > consider applying from other RMs. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11297) Improve Yarn Router Reservation Submission Code
[ https://issues.apache.org/jira/browse/YARN-11297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601076#comment-17601076 ] ASF GitHub Bot commented on YARN-11297: --- slfan1989 opened a new pull request, #4863: URL: https://github.com/apache/hadoop/pull/4863 JIRA: YARN-11297. [Federation]Improve Yarn Router Reservation Submission Code. > Improve Yarn Router Reservation Submission Code > --- > > Key: YARN-11297 > URL: https://issues.apache.org/jira/browse/YARN-11297 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > > The same reservation may be submitted repeatedly. At this time, we should use > the reserved results first. If the reserved results are not available, > consider applying from other RMs. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-3657) Federation maintenance mechanisms (simple CLI and command propagation)
[ https://issues.apache.org/jira/browse/YARN-3657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fanshilun reassigned YARN-3657: --- Assignee: fanshilun (was: Carlo Curino) > Federation maintenance mechanisms (simple CLI and command propagation) > -- > > Key: YARN-3657 > URL: https://issues.apache.org/jira/browse/YARN-3657 > Project: Hadoop YARN > Issue Type: Sub-task > Components: nodemanager, resourcemanager >Reporter: Carlo Curino >Assignee: fanshilun >Priority: Major > > The maintenance mechanisms provided by the RM are not sufficient in a > federated environment. In this JIRA we track few extensions > (more to come later) to allow basic maintenance mechanisms (and command > propagation) for the federated components. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-7720) Race condition between second app attempt and UAM timeout when first attempt node is down
[ https://issues.apache.org/jira/browse/YARN-7720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fanshilun reassigned YARN-7720: --- Assignee: fanshilun (was: Botong Huang) > Race condition between second app attempt and UAM timeout when first attempt > node is down > - > > Key: YARN-7720 > URL: https://issues.apache.org/jira/browse/YARN-7720 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Botong Huang >Assignee: fanshilun >Priority: Major > Attachments: YARN-7720.v1.patch, YARN-7720.v2.patch > > > In Federation, multiple attempts of an application share the same UAM in each > secondary sub-cluster. When first attempt fails, we reply on the fact that > secondary RM won't kill the existing UAM before the AM heartbeat timeout > (default at 10 min). When second attempt comes up in the home sub-cluster, it > will pick up the UAM token from Yarn Registry and resume the UAM heartbeat to > secondary RMs. > The default heartbeat timeout for NM and AM are both 10 mins. The problem is > that when the first attempt node goes down or out of connection, only after > 10 mins will the home RM mark the first attempt as failed, and then schedule > the 2nd attempt in some other node. By then the UAMs in secondaries are > already timing out, and they might not survive until the second attempt comes > up. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11297) Improve Yarn Router Reservation Submission Code
fanshilun created YARN-11297: Summary: Improve Yarn Router Reservation Submission Code Key: YARN-11297 URL: https://issues.apache.org/jira/browse/YARN-11297 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0 Reporter: fanshilun Assignee: fanshilun The same reservation may be submitted repeatedly. At this time, we should use the reserved results first. If the reserved results are not available, consider applying from other RMs. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601057#comment-17601057
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r964259520
##
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreTables.sql:
##
@@ -36,7 +36,7 @@ CREATE TABLE membership(
state varchar(32) NOT NULL,
lastStartTime bigint NULL,
capability varchar(6000),
- CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId)
+ CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId),
Review Comment:
In the original code, there is no test for this part. This problem is found
when our test database is executed. I will try to submit a new test code to
cover this part.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[ https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601055#comment-17601055 ] ASF GitHub Bot commented on YARN-11290: --- slfan1989 commented on code in PR #4846: URL: https://github.com/apache/hadoop/pull/4846#discussion_r964257610 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestableFederationClientInterceptor.java: ## @@ -90,6 +95,7 @@ protected ApplicationClientProtocol getClientRMProxyForSubCluster( mockRMs.put(subClusterId, mockRM); } initNodeAttributes(subClusterId, mockRM); + initReservationSystem(mockRM); Review Comment: The new pr will optimize the repetitive submission of reservations and refactor some test code. > Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster > - > > Key: YARN-11290 > URL: https://issues.apache.org/jira/browse/YARN-11290 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Attachments: SqlServer-StoredProc.png > > > 1. Currently this interface returns the number of apps in all sub-clusters, > increasing the limit on the number of query apps, limited to 1000 apps. > 2. Allows to query the App based on the specified HomeSubCluster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601054#comment-17601054
]
ASF GitHub Bot commented on YARN-11296:
---
goiri commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r964257406
##
hadoop-yarn-project/hadoop-yarn/bin/FederationStateStore/MySQL/FederationStateStoreTables.sql:
##
@@ -36,7 +36,7 @@ CREATE TABLE membership(
state varchar(32) NOT NULL,
lastStartTime bigint NULL,
capability varchar(6000),
- CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId)
+ CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId),
Review Comment:
We have no tests that would catch this?
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[ https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601053#comment-17601053 ] ASF GitHub Bot commented on YARN-11290: --- slfan1989 commented on code in PR #4846: URL: https://github.com/apache/hadoop/pull/4846#discussion_r964256887 ## hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestableFederationClientInterceptor.java: ## @@ -90,6 +95,7 @@ protected ApplicationClientProtocol getClientRMProxyForSubCluster( mockRMs.put(subClusterId, mockRM); } initNodeAttributes(subClusterId, mockRM); + initReservationSystem(mockRM); Review Comment: I will submit pr separately to improve this code. > Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster > - > > Key: YARN-11290 > URL: https://issues.apache.org/jira/browse/YARN-11290 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Attachments: SqlServer-StoredProc.png > > > 1. Currently this interface returns the number of apps in all sub-clusters, > increasing the limit on the number of query apps, limited to 1000 apps. > 2. Allows to query the App based on the specified HomeSubCluster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[
https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601052#comment-17601052
]
ASF GitHub Bot commented on YARN-11290:
---
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r964252561
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -949,7 +949,7 @@ public ReservationSubmissionResponse submitReservation(
// Second, determine whether the current ReservationId has a
corresponding subCluster.
// If it does not exist, add it. If it exists, update it.
Boolean exists = existsReservationHomeSubCluster(reservationId);
-if (!exists) {
+if (!exists || i == 0) {
Review Comment:
Reservation may be repeatedly submitted (the same reservationId is submitted
repeatedly). At this time, we should try to use the first reservation result.
If the first reservation encounters some abnormalities, find another RM for
reservation.
If a reservationId(**reservationId_1**) has successfully applied for
reserved resources in SC_1, and the **reservationId_1** is submitted
repeatedly, we should try to use the allocation result of SC_1
For the case of **reservationId_1**, the following code will return true,
```
Boolean exists = existsReservationHomeSubCluster(reservationId);
```
Then the first subcluster(SC_1) will be used instead of SC2
```
updateReservationHomeSubCluster(subClusterId, reservationId,
reservationHomeSubCluster);
```
But at this time we still want to go
`addReservationHomeSubCluster(reservationId, reservationHomeSubCluster)`
`addReservationHomeSubCluster`, will not be added repeatedly for the
existing reservationId.
I added a condition, if i=0, continue to use `addReservationHomeSubCluster`
> Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
> -
>
> Key: YARN-11290
> URL: https://issues.apache.org/jira/browse/YARN-11290
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: SqlServer-StoredProc.png
>
>
> 1. Currently this interface returns the number of apps in all sub-clusters,
> increasing the limit on the number of query apps, limited to 1000 apps.
> 2. Allows to query the App based on the specified HomeSubCluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[
https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601048#comment-17601048
]
ASF GitHub Bot commented on YARN-11290:
---
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r964252561
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -949,7 +949,7 @@ public ReservationSubmissionResponse submitReservation(
// Second, determine whether the current ReservationId has a
corresponding subCluster.
// If it does not exist, add it. If it exists, update it.
Boolean exists = existsReservationHomeSubCluster(reservationId);
-if (!exists) {
+if (!exists || i == 0) {
Review Comment:
Reservation may be repeatedly submitted (the same reservationId is submitted
repeatedly). At this time, we should try to use the first reservation result.
If the first reservation encounters some abnormalities, find another RM for
reservation.
> Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
> -
>
> Key: YARN-11290
> URL: https://issues.apache.org/jira/browse/YARN-11290
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: SqlServer-StoredProc.png
>
>
> 1. Currently this interface returns the number of apps in all sub-clusters,
> increasing the limit on the number of query apps, limited to 1000 apps.
> 2. Allows to query the App based on the specified HomeSubCluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[
https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601047#comment-17601047
]
ASF GitHub Bot commented on YARN-11290:
---
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r964250673
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/GetApplicationsHomeSubClusterRequest.java:
##
@@ -37,4 +38,33 @@ public static GetApplicationsHomeSubClusterRequest
newInstance() {
return request;
}
+ @Private
+ @Unstable
+ public static GetApplicationsHomeSubClusterRequest
+ newInstance(SubClusterId subClusterId) {
+GetApplicationsHomeSubClusterRequest request =
+Records.newRecord(GetApplicationsHomeSubClusterRequest.class);
+request.setSubClusterId(subClusterId);
+return request;
+ }
+
+ /**
+ * Get the {@link SubClusterId} representing the unique identifier of the
+ * subcluster.
+ *
+ * @return the subcluster identifier
+ */
+ @InterfaceAudience.Public
Review Comment:
I will fix it.
> Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
> -
>
> Key: YARN-11290
> URL: https://issues.apache.org/jira/browse/YARN-11290
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: SqlServer-StoredProc.png
>
>
> 1. Currently this interface returns the number of apps in all sub-clusters,
> increasing the limit on the number of query apps, limited to 1000 apps.
> 2. Allows to query the App based on the specified HomeSubCluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[
https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601046#comment-17601046
]
ASF GitHub Bot commented on YARN-11290:
---
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r964250594
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/ZookeeperFederationStateStore.java:
##
@@ -255,24 +261,52 @@ public GetApplicationHomeSubClusterResponse
getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
-long start = clock.getTime();
-List result = new ArrayList<>();
+
+if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+}
try {
- for (String child : zkManager.getChildren(appsZNode)) {
-ApplicationId appId = ApplicationId.fromString(child);
-SubClusterId homeSubCluster = getApp(appId);
-ApplicationHomeSubCluster app =
-ApplicationHomeSubCluster.newInstance(appId, homeSubCluster);
-result.add(app);
- }
+ long start = clock.getTime();
+ SubClusterId requestSC = request.getSubClusterId();
+ List children = zkManager.getChildren(appsZNode);
+ List result =
+ children.stream().map(child -> generateAppHomeSC(child))
+ .filter(appHomeSC -> judgeAdd(requestSC,
appHomeSC.getHomeSubCluster()))
+ .limit(maxAppsInStateStore)
+ .collect(Collectors.toList());
+ long end = clock.getTime();
+ opDurations.addGetAppsHomeSubClusterDuration(start, end);
+ LOG.info("filterSubClusterId = {}, appCount = {}.", requestSC,
result.size());
+ return GetApplicationsHomeSubClusterResponse.newInstance(result);
} catch (Exception e) {
String errMsg = "Cannot get apps: " + e.getMessage();
FederationStateStoreUtils.logAndThrowStoreException(LOG, errMsg);
}
-long end = clock.getTime();
-opDurations.addGetAppsHomeSubClusterDuration(start, end);
-return GetApplicationsHomeSubClusterResponse.newInstance(result);
+
+throw new YarnException("Cannot get app by request");
+ }
+
+ private ApplicationHomeSubCluster generateAppHomeSC(String appId) {
+try {
+ ApplicationId applicationId = ApplicationId.fromString(appId);
+ SubClusterId homeSubCluster = getApp(applicationId);
+ ApplicationHomeSubCluster app =
+ ApplicationHomeSubCluster.newInstance(applicationId, homeSubCluster);
+ return app;
+} catch (Exception ex) {
+ LOG.error("get homeSubCluster by appId = {}.", appId);
+}
+return null;
+ }
+
+ private boolean judgeAdd(SubClusterId filterSubCluster, SubClusterId
homeSubCluster) {
Review Comment:
I will re-select an appropriate method name.
> Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
> -
>
> Key: YARN-11290
> URL: https://issues.apache.org/jira/browse/YARN-11290
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: SqlServer-StoredProc.png
>
>
> 1. Currently this interface returns the number of apps in all sub-clusters,
> increasing the limit on the number of query apps, limited to 1000 apps.
> 2. Allows to query the App based on the specified HomeSubCluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[
https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601045#comment-17601045
]
ASF GitHub Bot commented on YARN-11290:
---
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r964250393
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##
@@ -255,17 +261,36 @@ public GetApplicationHomeSubClusterResponse
getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
-List result =
-new ArrayList();
-for (Entry e : applications.entrySet()) {
- result
- .add(ApplicationHomeSubCluster.newInstance(e.getKey(),
e.getValue()));
+
+if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
}
-GetApplicationsHomeSubClusterResponse.newInstance(result);
+SubClusterId requestSC = request.getSubClusterId();
+List result = applications.keySet().stream()
+.map(applicationId -> generateAppHomeSC(applicationId))
+.filter(appHomeSC -> judgeAdd(requestSC,
appHomeSC.getHomeSubCluster()))
+.limit(maxAppsInStateStore)
+.collect(Collectors.toList());
+
+LOG.info("filterSubClusterId = {}, appCount = {}.", requestSC,
result.size());
return GetApplicationsHomeSubClusterResponse.newInstance(result);
}
+ private ApplicationHomeSubCluster generateAppHomeSC(ApplicationId
applicationId) {
+SubClusterId subClusterId = applications.get(applicationId);
+return ApplicationHomeSubCluster.newInstance(applicationId, subClusterId);
+ }
+
+ private boolean judgeAdd(SubClusterId filterSubCluster, SubClusterId
homeSubCluster) {
+if (filterSubCluster == null) {
+ return true;
+} else if (filterSubCluster.equals(homeSubCluster)) {
Review Comment:
Thanks for your help reviewing the code, I will modify the code.
> Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
> -
>
> Key: YARN-11290
> URL: https://issues.apache.org/jira/browse/YARN-11290
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: SqlServer-StoredProc.png
>
>
> 1. Currently this interface returns the number of apps in all sub-clusters,
> increasing the limit on the number of query apps, limited to 1000 apps.
> 2. Allows to query the App based on the specified HomeSubCluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[
https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601041#comment-17601041
]
ASF GitHub Bot commented on YARN-11290:
---
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1238739439
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 24s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 22s | | trunk passed |
| +1 :green_heart: | compile | 10m 23s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 7s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 9m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 47s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 56s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 24s | | the patch passed |
| +1 :green_heart: | compile | 9m 16s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 16s | | the patch passed |
| +1 :green_heart: | javac | 9m 16s | | the patch passed |
| +1 :green_heart: | compile | 8m 35s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 35s | | the patch passed |
| +1 :green_heart: | javac | 8m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 55s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 46s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 9s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 238m 58s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 1m 34s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 22s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 29s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 4m 27s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 17s | | The patch does not
generate ASF License warnings. |
| | | 469m 17s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc
buflint bufcompat |
| uname | Linux d6d029dc8f61 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
|
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[
https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600963#comment-17600963
]
ASF GitHub Bot commented on YARN-11290:
---
goiri commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r964122414
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/ZookeeperFederationStateStore.java:
##
@@ -255,24 +261,52 @@ public GetApplicationHomeSubClusterResponse
getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
-long start = clock.getTime();
-List result = new ArrayList<>();
+
+if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+}
try {
- for (String child : zkManager.getChildren(appsZNode)) {
-ApplicationId appId = ApplicationId.fromString(child);
-SubClusterId homeSubCluster = getApp(appId);
-ApplicationHomeSubCluster app =
-ApplicationHomeSubCluster.newInstance(appId, homeSubCluster);
-result.add(app);
- }
+ long start = clock.getTime();
+ SubClusterId requestSC = request.getSubClusterId();
+ List children = zkManager.getChildren(appsZNode);
+ List result =
+ children.stream().map(child -> generateAppHomeSC(child))
+ .filter(appHomeSC -> judgeAdd(requestSC,
appHomeSC.getHomeSubCluster()))
+ .limit(maxAppsInStateStore)
+ .collect(Collectors.toList());
+ long end = clock.getTime();
+ opDurations.addGetAppsHomeSubClusterDuration(start, end);
+ LOG.info("filterSubClusterId = {}, appCount = {}.", requestSC,
result.size());
+ return GetApplicationsHomeSubClusterResponse.newInstance(result);
} catch (Exception e) {
String errMsg = "Cannot get apps: " + e.getMessage();
FederationStateStoreUtils.logAndThrowStoreException(LOG, errMsg);
}
-long end = clock.getTime();
-opDurations.addGetAppsHomeSubClusterDuration(start, end);
-return GetApplicationsHomeSubClusterResponse.newInstance(result);
+
+throw new YarnException("Cannot get app by request");
+ }
+
+ private ApplicationHomeSubCluster generateAppHomeSC(String appId) {
+try {
+ ApplicationId applicationId = ApplicationId.fromString(appId);
+ SubClusterId homeSubCluster = getApp(applicationId);
+ ApplicationHomeSubCluster app =
+ ApplicationHomeSubCluster.newInstance(applicationId, homeSubCluster);
+ return app;
+} catch (Exception ex) {
+ LOG.error("get homeSubCluster by appId = {}.", appId);
+}
+return null;
+ }
+
+ private boolean judgeAdd(SubClusterId filterSubCluster, SubClusterId
homeSubCluster) {
Review Comment:
`judgeAdd` is a weird name fot this function.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/FederationStateStoreBaseTest.java:
##
@@ -410,6 +412,89 @@ public void testGetApplicationsHomeSubCluster() throws
Exception {
Assert.assertTrue(result.getAppsHomeSubClusters().contains(ahsc2));
}
+ @Test
+ public void testGetApplicationsHomeSubClusterEmpty() throws Exception {
+LambdaTestUtils.intercept(YarnException.class,
+"Missing getApplicationsHomeSubCluster request",
+() -> stateStore.getApplicationsHomeSubCluster(null));
+ }
+
+ @Test
+ public void testGetApplicationsHomeSubClusterFilter() throws Exception {
+// Add ApplicationHomeSC - SC1
+long now = Time.now();
+
+Set appHomeSubClusters = new HashSet<>();
+
+for (int i = 0; i < 10; i++) {
Review Comment:
Make 10 and 20 constants.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/clientrm/TestableFederationClientInterceptor.java:
##
@@ -90,6 +95,7 @@ protected ApplicationClientProtocol
getClientRMProxyForSubCluster(
mockRMs.put(subClusterId, mockRM);
}
initNodeAttributes(subClusterId, mockRM);
+ initReservationSystem(mockRM);
Review Comment:
This seems out of scope.
Should we do it in a separate PR?
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##
@@ -255,17 +261,36 @@ public GetApplicationHomeSubClusterResponse
getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
-List result =
-n
[jira] [Commented] (YARN-9019) Ratio calculation of ResourceCalculator implementations could return NaN
[
https://issues.apache.org/jira/browse/YARN-9019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600907#comment-17600907
]
Eric Payne commented on YARN-9019:
--
I backported this to branch-3.2 and branch-2.10
> Ratio calculation of ResourceCalculator implementations could return NaN
>
>
> Key: YARN-9019
> URL: https://issues.apache.org/jira/browse/YARN-9019
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Fix For: 3.3.0, 2.10.3, 3.2.5
>
> Attachments: YARN-9019.001.patch
>
>
> Found out that ResourceCalculator.ratio (with implementors
> DefaultResourceCalculator and DominantResourceCalculator) can produce NaN
> (Not-A-Number) as a result.
> This is because [IEEE 754|http://grouper.ieee.org/groups/754/] defines {{1.0
> / 0.0}} as Infinity and {{-1.0 / 0.0}} as -Infinity and {{0.0 / 0.0}} as NaN,
> see here: [https://stackoverflow.com/a/14138032/1106893]
> I think it's very dangerous to rely on NaN can be returned from ratio
> calculations and this could have side-effects.
> When ratio calculates the result and if both the numerator and the
> denominator is zero, we should use 0 as a result, I think.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9019) Ratio calculation of ResourceCalculator implementations could return NaN
[
https://issues.apache.org/jira/browse/YARN-9019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated YARN-9019:
-
Fix Version/s: 2.10.3
3.2.5
> Ratio calculation of ResourceCalculator implementations could return NaN
>
>
> Key: YARN-9019
> URL: https://issues.apache.org/jira/browse/YARN-9019
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Fix For: 3.3.0, 2.10.3, 3.2.5
>
> Attachments: YARN-9019.001.patch
>
>
> Found out that ResourceCalculator.ratio (with implementors
> DefaultResourceCalculator and DominantResourceCalculator) can produce NaN
> (Not-A-Number) as a result.
> This is because [IEEE 754|http://grouper.ieee.org/groups/754/] defines {{1.0
> / 0.0}} as Infinity and {{-1.0 / 0.0}} as -Infinity and {{0.0 / 0.0}} as NaN,
> see here: [https://stackoverflow.com/a/14138032/1106893]
> I think it's very dangerous to rely on NaN can be returned from ratio
> calculations and this could have side-effects.
> When ratio calculates the result and if both the numerator and the
> denominator is zero, we should use 0 as a result, I think.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[
https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600886#comment-17600886
]
ASF GitHub Bot commented on YARN-11290:
---
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1238405786
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 50s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 25s | | trunk passed |
| +1 :green_heart: | compile | 10m 36s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 46s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 7s | | trunk passed |
| +1 :green_heart: | mvnsite | 9m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 35s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 40s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 44s | | the patch passed |
| +1 :green_heart: | compile | 10m 19s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 10m 19s | | the patch passed |
| +1 :green_heart: | javac | 10m 19s | | the patch passed |
| +1 :green_heart: | compile | 9m 22s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 9m 22s | | the patch passed |
| +1 :green_heart: | javac | 9m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 54s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 19s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 239m 38s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 1m 34s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 33s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 32s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 4m 12s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 21s | | The patch does not
generate ASF License warnings. |
| | | 474m 49s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/9/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle xmllint cc
buflint bufcompat |
| uname | Linux d11f227539ba 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
|
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[ https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600835#comment-17600835 ] ASF GitHub Bot commented on YARN-11290: --- slfan1989 commented on PR #4846: URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1238308788 @goiri Please help to review the code again, thank you very much! This PR changes the execution script of the stored procedure, I will check the script against the major version of the database. **SQLServer** The change script of the stored procedure has been verified on SQL Server 2012 Enterprise And SQL Server 2008 R2 Enterprise, I will verify it on the following version as soon as possible - [ ] SQL Server 2008 R2 Enterprise - [ ] SQL Server 2012 Enterprise SQL Server 2016 Enterprise SQL Server 2017 Enterprise SQL Server 2019 Enterprise **MySQL** MySQL 5.7 and above must be required, MySQL 5.5 and MySQL 5.6 cannot be supported. MySQL-5.5 and MySQL-5.6 Can't Create Table membership and policies, for the following reasons: 1.The primary key of the membership table is subClusterId varchar(256) , Mysql will prompt the following error: Specified key was too long; max key length is 767 bytes. 2.The primary key of the policies table is subClusterId varchar(256) , Mysql will prompt the following error: Specified key was too long; max key length is 767 bytes. - [ ] MySQL 5.7 - [ ] MySQL 8.0 > Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster > - > > Key: YARN-11290 > URL: https://issues.apache.org/jira/browse/YARN-11290 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Attachments: SqlServer-StoredProc.png > > > 1. Currently this interface returns the number of apps in all sub-clusters, > increasing the limit on the number of query apps, limited to 1000 apps. > 2. Allows to query the App based on the specified HomeSubCluster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10935) AM Total Queue Limit goes below per-user AM Limit if parent is full.
[
https://issues.apache.org/jira/browse/YARN-10935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-10935:
--
Description:
This happens when DRF is enabled and all of one resource is consumed but the
second resources still has plenty available.
This is reproduceable by setting up a parent queue where the capacity and max
capacity are the same, with 2 or more sub-queues whose max capacity is 100%.
In one of the sub-queues, start a long-running app that consumes all resources
in the parent queue's hierarchy. This app will consume all of the memory but
not vary many vcores (for example)
In a second queue, submit an app. The *{{Max Application Master Resources Per
User}}* limit is much more than the *{{Max Application Master Resources}}*
limit.
was:
This happens when DRF is enabled and all of one resource is consumed but the
second resources still has plenty available.
This is reproduceable by setting up a parent queue where the capacity and max
capacity are the same, with 2 or more sub-queues whose max capacity is 100%.
In one of the sub-queues, start a long-running app that consumes all resources
in the parent queue's hieararchy. This app will consume all of the memory but
not vary many vcores (for example)
In a second queue, submit an app. The *{{Max Application Master Resources Per
User}}* limit is much more than the *{{Max Application Master Resources}}*
limit.
> AM Total Queue Limit goes below per-user AM Limit if parent is full.
>
>
> Key: YARN-10935
> URL: https://issues.apache.org/jira/browse/YARN-10935
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler, capacityscheduler
>Reporter: Eric Payne
>Assignee: Eric Payne
>Priority: Major
> Fix For: 3.4.0, 2.10.2, 3.3.2, 3.2.4, 3.1.5
>
> Attachments: Screen Shot 2021-09-07 at 12.49.52 PM.png, Screen Shot
> 2021-09-07 at 12.55.37 PM.png, YARN-10935.001.patch, YARN-10935.002.patch,
> YARN-10935.003.patch, YARN-10935.branch-2.10.003.patch,
> YARN-10935.branch-3.2.003.patch
>
>
> This happens when DRF is enabled and all of one resource is consumed but the
> second resources still has plenty available.
> This is reproduceable by setting up a parent queue where the capacity and max
> capacity are the same, with 2 or more sub-queues whose max capacity is 100%.
> In one of the sub-queues, start a long-running app that consumes all
> resources in the parent queue's hierarchy. This app will consume all of the
> memory but not vary many vcores (for example)
> In a second queue, submit an app. The *{{Max Application Master Resources Per
> User}}* limit is much more than the *{{Max Application Master Resources}}*
> limit.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[
https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600789#comment-17600789
]
ASF GitHub Bot commented on YARN-11290:
---
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1238215896
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 40s | | trunk passed |
| +1 :green_heart: | compile | 10m 12s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 3s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 9m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 43s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 39s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 55s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 24s | | the patch passed |
| +1 :green_heart: | compile | 9m 16s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 16s | | the patch passed |
| +1 :green_heart: | javac | 9m 16s | | the patch passed |
| +1 :green_heart: | compile | 8m 54s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 54s | | the patch passed |
| +1 :green_heart: | javac | 8m 54s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 54s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/8/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 165 unchanged
- 0 fixed = 168 total (was 165) |
| +1 :green_heart: | mvnsite | 8m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 49s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 20s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 47s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 238m 30s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 1m 33s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 28s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 35s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 4m 24s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 23s | | The patch does not
generate ASF License warnings. |
| | | 472m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvns
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600700#comment-17600700
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1237921393
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | shadedclient | 32m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | shadedclient | 22m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 57m 58s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense codespell detsecrets |
| uname | Linux 3b0c251514a6 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / afba705768dd88e1453d451eaeb8e44856156309 |
| Max. process+thread count | 600 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/2/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[
https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600655#comment-17600655
]
ASF GitHub Bot commented on YARN-11290:
---
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1237793763
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 50s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 27s | | trunk passed |
| +1 :green_heart: | compile | 9m 53s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 8m 39s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 29s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 42s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 24s | | the patch passed |
| +1 :green_heart: | compile | 9m 15s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 15s | | the patch passed |
| +1 :green_heart: | javac | 9m 15s | | the patch passed |
| +1 :green_heart: | compile | 8m 36s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 8m 36s | | the patch passed |
| +1 :green_heart: | javac | 8m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 50s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 165 unchanged
- 0 fixed = 168 total (was 165) |
| +1 :green_heart: | mvnsite | 8m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 50s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 9s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 18m 32s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 28s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 238m 41s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 1m 31s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 40s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 35s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 4m 10s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 22s | | The patch does not
generate ASF License warnings. |
| | | 468m 31s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4846 |
| Optional Tests | dupname asflicense codespell detsecrets compile javac
javadoc mvninstall mvns
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17600637#comment-17600637
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1237763399
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
_ trunk Compile Tests _ |
| +1 :green_heart: | shadedclient | 33m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | shadedclient | 21m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 59m 15s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense codespell detsecrets |
| uname | Linux ac4fa9e3546f 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2512f043c415fbfc85d6d3971a84defc04d01532 |
| Max. process+thread count | 534 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn U:
hadoop-yarn-project/hadoop-yarn |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING
