[jira] [Commented] (YARN-11247) Remove unused classes introduced by YARN-9615
[
https://issues.apache.org/jira/browse/YARN-11247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605124#comment-17605124
]
ASF GitHub Bot commented on YARN-11247:
---
hadoop-yetus commented on PR #4720:
URL: https://github.com/apache/hadoop/pull/4720#issuecomment-1247613927
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 16s | | trunk passed |
| +1 :green_heart: | compile | 1m 4s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 54s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 6s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 29s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 45s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 4m 53s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 102m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4720/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4720 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 729ae4e40f8a 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0e74c16a1bb127b3d17b650babc55220d8cb2685 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4720/5/testReport/ |
| Max. process+thread count | 555 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4720/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
|
[jira] [Commented] (YARN-11293) [Federation] Router Support DelegationToken storeNewMasterKey/removeStoredMasterKey With MemoryStateStore
[
https://issues.apache.org/jira/browse/YARN-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605114#comment-17605114
]
ASF GitHub Bot commented on YARN-11293:
---
hadoop-yetus commented on PR #4852:
URL: https://github.com/apache/hadoop/pull/4852#issuecomment-1247588629
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 11s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 18s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 52s | | trunk passed |
| +1 :green_heart: | compile | 4m 0s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 3m 25s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 32s | | the patch passed |
| +1 :green_heart: | compile | 3m 51s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 3m 51s | | the patch passed |
| -1 :x: | javac | 3m 51s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4852/7/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 1 new + 447
unchanged - 0 fixed = 448 total (was 447) |
| +1 :green_heart: | compile | 3m 12s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 3m 12s | | the patch passed |
| -1 :x: | javac | 3m 12s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4852/7/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 1 new
+ 371 unchanged - 0 fixed = 372 total (was 371) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 31s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 10s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 102m 48s | |
hadoop-yarn-server-resourcemanager in the patch
[jira] [Commented] (YARN-11255) Support loading alternative docker client config from system environment
[
https://issues.apache.org/jira/browse/YARN-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605097#comment-17605097
]
ASF GitHub Bot commented on YARN-11255:
---
PrabhuJoseph commented on PR #4884:
URL: https://github.com/apache/hadoop/pull/4884#issuecomment-1247563160
Thanks @ashutoshcipher for the patch. Overall looks good to me except a
minor one.
Could you also document the steps to run an application like Spark using
this new environment variable YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG in
https://hadoop.apache.org/docs/r3.0.2/hadoop-yarn/hadoop-yarn-site/DockerContainers.html
For Example:
```
DOCKER_IMAGE_NAME=hadoop-docker
DOCKER_CLIENT_CONFIG=hdfs:///user/hadoop/config.json
spark-submit --master yarn \
--deploy-mode cluster \
--conf spark.executorEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf
spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
--conf
spark.executorEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
--conf spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_TYPE=docker \
--conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=$DOCKER_IMAGE_NAME \
--conf
spark.yarn.appMasterEnv.YARN_CONTAINER_RUNTIME_DOCKER_CLIENT_CONFIG=$DOCKER_CLIENT_CONFIG
\
sparkR.R
```
> Support loading alternative docker client config from system environment
>
>
> Key: YARN-11255
> URL: https://issues.apache.org/jira/browse/YARN-11255
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
>
> When using YARN docker support, although the hadoop shell supported
> {code:java}
> -docker_client_config{code}
> to pass the client config file that contains security token to generate the
> docker config for each job as a temporary file.
> For other applications that submit jobs to YARN, e.g. Spark, which loads the
> docker setting via system environment e.g.
> {code:java}
> spark.executorEnv.* {code}
> will not be able to add those authorization token because this system
> environment isn't considered in YARN.
> Add genetic solution to handle these kind of cases without making changes in
> spark code or others
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11255) Support loading alternative docker client config from system environment
[
https://issues.apache.org/jira/browse/YARN-11255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605096#comment-17605096
]
ASF GitHub Bot commented on YARN-11255:
---
PrabhuJoseph commented on code in PR #4884:
URL: https://github.com/apache/hadoop/pull/4884#discussion_r971506094
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java:
##
@@ -215,6 +217,12 @@ public class DockerLinuxContainerRuntime extends
OCIContainerRuntime {
public static final String ENV_DOCKER_CONTAINER_IMAGE =
"YARN_CONTAINER_RUNTIME_DOCKER_IMAGE";
@InterfaceAudience.Private
+ public static final String ENV_DOCKER_CONTAINER_IMAGE_FILE =
Review Comment:
Thanks @ashutoshcipher for the patch. This variable is not used anywhere.
Could you remove it if not needed.
> Support loading alternative docker client config from system environment
>
>
> Key: YARN-11255
> URL: https://issues.apache.org/jira/browse/YARN-11255
> Project: Hadoop YARN
> Issue Type: New Feature
>Reporter: Ashutosh Gupta
>Assignee: Ashutosh Gupta
>Priority: Major
> Labels: pull-request-available
>
> When using YARN docker support, although the hadoop shell supported
> {code:java}
> -docker_client_config{code}
> to pass the client config file that contains security token to generate the
> docker config for each job as a temporary file.
> For other applications that submit jobs to YARN, e.g. Spark, which loads the
> docker setting via system environment e.g.
> {code:java}
> spark.executorEnv.* {code}
> will not be able to add those authorization token because this system
> environment isn't considered in YARN.
> Add genetic solution to handle these kind of cases without making changes in
> spark code or others
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11226) [Federation] Add createNewReservation, submitReservation, updateReservation, deleteReservation REST APIs for Router
[
https://issues.apache.org/jira/browse/YARN-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605094#comment-17605094
]
ASF GitHub Bot commented on YARN-11226:
---
hadoop-yetus commented on PR #4851:
URL: https://github.com/apache/hadoop/pull/4851#issuecomment-1247552465
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 46s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 33s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 15s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 0s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 22s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 26s | | the patch passed |
| +1 :green_heart: | compile | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 0s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 57s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 106m 46s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4851/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4851 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4bf7044ffc29 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 73b27732933c3983ed7bad65708177af22513b31 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4851/8/testReport/ |
| Max. process+thread count | 711 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[
https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605028#comment-17605028
]
ASF GitHub Bot commented on YARN-11290:
---
slfan1989 commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r971434340
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/ZookeeperFederationStateStore.java:
##
@@ -255,24 +263,54 @@ public GetApplicationHomeSubClusterResponse
getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
-long start = clock.getTime();
-List result = new ArrayList<>();
+
+if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+}
try {
- for (String child : zkManager.getChildren(appsZNode)) {
-ApplicationId appId = ApplicationId.fromString(child);
-SubClusterId homeSubCluster = getApp(appId);
-ApplicationHomeSubCluster app =
-ApplicationHomeSubCluster.newInstance(appId, homeSubCluster);
-result.add(app);
- }
+ long start = clock.getTime();
+ SubClusterId requestSC = request.getSubClusterId();
+ List children = zkManager.getChildren(appsZNode);
+ List result =
+ children.stream().map(child -> generateAppHomeSC(child))
+
.sorted(Comparator.comparing(ApplicationHomeSubCluster::getCreateTime).reversed())
Review Comment:
Thanks for your suggestion, I will fix it.
> Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
> -
>
> Key: YARN-11290
> URL: https://issues.apache.org/jira/browse/YARN-11290
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Minor
> Labels: pull-request-available
> Attachments: SqlServer-StoredProc.png
>
>
> 1. Currently this interface returns the number of apps in all sub-clusters,
> increasing the limit on the number of query apps, limited to 1000 apps.
> 2. Allows to query the App based on the specified HomeSubCluster.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11293) [Federation] Router Support DelegationToken storeNewMasterKey/removeStoredMasterKey With MemoryStateStore
[
https://issues.apache.org/jira/browse/YARN-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605025#comment-17605025
]
ASF GitHub Bot commented on YARN-11293:
---
slfan1989 commented on code in PR #4852:
URL: https://github.com/apache/hadoop/pull/4852#discussion_r971429570
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterMasterKey.java:
##
@@ -0,0 +1,131 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceAudience.Public;
+import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.hadoop.yarn.util.Records;
+
+import java.nio.ByteBuffer;
+
+@Private
+@Unstable
+public abstract class RouterMasterKey {
+
+ @Private
+ @Unstable
+ public static RouterMasterKey newInstance(Integer keyId, ByteBuffer
keyBytes, Long expiryDate) {
+RouterMasterKey policy = Records.newRecord(RouterMasterKey.class);
+policy.setKeyId(keyId);
+policy.setKeyBytes(keyBytes);
+policy.setExpiryDate(expiryDate);
+return policy;
+ }
+
+ @Private
+ @Unstable
+ public static RouterMasterKey newInstance(RouterMasterKey masterKey) {
+RouterMasterKey routerMasterKey = Records.newRecord(RouterMasterKey.class);
+routerMasterKey.setKeyId(masterKey.getKeyId());
+routerMasterKey.setKeyBytes(masterKey.getKeyBytes());
+routerMasterKey.setExpiryDate(masterKey.getExpiryDate());
+return routerMasterKey;
+ }
+
+ /**
+ * Get the keyId of the MasterKey.
+ *
+ * @return MasterKeyId.
+ */
+ @Public
+ @Unstable
+ public abstract Integer getKeyId();
+
+ /**
+ * Set the keyId of the MasterKey.
+ *
+ * @param keyId MasterKeyId.
+ */
+ @Private
+ @Unstable
+ public abstract void setKeyId(Integer keyId);
+
+ /**
+ * Get the keyBytes of the DelegationKey.
+ *
+ * @return KeyBytes of the DelegationKey.
+ */
+ @Public
+ @Unstable
+ public abstract ByteBuffer getKeyBytes();
+
+ /**
+ * Set the keyBytes of the DelegationKey.
+ *
+ * @param keyBytes KeyBytes of the DelegationKey.
+ */
+ @Private
+ @Unstable
+ public abstract void setKeyBytes(ByteBuffer keyBytes);
+
+ /**
+ * Get the ExpiryDate of the DelegationKey.
+ *
+ * @return ExpiryDate of the DelegationKey.
+ */
+ @Private
+ @Unstable
+ public abstract Long getExpiryDate();
+
+ /**
+ * Set the expiryDate of the DelegationKey.
+ *
+ * @param expiryDate expiryDate of the DelegationKey.
+ */
+ @Private
+ @Unstable
+ public abstract void setExpiryDate(Long expiryDate);
+
+ @Override
+ public int hashCode() {
+return new HashCodeBuilder()
+.append(this.getExpiryDate().longValue())
+.append(this.getKeyId().intValue())
+.append(getKeyBytes().array())
+.hashCode();
+ }
+
+ @Override
+ public boolean equals(Object right) {
+if (this == right) {
+ return true;
+} else if (right == null || getClass() != right.getClass()) {
Review Comment:
I will fix it.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterMasterKey.java:
##
@@ -0,0 +1,131 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605022#comment-17605022
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r971426352
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -342,11 +353,20 @@ public void init(Configuration conf) {
conn.prepareStatement(SP_UPDATERESERVATIONHOMESUBCLUSTER).execute();
LOG.info("Database Init: Complete");
-} catch (SQLException e) {
+} catch (Exception e) {
LOG.error("ERROR: failed to inizialize HSQLDB " + e.getMessage());
}
}
+ public void initConnection(Configuration conf) {
+try {
+ super.init(conf);
+ conn = super.conn;
+} catch (YarnException e1) {
+ LOG.error("ERROR: failed to init HSQLDB " + e1.getMessage());
Review Comment:
I will fix it.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -356,4 +376,31 @@ public void closeConnection() {
}
}
+ /**
+ * Extract The Create Table Sql From The Script.
+ *
+ * @param dbIdentifier database identifier, Like Mysql / SqlServer
+ * @param regex the regex
+ * @throws IOException IO exception.
+ */
+ protected void extractCreateTableSQL(String dbIdentifier, String regex)
throws IOException {
+String createTableScriptPath = "." + File.separator + "target" +
File.separator +
+"test-classes" + File.separator + dbIdentifier +
"/FederationStateStoreTables.sql";
+String createTableSQL =
+FileUtils.readFileToString(new File(createTableScriptPath),
StandardCharsets.UTF_8);
+Pattern p = Pattern.compile(regex);
+Matcher m = p.matcher(createTableSQL);
+while(m!=null && m.find()) {
Review Comment:
I will fix it.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605019#comment-17605019
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r971425926
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -356,4 +376,31 @@ public void closeConnection() {
}
}
+ /**
+ * Extract The Create Table Sql From The Script.
+ *
+ * @param dbIdentifier database identifier, Like Mysql / SqlServer
+ * @param regex the regex
+ * @throws IOException IO exception.
+ */
+ protected void extractCreateTableSQL(String dbIdentifier, String regex)
throws IOException {
+String createTableScriptPath = "." + File.separator + "target" +
File.separator +
Review Comment:
Thanks for your suggestion, I will modify the code.
I found that if I use string.format, the code readability doesn't seem to be
very good
```
String.format(".%starget%stest-classes%s%s%sFederationStateStoreTables.sql");
```
I think it might be better to write the following.
```
String[] createTableScriptPathItem = new String[] {
".", "target", "test-classes", dbIdentifier,
"FederationStateStoreTables.sql" };
String createTableScriptPath = StringUtils.join(createTableScriptPathItem,
File.separator);
```
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605016#comment-17605016
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r971414849
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -342,11 +353,20 @@ public void init(Configuration conf) {
conn.prepareStatement(SP_UPDATERESERVATIONHOMESUBCLUSTER).execute();
LOG.info("Database Init: Complete");
-} catch (SQLException e) {
+} catch (Exception e) {
LOG.error("ERROR: failed to inizialize HSQLDB " + e.getMessage());
Review Comment:
I will fix it.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605001#comment-17605001
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r971358346
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -301,14 +309,17 @@ public class HSQLDBFederationStateStore extends
SQLFederationStateStore {
+ " WHERE reservationId = reservationId_IN;"
+ " SET rowCount_OUT = 2; END";
+ private List tables = new ArrayList<>();
@Override
public void init(Configuration conf) {
+
try {
super.init(conf);
} catch (YarnException e1) {
LOG.error("ERROR: failed to init HSQLDB " + e1.getMessage());
}
+
Review Comment:
I will fix it.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17605000#comment-17605000
]
ASF GitHub Bot commented on YARN-11296:
---
slfan1989 commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r971358259
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -301,14 +309,17 @@ public class HSQLDBFederationStateStore extends
SQLFederationStateStore {
+ " WHERE reservationId = reservationId_IN;"
+ " SET rowCount_OUT = 2; END";
+ private List tables = new ArrayList<>();
@Override
public void init(Configuration conf) {
+
Review Comment:
Thanks for your help reviewing the code, I will fix it!
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -301,14 +309,17 @@ public class HSQLDBFederationStateStore extends
SQLFederationStateStore {
+ " WHERE reservationId = reservationId_IN;"
+ " SET rowCount_OUT = 2; END";
+ private List tables = new ArrayList<>();
@Override
public void init(Configuration conf) {
+
Review Comment:
Thanks for your help reviewing the code, I will fix it.
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
> Attachments: Improve Query Condition of
> FederationStateStore#getApplicationsHomeSubCluster Test Report.pdf
>
>
> *MySQL:*
> FederationStateStoreTables.sql cannot create *membership* table, both in
> MySQL 5.7 and MySQL 8.0.
>
> {code:java}
> CREATE TABLE membership(
>subClusterId varchar(256) NOT NULL,
>amRMServiceAddress varchar(256) NOT NULL,
>clientRMServiceAddress varchar(256) NOT NULL,
>rmAdminServiceAddress varchar(256) NOT NULL,
>rmWebServiceAddress varchar(256) NOT NULL,
>lastHeartBeat datetime NOT NULL,
>state varchar(32) NOT NULL,
>lastStartTime bigint NULL,
>capability varchar(6000),
>CONSTRAINT pk_subClusterId PRIMARY KEY (subClusterId), -- missing comma
> UNIQUE(lastStartTime)
> ); {code}
>
> *SQLServer:*
> FederationStateStoreTables.sql
>
> {code:java}
> IF NOT EXISTS ( SELECT * FROM [FederationStateStore].sys.tables
> WHERE name = 'membership'
> AND schema_id = SCHEMA_ID('dbo'))
> BEGIN
> PRINT 'Table membership does not exist, create it...'
> SET ANSI_NULLS ON
> SET QUOTED_IDENTIFIER ON
> SET ANSI_PADDING ON
> CREATE TABLE [dbo].[membership](
> [subClusterId]VARCHAR(256) COLLATE
> Latin1_General_100_BIN2 NOT NULL,
> [amRMServiceAddress] VARCHAR(256) NOT NULL,
> [clientRMServiceAddress] VARCHAR(256) NOT NULL,
> [rmAdminServiceAddress] VARCHAR(256) NOT NULL,
> [rmWebServiceAddress] VARCHAR(256) NOT NULL,
> [lastHeartBeat] DATETIME2 NOT NULL,
> [state] VARCHAR(32) NOT NULL,
> [lastStartTime] BIGINT NOT NULL,
> [capability] VARCHAR(6000) NOT NULL,
> CONSTRAINT [pk_subClusterId] PRIMARY KEY
> (
> [subClusterId]
> ), -- missing comma
> CONSTRAINT [uc_lastStartTime] UNIQUE
> (
> [lastStartTime]
> )
> )
> SET ANSI_PADDING OFF
> PRINT 'Table membership created.'
> END
> ELSE
> PRINT 'Table membership exists, no operation required...'
> GO
> GO {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11293) [Federation] Router Support DelegationToken storeNewMasterKey/removeStoredMasterKey With MemoryStateStore
[
https://issues.apache.org/jira/browse/YARN-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17604934#comment-17604934
]
ASF GitHub Bot commented on YARN-11293:
---
hadoop-yetus commented on PR #4852:
URL: https://github.com/apache/hadoop/pull/4852#issuecomment-1247215147
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 54s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 53s | | trunk passed |
| +1 :green_heart: | compile | 4m 14s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 3m 27s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 59s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 31s | | the patch passed |
| +1 :green_heart: | compile | 4m 2s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 4m 2s | | the patch passed |
| -1 :x: | javac | 4m 2s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4852/6/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
with JDK Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 generated 1 new + 447
unchanged - 0 fixed = 448 total (was 447) |
| +1 :green_heart: | compile | 3m 18s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 3m 18s | | the patch passed |
| -1 :x: | javac | 3m 18s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4852/6/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 generated 1 new
+ 371 unchanged - 0 fixed = 372 total (was 371) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 11s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4852/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 3
new + 11 unchanged - 0 fixed = 14 total (was 11) |
| +1 :green_heart: | mvnsite | 1m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 14s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 41s | | the patch passed |
| +1 :green_heart:
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[
https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17604928#comment-17604928
]
ASF GitHub Bot commented on YARN-11290:
---
hadoop-yetus commented on PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#issuecomment-1247192785
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 39s | | trunk passed |
| +1 :green_heart: | compile | 10m 43s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 9m 1s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 8m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 6m 0s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 5m 7s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 57s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 21s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 7m 23s | | the patch passed |
| +1 :green_heart: | compile | 9m 52s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | cc | 9m 52s | | the patch passed |
| +1 :green_heart: | javac | 9m 52s | | the patch passed |
| +1 :green_heart: | compile | 9m 17s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 9m 17s | | the patch passed |
| +1 :green_heart: | javac | 9m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 50s | | the patch passed |
| +1 :green_heart: | mvnsite | 8m 4s | | the patch passed |
| +1 :green_heart: | javadoc | 5m 35s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 58s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 19m 55s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 241m 32s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/23/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn in the patch passed. |
| +1 :green_heart: | unit | 1m 35s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 35s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 37s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 25s | | The patch does not
generate ASF License warnings. |
| | | 471m 2s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.nodemanager.containermanager.logaggregation.TestLogAggregationService
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4846/23/artifact/out/Dockerfile
|
| GITHUB PR |
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17604831#comment-17604831
]
ASF GitHub Bot commented on YARN-11296:
---
goiri commented on code in PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#discussion_r971047522
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -356,4 +376,31 @@ public void closeConnection() {
}
}
+ /**
+ * Extract The Create Table Sql From The Script.
+ *
+ * @param dbIdentifier database identifier, Like Mysql / SqlServer
+ * @param regex the regex
+ * @throws IOException IO exception.
+ */
+ protected void extractCreateTableSQL(String dbIdentifier, String regex)
throws IOException {
+String createTableScriptPath = "." + File.separator + "target" +
File.separator +
+"test-classes" + File.separator + dbIdentifier +
"/FederationStateStoreTables.sql";
+String createTableSQL =
+FileUtils.readFileToString(new File(createTableScriptPath),
StandardCharsets.UTF_8);
+Pattern p = Pattern.compile(regex);
+Matcher m = p.matcher(createTableSQL);
+while(m!=null && m.find()) {
Review Comment:
space after while
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -301,14 +309,17 @@ public class HSQLDBFederationStateStore extends
SQLFederationStateStore {
+ " WHERE reservationId = reservationId_IN;"
+ " SET rowCount_OUT = 2; END";
+ private List tables = new ArrayList<>();
@Override
public void init(Configuration conf) {
+
Review Comment:
Avoid
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -342,11 +353,20 @@ public void init(Configuration conf) {
conn.prepareStatement(SP_UPDATERESERVATIONHOMESUBCLUSTER).execute();
LOG.info("Database Init: Complete");
-} catch (SQLException e) {
+} catch (Exception e) {
LOG.error("ERROR: failed to inizialize HSQLDB " + e.getMessage());
Review Comment:
Use logger {}
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -301,14 +309,17 @@ public class HSQLDBFederationStateStore extends
SQLFederationStateStore {
+ " WHERE reservationId = reservationId_IN;"
+ " SET rowCount_OUT = 2; END";
+ private List tables = new ArrayList<>();
@Override
public void init(Configuration conf) {
+
try {
super.init(conf);
} catch (YarnException e1) {
LOG.error("ERROR: failed to init HSQLDB " + e1.getMessage());
}
+
Review Comment:
Avoid
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -342,11 +353,20 @@ public void init(Configuration conf) {
conn.prepareStatement(SP_UPDATERESERVATIONHOMESUBCLUSTER).execute();
LOG.info("Database Init: Complete");
-} catch (SQLException e) {
+} catch (Exception e) {
LOG.error("ERROR: failed to inizialize HSQLDB " + e.getMessage());
}
}
+ public void initConnection(Configuration conf) {
+try {
+ super.init(conf);
+ conn = super.conn;
+} catch (YarnException e1) {
+ LOG.error("ERROR: failed to init HSQLDB " + e1.getMessage());
Review Comment:
Use {}
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/store/impl/HSQLDBFederationStateStore.java:
##
@@ -356,4 +376,31 @@ public void closeConnection() {
}
}
+ /**
+ * Extract The Create Table Sql From The Script.
+ *
+ * @param dbIdentifier database identifier, Like Mysql / SqlServer
+ * @param regex the regex
+ * @throws IOException IO exception.
+ */
+ protected void extractCreateTableSQL(String dbIdentifier, String regex)
throws IOException {
+String createTableScriptPath = "." + File.separator + "target" +
File.separator +
Review Comment:
It might be cleaner to use String.format()
> Fix SQLFederationStateStore#Sql script bug
> --
>
> Key: YARN-11296
> URL: https://issues.apache.org/jira/browse/YARN-11296
> Project: Hadoop YARN
> Issue Type: Bug
> Components: federation
>Affects Versions: 3.4.0
>Reporter:
[jira] [Commented] (YARN-11293) [Federation] Router Support DelegationToken storeNewMasterKey/removeStoredMasterKey With MemoryStateStore
[
https://issues.apache.org/jira/browse/YARN-11293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17604827#comment-17604827
]
ASF GitHub Bot commented on YARN-11293:
---
goiri commented on code in PR #4852:
URL: https://github.com/apache/hadoop/pull/4852#discussion_r971043197
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterMasterKey.java:
##
@@ -0,0 +1,131 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceAudience.Public;
+import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.hadoop.yarn.util.Records;
+
+import java.nio.ByteBuffer;
+
+@Private
+@Unstable
+public abstract class RouterMasterKey {
+
+ @Private
+ @Unstable
+ public static RouterMasterKey newInstance(Integer keyId, ByteBuffer
keyBytes, Long expiryDate) {
+RouterMasterKey policy = Records.newRecord(RouterMasterKey.class);
+policy.setKeyId(keyId);
+policy.setKeyBytes(keyBytes);
+policy.setExpiryDate(expiryDate);
+return policy;
+ }
+
+ @Private
+ @Unstable
+ public static RouterMasterKey newInstance(RouterMasterKey masterKey) {
+RouterMasterKey routerMasterKey = Records.newRecord(RouterMasterKey.class);
+routerMasterKey.setKeyId(masterKey.getKeyId());
+routerMasterKey.setKeyBytes(masterKey.getKeyBytes());
+routerMasterKey.setExpiryDate(masterKey.getExpiryDate());
+return routerMasterKey;
+ }
+
+ /**
+ * Get the keyId of the MasterKey.
+ *
+ * @return MasterKeyId.
+ */
+ @Public
+ @Unstable
+ public abstract Integer getKeyId();
+
+ /**
+ * Set the keyId of the MasterKey.
+ *
+ * @param keyId MasterKeyId.
+ */
+ @Private
+ @Unstable
+ public abstract void setKeyId(Integer keyId);
+
+ /**
+ * Get the keyBytes of the DelegationKey.
+ *
+ * @return KeyBytes of the DelegationKey.
+ */
+ @Public
+ @Unstable
+ public abstract ByteBuffer getKeyBytes();
+
+ /**
+ * Set the keyBytes of the DelegationKey.
+ *
+ * @param keyBytes KeyBytes of the DelegationKey.
+ */
+ @Private
+ @Unstable
+ public abstract void setKeyBytes(ByteBuffer keyBytes);
+
+ /**
+ * Get the ExpiryDate of the DelegationKey.
+ *
+ * @return ExpiryDate of the DelegationKey.
+ */
+ @Private
+ @Unstable
+ public abstract Long getExpiryDate();
+
+ /**
+ * Set the expiryDate of the DelegationKey.
+ *
+ * @param expiryDate expiryDate of the DelegationKey.
+ */
+ @Private
+ @Unstable
+ public abstract void setExpiryDate(Long expiryDate);
+
+ @Override
+ public int hashCode() {
+return new HashCodeBuilder()
+.append(this.getExpiryDate().longValue())
+.append(this.getKeyId().intValue())
+.append(getKeyBytes().array())
+.hashCode();
+ }
+
+ @Override
+ public boolean equals(Object right) {
+if (this == right) {
+ return true;
+} else if (right == null || getClass() != right.getClass()) {
Review Comment:
No need for else when you return
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterMasterKey.java:
##
@@ -0,0 +1,131 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is
[jira] [Commented] (YARN-11290) Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
[
https://issues.apache.org/jira/browse/YARN-11290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17604824#comment-17604824
]
ASF GitHub Bot commented on YARN-11290:
---
goiri commented on code in PR #4846:
URL: https://github.com/apache/hadoop/pull/4846#discussion_r971038757
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/ZookeeperFederationStateStore.java:
##
@@ -255,24 +263,54 @@ public GetApplicationHomeSubClusterResponse
getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
-long start = clock.getTime();
-List result = new ArrayList<>();
+
+if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+}
try {
- for (String child : zkManager.getChildren(appsZNode)) {
-ApplicationId appId = ApplicationId.fromString(child);
-SubClusterId homeSubCluster = getApp(appId);
-ApplicationHomeSubCluster app =
-ApplicationHomeSubCluster.newInstance(appId, homeSubCluster);
-result.add(app);
- }
+ long start = clock.getTime();
+ SubClusterId requestSC = request.getSubClusterId();
+ List children = zkManager.getChildren(appsZNode);
+ List result =
+ children.stream().map(child -> generateAppHomeSC(child))
+
.sorted(Comparator.comparing(ApplicationHomeSubCluster::getCreateTime).reversed())
+ .filter(appHomeSC -> filterHomeSubCluster(requestSC,
appHomeSC.getHomeSubCluster()))
+ .limit(maxAppsInStateStore)
+ .collect(Collectors.toList());
+ long end = clock.getTime();
+ opDurations.addGetAppsHomeSubClusterDuration(start, end);
+ LOG.info("filterSubClusterId = {}, appCount = {}.", requestSC,
result.size());
+ return GetApplicationsHomeSubClusterResponse.newInstance(result);
} catch (Exception e) {
String errMsg = "Cannot get apps: " + e.getMessage();
FederationStateStoreUtils.logAndThrowStoreException(LOG, errMsg);
}
-long end = clock.getTime();
-opDurations.addGetAppsHomeSubClusterDuration(start, end);
-return GetApplicationsHomeSubClusterResponse.newInstance(result);
+
+throw new YarnException("Cannot get app by request");
+ }
+
+ private ApplicationHomeSubCluster generateAppHomeSC(String appId) {
+try {
+ ApplicationId applicationId = ApplicationId.fromString(appId);
+ SubClusterId homeSubCluster = getApp(applicationId);
+ ApplicationHomeSubCluster app =
+ ApplicationHomeSubCluster.newInstance(applicationId, Time.now(),
homeSubCluster);
+ return app;
+} catch (Exception ex) {
+ LOG.error("get homeSubCluster by appId = {}.", appId);
+}
+return null;
+ }
+
+ private boolean filterHomeSubCluster(SubClusterId filterSubCluster,
Review Comment:
We have this function defined three times in three places.
It should also be static.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/ZookeeperFederationStateStore.java:
##
@@ -255,24 +263,54 @@ public GetApplicationHomeSubClusterResponse
getApplicationHomeSubCluster(
@Override
public GetApplicationsHomeSubClusterResponse getApplicationsHomeSubCluster(
GetApplicationsHomeSubClusterRequest request) throws YarnException {
-long start = clock.getTime();
-List result = new ArrayList<>();
+
+if (request == null) {
+ throw new YarnException("Missing getApplicationsHomeSubCluster request");
+}
try {
- for (String child : zkManager.getChildren(appsZNode)) {
-ApplicationId appId = ApplicationId.fromString(child);
-SubClusterId homeSubCluster = getApp(appId);
-ApplicationHomeSubCluster app =
-ApplicationHomeSubCluster.newInstance(appId, homeSubCluster);
-result.add(app);
- }
+ long start = clock.getTime();
+ SubClusterId requestSC = request.getSubClusterId();
+ List children = zkManager.getChildren(appsZNode);
+ List result =
+ children.stream().map(child -> generateAppHomeSC(child))
+
.sorted(Comparator.comparing(ApplicationHomeSubCluster::getCreateTime).reversed())
Review Comment:
I don't think indentation is correct.
> Improve Query Condition of FederationStateStore#getApplicationsHomeSubCluster
> -
>
> Key: YARN-11290
> URL: https://issues.apache.org/jira/browse/YARN-11290
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: federation
>Affects Versions:
[jira] [Commented] (YARN-11226) [Federation] Add createNewReservation, submitReservation, updateReservation, deleteReservation REST APIs for Router
[
https://issues.apache.org/jira/browse/YARN-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17604034#comment-17604034
]
ASF GitHub Bot commented on YARN-11226:
---
hadoop-yetus commented on PR #4851:
URL: https://github.com/apache/hadoop/pull/4851#issuecomment-1246632962
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 8s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 10s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 32s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 59s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 31s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 23s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4851/7/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router:
The patch generated 3 new + 1 unchanged - 0 fixed = 4 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 0m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 47s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 95m 21s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4851/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4851 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4cd60f222a18 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5ba8ac8bcbedfe63d89a2a997545c4f61cd0f1b6 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Commented] (YARN-11226) [Federation] Add createNewReservation, submitReservation, updateReservation, deleteReservation REST APIs for Router
[
https://issues.apache.org/jira/browse/YARN-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17604030#comment-17604030
]
ASF GitHub Bot commented on YARN-11226:
---
hadoop-yetus commented on PR #4851:
URL: https://github.com/apache/hadoop/pull/4851#issuecomment-1246625513
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 10s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 39s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 39s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 3s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 28s | | the patch passed |
| +1 :green_heart: | compile | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 26s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4851/6/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router:
The patch generated 13 new + 1 unchanged - 0 fixed = 14 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 29s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 0m 59s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 49s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 95m 16s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4851/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4851 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9f6975967cf1 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e971bc694c5c35c97aff65521018753fbbb274ba |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17604010#comment-17604010
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1246562711
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 36s | | trunk passed |
| +1 :green_heart: | compile | 26m 4s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 23m 3s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 43s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 5s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 15s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 47s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 21m 27s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 54s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 3s | | the patch passed |
| +1 :green_heart: | compile | 25m 26s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 25m 26s | | the patch passed |
| +1 :green_heart: | compile | 23m 17s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 23m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 6s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/13/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 0 unchanged - 1 fixed = 1 total (was 1) |
| +1 :green_heart: | mvnsite | 7m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 51s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 20s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 35s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 21m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 33s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 239m 0s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 3m 40s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 29s | | The patch does not
generate ASF License warnings. |
| | | 511m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux c098b10d1935 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk /
[jira] [Commented] (YARN-11296) Fix SQLFederationStateStore#Sql script bug
[
https://issues.apache.org/jira/browse/YARN-11296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17604009#comment-17604009
]
ASF GitHub Bot commented on YARN-11296:
---
hadoop-yetus commented on PR #4858:
URL: https://github.com/apache/hadoop/pull/4858#issuecomment-1246560423
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 13s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 40s | | trunk passed |
| +1 :green_heart: | compile | 26m 0s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 23m 26s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 16s | | trunk passed |
| +1 :green_heart: | mvnsite | 7m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 0s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 15s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 44s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 21m 10s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 37s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 38s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 6m 7s | | the patch passed |
| +1 :green_heart: | compile | 25m 50s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 25m 50s | | the patch passed |
| +1 :green_heart: | compile | 23m 16s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 23m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 12s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/12/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 1 new + 0 unchanged - 1 fixed = 1 total (was 1) |
| +1 :green_heart: | mvnsite | 7m 29s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 53s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 4m 24s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 0m 34s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 20m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 30s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 238m 17s | | hadoop-yarn in the patch
passed. |
| +1 :green_heart: | unit | 3m 34s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 1m 26s | | The patch does not
generate ASF License warnings. |
| | | 511m 33s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4858/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4858 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux bac311eb18c0 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk /
[jira] [Commented] (YARN-11226) [Federation] Add createNewReservation, submitReservation, updateReservation, deleteReservation REST APIs for Router
[
https://issues.apache.org/jira/browse/YARN-11226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603985#comment-17603985
]
ASF GitHub Bot commented on YARN-11226:
---
hadoop-yetus commented on PR #4851:
URL: https://github.com/apache/hadoop/pull/4851#issuecomment-1246507991
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 13s | | trunk passed |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 47s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 43s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 25s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 48s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 20s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4851/5/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router:
The patch generated 11 new + 1 unchanged - 0 fixed = 12 total (was 1) |
| +1 :green_heart: | mvnsite | 0m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 28s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 26s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 0m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 47s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 95m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4851/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4851 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3c58c6a3d04f 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b121124d6c668fa1cb26affbac36a6319228b67f |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Commented] (YARN-11304) Upgrade nodemanager's exception log level
[
https://issues.apache.org/jira/browse/YARN-11304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603973#comment-17603973
]
ASF GitHub Bot commented on YARN-11304:
---
hadoop-yetus commented on PR #4887:
URL: https://github.com/apache/hadoop/pull/4887#issuecomment-1246453874
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 19s | | trunk passed |
| +1 :green_heart: | compile | 1m 47s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 0s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 20s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 1m 26s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 1m 21s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 29s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 47s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 3s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 24m 4s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 124m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4887/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4887 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 77124e5bf287 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 87b74f49c3ff77d4143949e32aea7629ba4d27fb |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.16+8-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_342-8u342-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4887/1/testReport/ |
| Max. process+thread count | 560 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
|
| Console output |
[jira] [Updated] (YARN-11304) Upgrade nodemanager's exception log level
[ https://issues.apache.org/jira/browse/YARN-11304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liu bin updated YARN-11304: --- External issue URL: https://github.com/apache/hadoop/pull/4887 > Upgrade nodemanager's exception log level > - > > Key: YARN-11304 > URL: https://issues.apache.org/jira/browse/YARN-11304 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: liu bin >Priority: Trivial > Labels: pull-request-available > > Upgrade nodemanager's localizer failed log level from info to warn. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11304) Upgrade nodemanager's exception log level
[ https://issues.apache.org/jira/browse/YARN-11304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11304: -- Labels: pull-request-available (was: ) > Upgrade nodemanager's exception log level > - > > Key: YARN-11304 > URL: https://issues.apache.org/jira/browse/YARN-11304 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: liu bin >Priority: Trivial > Labels: pull-request-available > > Upgrade nodemanager's localizer failed log level from info to warn. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11304) Upgrade nodemanager's exception log level
[ https://issues.apache.org/jira/browse/YARN-11304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17603910#comment-17603910 ] ASF GitHub Bot commented on YARN-11304: --- liubin101 opened a new pull request, #4887: URL: https://github.com/apache/hadoop/pull/4887 ### Description of PR Upgrade nodemanager's localizer failed log level from info to warn. > Upgrade nodemanager's exception log level > - > > Key: YARN-11304 > URL: https://issues.apache.org/jira/browse/YARN-11304 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: liu bin >Priority: Trivial > > Upgrade nodemanager's localizer failed log level from info to warn. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11304) Upgrade nodemanager's exception log level
liu bin created YARN-11304: -- Summary: Upgrade nodemanager's exception log level Key: YARN-11304 URL: https://issues.apache.org/jira/browse/YARN-11304 Project: Hadoop YARN Issue Type: Improvement Reporter: liu bin Upgrade nodemanager's localizer failed log level from info to warn. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
