[jira] [Commented] (YARN-9425) Make initialDelay configurable for FederationStateStoreService#scheduledExecutorService
[
https://issues.apache.org/jira/browse/YARN-9425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578272#comment-17578272
]
ASF GitHub Bot commented on YARN-9425:
--
hadoop-yetus commented on PR #4731:
URL: https://github.com/apache/hadoop/pull/4731#issuecomment-1211578864
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 47s | | trunk passed |
| +1 :green_heart: | compile | 9m 57s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 8m 56s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 33s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 19s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 31s | | the patch passed |
| +1 :green_heart: | compile | 9m 8s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 9m 8s | | the patch passed |
| +1 :green_heart: | compile | 8m 47s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 8m 47s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 5s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 55s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 3m 51s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 43s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 32s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 99m 1s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 1m 25s | | The patch does not
generate ASF License warnings. |
| | | 283m 53s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4731/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4731 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux a7d61b49b5f6 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / affc62b151b64ea8d3a90ec421dfcfb3884ede06 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Updated] (YARN-11223) [Federation] Add getAppPriority, updateApplicationPriority REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fanshilun updated YARN-11223: - Affects Version/s: 3.4.0 3.3.4 > [Federation] Add getAppPriority, updateApplicationPriority REST APIs for > Router > --- > > Key: YARN-11223 > URL: https://issues.apache.org/jira/browse/YARN-11223 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0, 3.3.4 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11223) [Federation] Add getAppPriority, updateApplicationPriority REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11223: -- Labels: pull-request-available (was: ) > [Federation] Add getAppPriority, updateApplicationPriority REST APIs for > Router > --- > > Key: YARN-11223 > URL: https://issues.apache.org/jira/browse/YARN-11223 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11223) [Federation] Add getAppPriority, updateApplicationPriority REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] fanshilun updated YARN-11223: - Fix Version/s: (was: 3.4.0) (was: 3.3.4) > [Federation] Add getAppPriority, updateApplicationPriority REST APIs for > Router > --- > > Key: YARN-11223 > URL: https://issues.apache.org/jira/browse/YARN-11223 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11223) [Federation] Add getAppPriority, updateApplicationPriority REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578269#comment-17578269 ] ASF GitHub Bot commented on YARN-11223: --- slfan1989 opened a new pull request, #4733: URL: https://github.com/apache/hadoop/pull/4733 JIRA: YARN-11223. [Federation] Add getAppPriority, updateApplicationPriority REST APIs for Router. > [Federation] Add getAppPriority, updateApplicationPriority REST APIs for > Router > --- > > Key: YARN-11223 > URL: https://issues.apache.org/jira/browse/YARN-11223 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Fix For: 3.4.0, 3.3.4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8503) Add unit test for FINISHED_CONTAINERS_PULLED_BY_AM event on DECOMMISSIONING
[ https://issues.apache.org/jira/browse/YARN-8503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578224#comment-17578224 ] fanshilun commented on YARN-8503: - I will continue to follow up on this pr. > Add unit test for FINISHED_CONTAINERS_PULLED_BY_AM event on DECOMMISSIONING > --- > > Key: YARN-8503 > URL: https://issues.apache.org/jira/browse/YARN-8503 > Project: Hadoop YARN > Issue Type: Test > Components: test >Affects Versions: 2.7.2 >Reporter: Amiya Chakraborty >Assignee: Amiya Chakraborty >Priority: Major > Labels: patch-available, yarn > Attachments: YARN-8503.001.patch, YARN-8503.001.patch > > > Currently, there is no unit test for testing the functionality - > FINISHED_CONTAINERS_PULLED_BY_AM event while Decommissioning of node. This > patch provides the same to check the AM has pulled the containers from the > RM; then the RM will inform the NM about it and the NM can remove the > completed container from its list during DECOMMISSIONING. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8482) [Router] Add cache service for fast answers to getApps
[ https://issues.apache.org/jira/browse/YARN-8482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578223#comment-17578223 ] fanshilun commented on YARN-8482: - I will continue to follow up on this pr. > [Router] Add cache service for fast answers to getApps > -- > > Key: YARN-8482 > URL: https://issues.apache.org/jira/browse/YARN-8482 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Young Chen >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11250) Capture the Performance Metrics of ZookeeperFederationStateStore
fanshilun created YARN-11250: Summary: Capture the Performance Metrics of ZookeeperFederationStateStore Key: YARN-11250 URL: https://issues.apache.org/jira/browse/YARN-11250 Project: Hadoop YARN Issue Type: Improvement Components: federation Affects Versions: 3.4.0, 3.3.4 Reporter: fanshilun Assignee: fanshilun -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7898) [FederationStateStore] Create a proxy chain for FederationStateStore API in the Router
[ https://issues.apache.org/jira/browse/YARN-7898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578202#comment-17578202 ] fanshilun commented on YARN-7898: - I will continue to follow up on this pr. > [FederationStateStore] Create a proxy chain for FederationStateStore API in > the Router > -- > > Key: YARN-7898 > URL: https://issues.apache.org/jira/browse/YARN-7898 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Minni Mittal >Priority: Major > Labels: pull-request-available > Attachments: StateStoreProxy StressTest.jpg, > YARN-7898-YARN-7402.proto.patch, YARN-7898-YARN-7402.v1.patch, > YARN-7898-YARN-7402.v2.patch, YARN-7898-YARN-7402.v3.patch, > YARN-7898-YARN-7402.v4.patch, YARN-7898-YARN-7402.v5.patch, > YARN-7898-YARN-7402.v6.patch, YARN-7898-YARN-7402.v7.patch, > YARN-7898-YARN-7402.v8.patch, YARN-7898-YARN-7402.v9.patch, YARN-7898.v7.patch > > Time Spent: 20m > Remaining Estimate: 0h > > As detailed in the proposal in the umbrella JIRA, we are introducing a new > component that routes client request to appropriate FederationStateStore. > This JIRA tracks the creation of a proxy for FederationStateStore in the > Router. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11227) [Federation] Add getAppTimeout, getAppTimeouts, updateApplicationTimeout REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578201#comment-17578201 ] ASF GitHub Bot commented on YARN-11227: --- slfan1989 commented on PR #4715: URL: https://github.com/apache/hadoop/pull/4715#issuecomment-1211465939 @goiri Thank you very much for your help reviewing the code！ > [Federation] Add getAppTimeout, getAppTimeouts, updateApplicationTimeout REST > APIs for Router > - > > Key: YARN-11227 > URL: https://issues.apache.org/jira/browse/YARN-11227 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0, 3.3.4 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9425) Make initialDelay configurable for FederationStateStoreService#scheduledExecutorService
[ https://issues.apache.org/jira/browse/YARN-9425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578196#comment-17578196 ] ASF GitHub Bot commented on YARN-9425: -- ashutoshcipher opened a new pull request, #4731: URL: https://github.com/apache/hadoop/pull/4731 ### Description of PR Make initialDelay configurable for FederationStateStoreService#scheduledExecutorService JIRA - YARN-9425 ### For code changes: - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Make initialDelay configurable for > FederationStateStoreService#scheduledExecutorService > --- > > Key: YARN-9425 > URL: https://issues.apache.org/jira/browse/YARN-9425 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Affects Versions: 3.1.0 >Reporter: Shen Yinjie >Assignee: groot >Priority: Major > Attachments: YARN-9425_1.patch, YARN-9425_2.patch, YARN-9425_3.patch, > YARN-9425_4.patch > > > When enable YARN federation, subclusters info in Router Web UI cannot be > loaded immediately, and client cannot find any active subclusters after 5mins > by default ,which is configured by > "yarn.federation.state-store.heartbeat-interval-secs". > IMA,we should seperate 'initialDely' and 'delay' for > FederationStateStoreService#scheduledExecutorService. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9425) Make initialDelay configurable for FederationStateStoreService#scheduledExecutorService
[ https://issues.apache.org/jira/browse/YARN-9425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-9425: - Labels: pull-request-available (was: ) > Make initialDelay configurable for > FederationStateStoreService#scheduledExecutorService > --- > > Key: YARN-9425 > URL: https://issues.apache.org/jira/browse/YARN-9425 > Project: Hadoop YARN > Issue Type: Bug > Components: federation >Affects Versions: 3.1.0 >Reporter: Shen Yinjie >Assignee: groot >Priority: Major > Labels: pull-request-available > Attachments: YARN-9425_1.patch, YARN-9425_2.patch, YARN-9425_3.patch, > YARN-9425_4.patch > > > When enable YARN federation, subclusters info in Router Web UI cannot be > loaded immediately, and client cannot find any active subclusters after 5mins > by default ,which is configured by > "yarn.federation.state-store.heartbeat-interval-secs". > IMA,we should seperate 'initialDely' and 'delay' for > FederationStateStoreService#scheduledExecutorService. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-11227) [Federation] Add getAppTimeout, getAppTimeouts, updateApplicationTimeout REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved YARN-11227. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > [Federation] Add getAppTimeout, getAppTimeouts, updateApplicationTimeout REST > APIs for Router > - > > Key: YARN-11227 > URL: https://issues.apache.org/jira/browse/YARN-11227 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0, 3.3.4 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11227) [Federation] Add getAppTimeout, getAppTimeouts, updateApplicationTimeout REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578164#comment-17578164 ] ASF GitHub Bot commented on YARN-11227: --- goiri merged PR #4715: URL: https://github.com/apache/hadoop/pull/4715 > [Federation] Add getAppTimeout, getAppTimeouts, updateApplicationTimeout REST > APIs for Router > - > > Key: YARN-11227 > URL: https://issues.apache.org/jira/browse/YARN-11227 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0, 3.3.4 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[
https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578113#comment-17578113
]
ASF GitHub Bot commented on YARN-11236:
---
hadoop-yetus commented on PR #4711:
URL: https://github.com/apache/hadoop/pull/4711#issuecomment-1211131931
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 17m 19s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 21s | | trunk passed |
| +1 :green_heart: | compile | 4m 4s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 30s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 31s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 17s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 8s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 56s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 54s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 41s | | the patch passed |
| +1 :green_heart: | compile | 3m 48s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 3m 48s | | the patch passed |
| -1 :x: | javac | 3m 48s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4711/7/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 3 new +
446 unchanged - 0 fixed = 449 total (was 446) |
| +1 :green_heart: | compile | 3m 16s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 3m 16s | | the patch passed |
| -1 :x: | javac | 3m 16s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4711/7/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 3 new
+ 370 unchanged - 0 fixed = 373 total (was 370) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 11s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new +
11 unchanged - 1 fixed = 11 total (was 12) |
| +1 :green_heart: | mvnsite | 1m 52s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 29s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 39s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 2s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart:
[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[
https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578112#comment-17578112
]
ASF GitHub Bot commented on YARN-11236:
---
hadoop-yetus commented on PR #4711:
URL: https://github.com/apache/hadoop/pull/4711#issuecomment-1211127930
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 2s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 19s | | trunk passed |
| +1 :green_heart: | compile | 4m 17s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 25s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 22s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 37s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 57s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 37s | | the patch passed |
| +1 :green_heart: | compile | 3m 59s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 3m 59s | | the patch passed |
| -1 :x: | javac | 3m 59s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4711/6/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 3 new +
444 unchanged - 0 fixed = 447 total (was 444) |
| +1 :green_heart: | compile | 3m 18s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 3m 18s | | the patch passed |
| -1 :x: | javac | 3m 18s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4711/6/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 3 new
+ 368 unchanged - 0 fixed = 371 total (was 368) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 8s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new +
11 unchanged - 1 fixed = 11 total (was 12) |
| +1 :green_heart: | mvnsite | 1m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 5s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart:
[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[
https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578108#comment-17578108
]
ASF GitHub Bot commented on YARN-11236:
---
hadoop-yetus commented on PR #4711:
URL: https://github.com/apache/hadoop/pull/4711#issuecomment-1211103628
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 11s | | trunk passed |
| +1 :green_heart: | compile | 4m 26s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 44s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 51s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 16s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 32s | | the patch passed |
| +1 :green_heart: | compile | 3m 55s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 3m 55s | | the patch passed |
| -1 :x: | javac | 3m 55s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4711/5/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 3 new +
446 unchanged - 0 fixed = 449 total (was 446) |
| +1 :green_heart: | compile | 3m 15s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 3m 15s | | the patch passed |
| -1 :x: | javac | 3m 15s |
[/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4711/5/artifact/out/results-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
|
hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 3 new
+ 370 unchanged - 0 fixed = 373 total (was 370) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 4s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new +
11 unchanged - 1 fixed = 11 total (was 12) |
| +1 :green_heart: | mvnsite | 1m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 32s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 25s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 52s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart:
[jira] [Commented] (YARN-10885) Make FederationStateStoreFacade#getApplicationHomeSubCluster use JCache
[
https://issues.apache.org/jira/browse/YARN-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578078#comment-17578078
]
ASF GitHub Bot commented on YARN-10885:
---
hadoop-yetus commented on PR #4701:
URL: https://github.com/apache/hadoop/pull/4701#issuecomment-1211012699
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 29s | | trunk passed |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 52s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 40s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 35s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 21s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 0 new + 10 unchanged - 1 fixed = 10 total (was 11) |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 45s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 107m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4701/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4701 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 1d075bf83006 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7a9c9a5c322f29af22c0153f25aa241e32f7cb48 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4701/10/testReport/ |
| Max. process+thread count | 609 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common |
| Console output |
[jira] [Commented] (YARN-6539) Create SecureLogin inside Router
[ https://issues.apache.org/jira/browse/YARN-6539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578045#comment-17578045 ] ASF GitHub Bot commented on YARN-6539: -- slfan1989 commented on PR #4712: URL: https://github.com/apache/hadoop/pull/4712#issuecomment-1210888996 @goiri Please help to review the code again, Thank you very much! I want to follow up on [YARN-11158](https://issues.apache.org/jira/browse/YARN-11158), need this pr. > Create SecureLogin inside Router > > > Key: YARN-6539 > URL: https://issues.apache.org/jira/browse/YARN-6539 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Giovanni Matteo Fumarola >Assignee: Xie YiFan >Priority: Minor > Labels: pull-request-available > Attachments: YARN-6359_1.patch, YARN-6359_2.patch, > YARN-6539-branch-3.1.0.004.patch, YARN-6539-branch-3.1.0.005.patch, > YARN-6539.006.patch, YARN-6539.007.patch, YARN-6539.008.patch, > YARN-6539_3.patch, YARN-6539_4.patch > > Time Spent: 5.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10885) Make FederationStateStoreFacade#getApplicationHomeSubCluster use JCache
[ https://issues.apache.org/jira/browse/YARN-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578044#comment-17578044 ] ASF GitHub Bot commented on YARN-10885: --- slfan1989 commented on PR #4701: URL: https://github.com/apache/hadoop/pull/4701#issuecomment-1210887671 @goiri Please help to review the code again, thank you very much! > Make FederationStateStoreFacade#getApplicationHomeSubCluster use JCache > --- > > Key: YARN-10885 > URL: https://issues.apache.org/jira/browse/YARN-10885 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: chaosju >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > Yarn Client getApplicationReport function may produce lots of zookeeper ops, > Its import to use the JCache that cache the mapping of application and > subcluster id. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11247) Remove unused classes introduced by YARN-9615
[ https://issues.apache.org/jira/browse/YARN-11247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578043#comment-17578043 ] ASF GitHub Bot commented on YARN-11247: --- slfan1989 commented on PR #4720: URL: https://github.com/apache/hadoop/pull/4720#issuecomment-1210885913 @ayushtkn Can you help review this pr? Thank you very much! > Remove unused classes introduced by YARN-9615 > - > > Key: YARN-11247 > URL: https://issues.apache.org/jira/browse/YARN-11247 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Minor > Labels: pull-request-available > Attachments: DisableEventTypeMetrics-Not used.png > > > YARN-9615 adds Metric to RM's dispatcher, but the patch introduces a class > without any usage > org.apache.hadoop.yarn.metrics#DisableEventTypeMetrics > 1. Without any code references > 2. Without any test code references > 3. Delete this class, the local can be compiled successfully > I think this class can be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[ https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578032#comment-17578032 ] ASF GitHub Bot commented on YARN-11236: --- slfan1989 commented on PR #4711: URL: https://github.com/apache/hadoop/pull/4711#issuecomment-1210819769 @goiri Please help to review the code again, Thank you very much! > Implement FederationReservationHomeSubClusterStore With MemoryStore > --- > > Key: YARN-11236 > URL: https://issues.apache.org/jira/browse/YARN-11236 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[
https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578030#comment-17578030
]
ASF GitHub Bot commented on YARN-11236:
---
slfan1989 commented on code in PR #4711:
URL: https://github.com/apache/hadoop/pull/4711#discussion_r942573705
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/policies/router/BaseRouterPoliciesTest.java:
##
@@ -126,4 +136,62 @@ public void testNullReservationContext() throws Exception {
() -> policy.getReservationHomeSubcluster(null));
}
+ @Test
+ public void testUnknownReservation() throws Exception {
+
+long now = Time.now();
+ReservationSubmissionRequest resReq = getReservationSubmissionRequest();
+ReservationId reservationId = ReservationId.newInstance(now, 1);
+when(resReq.getQueue()).thenReturn("queue1");
+when(resReq.getReservationId()).thenReturn(reservationId);
+
+// route an application that uses this app
+ApplicationSubmissionContext applicationSubmissionContext =
+ApplicationSubmissionContext.newInstance(
+ApplicationId.newInstance(now, 1), "app1", "queue1",
Priority.newInstance(1),
+null, false, false, 1, null, null, false);
+
+applicationSubmissionContext.setReservationID(resReq.getReservationId());
+FederationRouterPolicy policy = (FederationRouterPolicy) getPolicy();
+
+LambdaTestUtils.intercept(YarnException.class,
+"Reservation " + reservationId + " does not exist",
+() -> policy.getHomeSubcluster(applicationSubmissionContext, new
ArrayList<>()));
+ }
+
+ @Test
+ public void testFollowReservation() throws YarnException {
+
+long now = Time.now();
+ReservationSubmissionRequest resReq = getReservationSubmissionRequest();
+when(resReq.getQueue()).thenReturn("queue1");
+when(resReq.getReservationId())
Review Comment:
I will fix it.
> Implement FederationReservationHomeSubClusterStore With MemoryStore
> ---
>
> Key: YARN-11236
> URL: https://issues.apache.org/jira/browse/YARN-11236
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[
https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578028#comment-17578028
]
ASF GitHub Bot commented on YARN-11236:
---
slfan1989 commented on code in PR #4711:
URL: https://github.com/apache/hadoop/pull/4711#discussion_r942573421
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/impl/pb/ReservationHomeSubClusterPBImpl.java:
##
@@ -0,0 +1,167 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.federation.store.records.impl.pb;
+
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.hadoop.yarn.api.records.ReservationId;
+import org.apache.hadoop.yarn.api.records.impl.pb.ReservationIdPBImpl;
+import
org.apache.hadoop.yarn.federation.proto.YarnServerFederationProtos.ReservationHomeSubClusterProto;
+import
org.apache.hadoop.yarn.federation.proto.YarnServerFederationProtos.ReservationHomeSubClusterProtoOrBuilder;
+import
org.apache.hadoop.yarn.federation.proto.YarnServerFederationProtos.SubClusterIdProto;
+import org.apache.hadoop.yarn.proto.YarnProtos.ReservationIdProto;
+import
org.apache.hadoop.yarn.server.federation.store.records.ReservationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+
+import org.apache.hadoop.thirdparty.protobuf.TextFormat;
+
+/**
+ * Protocol buffer based implementation of {@link ReservationHomeSubCluster}.
+ */
+@Private
+@Unstable
+public class ReservationHomeSubClusterPBImpl extends ReservationHomeSubCluster
{
+
+ private ReservationHomeSubClusterProto proto =
+ ReservationHomeSubClusterProto.getDefaultInstance();
+ private ReservationHomeSubClusterProto.Builder builder = null;
+ private boolean viaProto = false;
+
+ private ReservationId reservationId = null;
+ private SubClusterId homeSubCluster = null;
+
+ public ReservationHomeSubClusterPBImpl() {
+builder = ReservationHomeSubClusterProto.newBuilder();
+ }
+
+ public ReservationHomeSubClusterPBImpl(ReservationHomeSubClusterProto proto)
{
+this.proto = proto;
+viaProto = true;
+ }
+
+ public ReservationHomeSubClusterProto getProto() {
+mergeLocalToProto();
+proto = viaProto ? proto : builder.build();
+viaProto = true;
+return proto;
+ }
+
+ private void mergeLocalToProto() {
+if (viaProto) {
+ maybeInitBuilder();
+}
+mergeLocalToBuilder();
+proto = builder.build();
+viaProto = true;
+ }
+
+ private void maybeInitBuilder() {
+if (viaProto || builder == null) {
+ builder = ReservationHomeSubClusterProto.newBuilder(proto);
+}
+viaProto = false;
+ }
+
+ private void mergeLocalToBuilder() {
+if (this.reservationId != null) {
+ builder.setReservationId(convertToProtoFormat(this.reservationId));
+}
+if (this.homeSubCluster != null) {
+ builder.setHomeSubCluster(convertToProtoFormat(this.homeSubCluster));
+}
+ }
+
+ @Override
+ public int hashCode() {
+return getProto().hashCode();
+ }
+
+ @Override
+ public boolean equals(Object other) {
+if (other == null) {
+ return false;
+}
+if (other.getClass().isAssignableFrom(this.getClass())) {
+ return this.getProto().equals(this.getClass().cast(other).getProto());
+}
+return false;
+ }
+
+ @Override
+ public String toString() {
+return TextFormat.shortDebugString(getProto());
+ }
+
+ @Override
+ public ReservationId getReservationId() {
+ReservationHomeSubClusterProtoOrBuilder p = viaProto ? proto : builder;
+if (!p.hasReservationId()) {
+ return null;
+}
+this.reservationId = convertFromProtoFormat(p.getReservationId());
+return reservationId;
+ }
+
+ @Override
+ public void setReservationId(ReservationId reservationId) {
+maybeInitBuilder();
+if (reservationId == null) {
Review Comment:
Thanks for your suggestion, I will modify the code.
##
[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[
https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578019#comment-17578019
]
ASF GitHub Bot commented on YARN-11236:
---
slfan1989 commented on code in PR #4711:
URL: https://github.com/apache/hadoop/pull/4711#discussion_r942540670
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##
@@ -312,4 +324,45 @@ public Version loadVersion() {
return null;
}
+ @Override
+ public AddReservationHomeSubClusterResponse addReservationHomeSubCluster(
+ AddReservationHomeSubClusterRequest request) throws YarnException {
+FederationReservationHomeSubClusterStoreInputValidator.validate(request);
+ReservationId reservationId =
+request.getReservationHomeSubCluster().getReservationId();
+if (!reservations.containsKey(reservationId)) {
+ reservations.put(reservationId,
+ request.getReservationHomeSubCluster().getHomeSubCluster());
+}
+return
AddReservationHomeSubClusterResponse.newInstance(reservations.get(reservationId));
+ }
+
+ @Override
+ public GetReservationHomeSubClusterResponse getReservationHomeSubCluster(
+ GetReservationHomeSubClusterRequest request) throws YarnException {
+FederationReservationHomeSubClusterStoreInputValidator.validate(request);
+ReservationId reservationId = request.getReservationId();
+if (!reservations.containsKey(reservationId)) {
+ throw new YarnException("Reservation " + reservationId + " does not
exist");
+}
+SubClusterId subClusterId = reservations.get(reservationId);
+return GetReservationHomeSubClusterResponse.newInstance(
+ReservationHomeSubCluster.newInstance(reservationId, subClusterId));
+ }
+
+ @Override
+ public GetReservationsHomeSubClusterResponse getReservationsHomeSubCluster(
+ GetReservationsHomeSubClusterRequest request) throws YarnException {
+List result = new ArrayList<>();
+
+for (Entry entry : reservations.entrySet()) {
+ ReservationId key = entry.getKey();
Review Comment:
I will fix it.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/ReservationHomeSubCluster.java:
##
@@ -0,0 +1,130 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.commons.lang3.builder.EqualsBuilder;
+import org.apache.commons.lang3.builder.HashCodeBuilder;
+import org.apache.hadoop.classification.InterfaceAudience.Private;
+import org.apache.hadoop.classification.InterfaceAudience.Public;
+import org.apache.hadoop.classification.InterfaceStability.Unstable;
+import org.apache.hadoop.yarn.api.records.ReservationId;
+import org.apache.hadoop.yarn.util.Records;
+
+/**
+ *
+ * ReservationHomeSubCluster is a report of the runtime information of the
+ * reservation that is running in the federated cluster.
+ *
+ *
+ * It includes information such as:
+ *
+ * {@link ReservationId}
+ * {@link SubClusterId}
+ *
+ *
+ */
+@Private
+@Unstable
+public abstract class ReservationHomeSubCluster {
+
+ @Private
+ @Unstable
+ public static ReservationHomeSubCluster newInstance(ReservationId appId,
+ SubClusterId homeSubCluster) {
+ReservationHomeSubCluster appMapping =
Review Comment:
I will fix it.
> Implement FederationReservationHomeSubClusterStore With MemoryStore
> ---
>
> Key: YARN-11236
> URL: https://issues.apache.org/jira/browse/YARN-11236
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[
https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578016#comment-17578016
]
ASF GitHub Bot commented on YARN-11236:
---
slfan1989 commented on code in PR #4711:
URL: https://github.com/apache/hadoop/pull/4711#discussion_r942539432
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##
@@ -312,4 +324,45 @@ public Version loadVersion() {
return null;
}
+ @Override
+ public AddReservationHomeSubClusterResponse addReservationHomeSubCluster(
+ AddReservationHomeSubClusterRequest request) throws YarnException {
+FederationReservationHomeSubClusterStoreInputValidator.validate(request);
+ReservationId reservationId =
+request.getReservationHomeSubCluster().getReservationId();
+if (!reservations.containsKey(reservationId)) {
+ reservations.put(reservationId,
+ request.getReservationHomeSubCluster().getHomeSubCluster());
Review Comment:
I will fix it.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##
@@ -312,4 +324,45 @@ public Version loadVersion() {
return null;
}
+ @Override
+ public AddReservationHomeSubClusterResponse addReservationHomeSubCluster(
+ AddReservationHomeSubClusterRequest request) throws YarnException {
+FederationReservationHomeSubClusterStoreInputValidator.validate(request);
+ReservationId reservationId =
+request.getReservationHomeSubCluster().getReservationId();
+if (!reservations.containsKey(reservationId)) {
+ reservations.put(reservationId,
+ request.getReservationHomeSubCluster().getHomeSubCluster());
+}
+return
AddReservationHomeSubClusterResponse.newInstance(reservations.get(reservationId));
+ }
+
+ @Override
+ public GetReservationHomeSubClusterResponse getReservationHomeSubCluster(
+ GetReservationHomeSubClusterRequest request) throws YarnException {
+FederationReservationHomeSubClusterStoreInputValidator.validate(request);
+ReservationId reservationId = request.getReservationId();
+if (!reservations.containsKey(reservationId)) {
+ throw new YarnException("Reservation " + reservationId + " does not
exist");
+}
+SubClusterId subClusterId = reservations.get(reservationId);
+return GetReservationHomeSubClusterResponse.newInstance(
+ReservationHomeSubCluster.newInstance(reservationId, subClusterId));
Review Comment:
I will fix it.
> Implement FederationReservationHomeSubClusterStore With MemoryStore
> ---
>
> Key: YARN-11236
> URL: https://issues.apache.org/jira/browse/YARN-11236
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11236) Implement FederationReservationHomeSubClusterStore With MemoryStore
[
https://issues.apache.org/jira/browse/YARN-11236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578017#comment-17578017
]
ASF GitHub Bot commented on YARN-11236:
---
slfan1989 commented on code in PR #4711:
URL: https://github.com/apache/hadoop/pull/4711#discussion_r942539715
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##
@@ -312,4 +324,45 @@ public Version loadVersion() {
return null;
}
+ @Override
+ public AddReservationHomeSubClusterResponse addReservationHomeSubCluster(
+ AddReservationHomeSubClusterRequest request) throws YarnException {
+FederationReservationHomeSubClusterStoreInputValidator.validate(request);
+ReservationId reservationId =
+request.getReservationHomeSubCluster().getReservationId();
+if (!reservations.containsKey(reservationId)) {
+ reservations.put(reservationId,
+ request.getReservationHomeSubCluster().getHomeSubCluster());
+}
+return
AddReservationHomeSubClusterResponse.newInstance(reservations.get(reservationId));
+ }
+
+ @Override
+ public GetReservationHomeSubClusterResponse getReservationHomeSubCluster(
+ GetReservationHomeSubClusterRequest request) throws YarnException {
+FederationReservationHomeSubClusterStoreInputValidator.validate(request);
+ReservationId reservationId = request.getReservationId();
+if (!reservations.containsKey(reservationId)) {
+ throw new YarnException("Reservation " + reservationId + " does not
exist");
+}
+SubClusterId subClusterId = reservations.get(reservationId);
+return GetReservationHomeSubClusterResponse.newInstance(
+ReservationHomeSubCluster.newInstance(reservationId, subClusterId));
+ }
+
+ @Override
+ public GetReservationsHomeSubClusterResponse getReservationsHomeSubCluster(
+ GetReservationsHomeSubClusterRequest request) throws YarnException {
+List result = new ArrayList<>();
+
+for (Entry entry : reservations.entrySet()) {
+ ReservationId key = entry.getKey();
+ SubClusterId value = entry.getValue();
+ ReservationHomeSubCluster homeSubCluster =
Review Comment:
I will fix it.
> Implement FederationReservationHomeSubClusterStore With MemoryStore
> ---
>
> Key: YARN-11236
> URL: https://issues.apache.org/jira/browse/YARN-11236
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: federation
>Affects Versions: 3.4.0
>Reporter: fanshilun
>Assignee: fanshilun
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11227) [Federation] Add getAppTimeout, getAppTimeouts, updateApplicationTimeout REST APIs for Router
[ https://issues.apache.org/jira/browse/YARN-11227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578003#comment-17578003 ] ASF GitHub Bot commented on YARN-11227: --- slfan1989 commented on PR #4715: URL: https://github.com/apache/hadoop/pull/4715#issuecomment-1210745240 @goiri Thank you very much for helping to review the code, can you help merge this pr into the trunk branch? I will follow up with YARN-11223. > [Federation] Add getAppTimeout, getAppTimeouts, updateApplicationTimeout REST > APIs for Router > - > > Key: YARN-11227 > URL: https://issues.apache.org/jira/browse/YARN-11227 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0, 3.3.4 >Reporter: fanshilun >Assignee: fanshilun >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577886#comment-17577886
]
ASF GitHub Bot commented on YARN-11191:
---
hadoop-yetus commented on PR #4726:
URL: https://github.com/apache/hadoop/pull/4726#issuecomment-1210402542
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 33s | | trunk passed |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 1m 8s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 13s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 55s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 56s | | the patch passed |
| +1 :green_heart: | compile | 1m 6s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 1m 6s | | the patch passed |
| +1 :green_heart: | compile | 0m 55s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 55s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 46s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4726/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 2 new + 95 unchanged - 0 fixed = 97 total (was 95) |
| +1 :green_heart: | mvnsite | 1m 0s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 11s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 45s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 102m 29s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 211m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4726/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4726 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f7c0894bff81 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 65650a20ef43e8eb3311c63e3a6b0585e9e2d95a |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Updated] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ben yang updated YARN-11191:
Attachment: Lock holding status.png
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, Lock holding status.png, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ben yang updated YARN-11191:
Attachment: (was: 未命名.png)
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ben yang updated YARN-11191:
Attachment: 未命名.png
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577834#comment-17577834
]
ASF GitHub Bot commented on YARN-11191:
---
yb12138 commented on PR #4726:
URL: https://github.com/apache/hadoop/pull/4726#issuecomment-1210271008
@luoyuan3471
you can see this image.
this problem will occupy when refresh thread is calling
PreemptionManager.refreshQueue and schedule thread is calling
AbstractCSQueue.getTotalKillableResource.At this time, refresh thread will
require csqueue.readLock，but csqueue.readLock will blocked by schedule thread
and "other thread"( https://bugs.openjdk.org/browse/JDK-6893626 ).And schedule
thread will require PremmptionManager.readLock,but this readLock will blocked
by refresh thread held writeLock. so i use tryLock to make refresh thread get
csqueue.readLock. Wait for the refresh thread complete
PreemptionManager.refreshQueue,the schedule thread will get
premmptionManager.readLock, then can allocate new container.
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577807#comment-17577807
]
ASF GitHub Bot commented on YARN-11191:
---
luoyuan3471 commented on PR #4726:
URL: https://github.com/apache/hadoop/pull/4726#issuecomment-1210237613
CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
require: csqueue.readLock
CapacityScheduler.schedule: hold: csqueue.readLock
require: PremmptionManager.readLock
other thread(completeContainer,release Resource,etc.): require:
csqueue.writeLock
@yb12138
schedule thread hold csqueue.readLock and it is blocked by
PremmptionManager.readLock , and PremmptionManager.writeLock is hold by
refreshQueue thread, seems refreshQueue have no chance to get csqueue.readLock.
Very sorry, I'm still a little confused on this point. Can you explain more
about it? Thank you！
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577800#comment-17577800
]
ASF GitHub Bot commented on YARN-11191:
---
yb12138 commented on PR #4726:
URL: https://github.com/apache/hadoop/pull/4726#issuecomment-1210219429
@luoyuan3471
1.The key to deadlock is that the refresh thread can‘t acquire the csqueue
read lock. beacuse it’s read lock request is blocked by a write lock (as:
https://bugs.openjdk.org/browse/JDK-6893626).so i use tryLock to break the
conditions for deadlock.The PremmptionManager lock will be released soon after
it gets csqueue read lock.
2.just preemption, but global scheduler increases the chance of deadlock
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577784#comment-17577784
]
Yuan Luo commented on YARN-11191:
-
I want to ask two question:
+ @Override
+ public List getChildQueuesByTryLock() {
+ try {
+ while (!readLock.tryLock()){
+ LockSupport.parkNanos(1);
+ }
+ return new ArrayList(childQueues);
+ } finally {
+ readLock.unlock();
+ }
+ }
1.Though you use tryLock and park, so refresh queue thread switch to block
state, but this thread still hold PremmptionManager lock ,so scheduler thread
still can't allocate new container. Is it right?
2.Does this issue related to global Scheduler or just the preemption function?
Looking forward to your reply, thanks!
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577780#comment-17577780
]
ASF GitHub Bot commented on YARN-11191:
---
luoyuan3471 commented on PR #4726:
URL: https://github.com/apache/hadoop/pull/4726#issuecomment-1210199622
@yb12138 I want to ask two question:
+ @Override
+ public List getChildQueuesByTryLock() {
+try {
+ while (!readLock.tryLock()){
+LockSupport.parkNanos(1);
+ }
+ return new ArrayList(childQueues);
+} finally {
+ readLock.unlock();
+}
+ }
1.Though you use tryLock and park, so refresh queue thread switch to block
state, but this thread still hold PremmptionManager lock ,so scheduler thread
still can allocate new container. Is it right?
2.Does this issue related to global Scheduler or just the preemption
function?
Looking forward to your reply, thanks!
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11191) Global Scheduler refreshQueue cause deadLock
[
https://issues.apache.org/jira/browse/YARN-11191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1757#comment-1757
]
ben yang commented on YARN-11191:
-
it is difficult to add a test, because this problem under concurrency not
always happen
> Global Scheduler refreshQueue cause deadLock
> -
>
> Key: YARN-11191
> URL: https://issues.apache.org/jira/browse/YARN-11191
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
>Affects Versions: 2.9.0, 3.0.0, 3.1.0, 2.10.0, 3.2.0, 3.3.0
>Reporter: ben yang
>Priority: Major
> Labels: pull-request-available
> Attachments: 1.jstack, YARN-11191.001.patch
>
>
> This is a potential bug may impact all open premmption cluster.In our
> current version with preemption enabled, the capacityScheduler will call the
> refreshQueue method of the PreemptionManager when it refreshQueue. This
> process hold the preemptionManager write lock and require csqueue read
> lock.Meanwhile,ParentQueue.canAssignToThisQueue will hold csqueue readLock
> and require PreemptionManager ReadLock.
> There is a possibility of deadlock at this time.Because readlock has one rule
> on unfair policy, when a lock is already occupied by a read lock and the
> first request in the lock competition queue is a write lock request,other
> read lock requests cann‘t acquire the lock.
> So the potential deadlock is:
> {code:java}
> CapacityScheduler.refreshQueue: hold: PremmptionManager.writeLock
> require: csqueue.readLock
> CapacityScheduler.schedule: hold: csqueue.readLock
> require: PremmptionManager.readLock
> other thread(completeContainer,release Resource,etc.): require:
> csqueue.writeLock
> {code}
> The jstack logs at the time were as follows
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
