[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[ https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753430#comment-17753430 ] ASF GitHub Bot commented on YARN-11153: --- slfan1989 commented on PR #4314: URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1675571426 @goiri Thank you very much for your help in reviewing the code! @zhengchenyu Thank you for your contribution! > Make proxy server support YARN federation. > -- > > Key: YARN-11153 > URL: https://issues.apache.org/jira/browse/YARN-11153 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 8h 40m > Remaining Estimate: 0h > > Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and > YARN-10775-design-doc.001.pdf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[ https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753337#comment-17753337 ] ASF GitHub Bot commented on YARN-11153: --- goiri merged PR #4314: URL: https://github.com/apache/hadoop/pull/4314 > Make proxy server support YARN federation. > -- > > Key: YARN-11153 > URL: https://issues.apache.org/jira/browse/YARN-11153 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 8h 40m > Remaining Estimate: 0h > > Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and > YARN-10775-design-doc.001.pdf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[ https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17753056#comment-17753056 ] ASF GitHub Bot commented on YARN-11153: --- slfan1989 commented on PR #4314: URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1674237206 @goiri Can you double check this pr again? Thank you very much! > Make proxy server support YARN federation. > -- > > Key: YARN-11153 > URL: https://issues.apache.org/jira/browse/YARN-11153 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 8h 40m > Remaining Estimate: 0h > > Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and > YARN-10775-design-doc.001.pdf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752035#comment-17752035
]
ASF GitHub Bot commented on YARN-11153:
---
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1669555702
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 38s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 26s | | trunk passed |
| +1 :green_heart: | compile | 1m 39s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 5s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 13s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 26m 31s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 48s | | the patch passed |
| +1 :green_heart: | compile | 1m 40s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 40s | | the patch passed |
| +1 :green_heart: | compile | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 42s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new +
67 unchanged - 2 fixed = 67 total (was 69) |
| +1 :green_heart: | mvnsite | 0m 52s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 0s | | hadoop-yarn-server-web-proxy in
the patch passed. |
| +1 :green_heart: | unit | 85m 58s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 190m 24s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/28/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 89dae1d090e4 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a16b568ac37c0acd36c80f2628f1e7b85017cfa7 |
| Default Java | Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| Test Results |
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[ https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17752031#comment-17752031 ] ASF GitHub Bot commented on YARN-11153: --- slfan1989 commented on PR #4314: URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1669545487 @zhengchenyu Thanks for the contribution! LGTM. > Make proxy server support YARN federation. > -- > > Key: YARN-11153 > URL: https://issues.apache.org/jira/browse/YARN-11153 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 8h 40m > Remaining Estimate: 0h > > Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and > YARN-10775-design-doc.001.pdf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751944#comment-17751944
]
ASF GitHub Bot commented on YARN-11153:
---
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1669146353
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 30s | | trunk passed |
| +1 :green_heart: | compile | 1m 43s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 34s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 5m 29s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| -1 :x: | spotbugs | 0m 43s |
[/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/27/artifact/out/branch-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt)
| hadoop-yarn-server-web-proxy in trunk failed. |
| +1 :green_heart: | shadedclient | 23m 46s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 4s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 1m 43s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 43s | | the patch passed |
| +1 :green_heart: | compile | 1m 31s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 43s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new +
67 unchanged - 2 fixed = 67 total (was 69) |
| +1 :green_heart: | mvnsite | 0m 54s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 15s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 59s | | hadoop-yarn-server-web-proxy in
the patch passed. |
| -1 :x: | unit | 86m 19s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/27/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 29s | | The patch does not
generate ASF License warnings. |
| | | 202m 37s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.reservation.TestCapacityOverTimePolicy |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/27/artifact/out/Dockerfile
|
| GITHUB PR |
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751914#comment-17751914
]
ASF GitHub Bot commented on YARN-11153:
---
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1669019094
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 15s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 23m 39s | | trunk passed |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 29s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 58s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 50s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 25m 7s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 51s | | the patch passed |
| +1 :green_heart: | compile | 1m 34s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javac | 1m 34s | | the patch passed |
| +1 :green_heart: | compile | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | javac | 1m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 42s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new +
67 unchanged - 2 fixed = 67 total (was 69) |
| +1 :green_heart: | mvnsite | 0m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 51s | | the patch passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 1m 1s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/26/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt)
| hadoop-yarn-server-web-proxy in the patch passed. |
| +1 :green_heart: | unit | 97m 30s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 205m 32s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.yarn.server.webproxy.TestWebAppProxyServlet |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/26/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6a9cc211415e 4.15.0-213-generic #224-Ubuntu SMP Mon Jun 19
13:30:12 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17751894#comment-17751894
]
ASF GitHub Bot commented on YARN-11153:
---
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1668903090
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 13m 51s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 22m 38s | | trunk passed |
| +1 :green_heart: | compile | 1m 39s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | compile | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | checkstyle | 0m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 57s | | trunk passed with JDK
Private Build-1.8.0_382-8u382-ga-1~20.04.1-b05 |
| +1 :green_heart: | spotbugs | 1m 47s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 34s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 50s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 13s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt)
| hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 1m 2s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| hadoop-yarn-server in the patch failed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | javac | 1m 2s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04.txt)
| hadoop-yarn-server in the patch failed with JDK
Ubuntu-11.0.20+8-post-Ubuntu-1ubuntu120.04. |
| -1 :x: | compile | 0m 54s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-yarn-server in the patch failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| -1 :x: | javac | 0m 54s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/25/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_382-8u382-ga-1~20.04.1-b05.txt)
| hadoop-yarn-server in the patch failed with JDK Private
Build-1.8.0_382-8u382-ga-1~20.04.1-b05. |
| -1
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17596991#comment-17596991
]
ASF GitHub Bot commented on YARN-11153:
---
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-122982
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 26s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 31s | | trunk passed |
| +1 :green_heart: | compile | 4m 3s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 31s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 30s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 7s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 25s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 48s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 3m 44s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 44s | | the patch passed |
| +1 :green_heart: | compile | 3m 15s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 15s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 12s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new +
74 unchanged - 24 fixed = 74 total (was 98) |
| +1 :green_heart: | mvnsite | 1m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 11s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 25s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 26s | | hadoop-yarn-server-web-proxy in
the patch passed. |
| +1 :green_heart: | unit | 117m 39s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 239m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/24/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b6d57850ef99 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ada446b9d7d9ddc81e4521bf9c9f9a28ad241dbe |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17596937#comment-17596937
]
ASF GitHub Bot commented on YARN-11153:
---
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r956841060
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java:
##
@@ -134,16 +132,19 @@ public WebAppProxyServlet() {
this.trackingUriPlugins =
conf.getInstances(YarnConfiguration.YARN_TRACKING_URL_GENERATOR,
TrackingUriPlugin.class);
-this.rmAppPageUrlBase =
-StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
- "cluster", "app");
this.failurePageUrlBase =
StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "failure");
-this.ahsAppPageUrlBase =
-StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
- + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
- "applicationhistory", "app");
+ }
+
+ private String getRmAppPageUrlBase(ApplicationId id) throws YarnException,
IOException {
+return ((AppReportFetcher)
getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
Review Comment:
Thanks for review, I will fix it.
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17596936#comment-17596936
]
ASF GitHub Bot commented on YARN-11153:
---
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r956840427
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import
org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+super(conf);
+subClusters = new ConcurrentHashMap<>();
+federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+createSubclusterIfAbsent(scid);
+ApplicationClientProtocol applicationsManager =
subClusters.get(scid).getRight();
+
+return super.getApplicationReport(applicationsManager, appId);
+ }
+
+ @Override
+ public String getRmAppPageUrlBase(ApplicationId appId)
+ throws IOException, YarnException {
+SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+createSubclusterIfAbsent(scid);
+
+SubClusterInfo subClusterInfo = subClusters.get(scid).getLeft();
+String scheme = WebAppUtils.getHttpSchemePrefix(getConf());
+return StringHelper.pjoin(scheme +
subClusterInfo.getRMWebServiceAddress(), "cluster", "app");
+ }
+
+ private void createSubclusterIfAbsent(SubClusterId scId) throws
YarnException, IOException {
+if (subClusters.containsKey(scId)) {
+ return;
+}
+SubClusterInfo subClusterInfo = federationFacade.getSubCluster(scId);
+Configuration subClusterConf = new Configuration(getConf());
+FederationProxyProviderUtil
+.updateConfForFederation(subClusterConf,
subClusterInfo.getSubClusterId().toString());
Review Comment:
Hi, I tried to remove toString. It will cause compile error. Warns "Required
type: String, Provided: SubClusterId".
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
>
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17585454#comment-17585454
]
ASF GitHub Bot commented on YARN-11153:
---
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r956166342
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java:
##
@@ -134,16 +132,19 @@ public WebAppProxyServlet() {
this.trackingUriPlugins =
conf.getInstances(YarnConfiguration.YARN_TRACKING_URL_GENERATOR,
TrackingUriPlugin.class);
-this.rmAppPageUrlBase =
-StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
- "cluster", "app");
this.failurePageUrlBase =
StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "failure");
-this.ahsAppPageUrlBase =
-StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
- + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
- "applicationhistory", "app");
+ }
+
+ private String getRmAppPageUrlBase(ApplicationId id) throws YarnException,
IOException {
+return ((AppReportFetcher)
getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
Review Comment:
For example, it can be extracted here to make the code easier to read. I
think the first way you provide is feasible.
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17585456#comment-17585456
]
ASF GitHub Bot commented on YARN-11153:
---
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r956166577
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java:
##
@@ -134,16 +132,19 @@ public WebAppProxyServlet() {
this.trackingUriPlugins =
conf.getInstances(YarnConfiguration.YARN_TRACKING_URL_GENERATOR,
TrackingUriPlugin.class);
-this.rmAppPageUrlBase =
-StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
- "cluster", "app");
this.failurePageUrlBase =
StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "failure");
-this.ahsAppPageUrlBase =
-StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
- + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
- "applicationhistory", "app");
+ }
+
+ private String getRmAppPageUrlBase(ApplicationId id) throws YarnException,
IOException {
+return ((AppReportFetcher)
getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
+.getRmAppPageUrlBase(id);
+ }
+
+ private String getAhsAppPageUrlBase() {
+return ((AppReportFetcher)
getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
Review Comment:
extracted too
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[ https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17585451#comment-17585451 ] ASF GitHub Bot commented on YARN-11153: --- slfan1989 commented on PR #4314: URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1228639768 Thank you very much for your contribution, overall it looks ok, but some code is better to extract. > Make proxy server support YARN federation. > -- > > Key: YARN-11153 > URL: https://issues.apache.org/jira/browse/YARN-11153 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 8h 40m > Remaining Estimate: 0h > > Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and > YARN-10775-design-doc.001.pdf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[ https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17585099#comment-17585099 ] ASF GitHub Bot commented on YARN-11153: --- zhengchenyu commented on PR #4314: URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1227976708 Is there any problem about this PR? How about merge trunk? Then I will continue YARN-11153. @goiri @slfan1989 > Make proxy server support YARN federation. > -- > > Key: YARN-11153 > URL: https://issues.apache.org/jira/browse/YARN-11153 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 8h 40m > Remaining Estimate: 0h > > Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and > YARN-10775-design-doc.001.pdf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17582718#comment-17582718
]
ASF GitHub Bot commented on YARN-11153:
---
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r950986663
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import
org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+super(conf);
+subClusters = new ConcurrentHashMap<>();
+federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+createSubclusterIfAbsent(scid);
+ApplicationClientProtocol applicationsManager =
subClusters.get(scid).getRight();
+
+return super.getApplicationReport(applicationsManager, appId);
+ }
+
+ @Override
+ public String getRmAppPageUrlBase(ApplicationId appId)
+ throws IOException, YarnException {
+SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+createSubclusterIfAbsent(scid);
+
+SubClusterInfo subClusterInfo = subClusters.get(scid).getLeft();
+String scheme = WebAppUtils.getHttpSchemePrefix(getConf());
+return StringHelper.pjoin(scheme +
subClusterInfo.getRMWebServiceAddress(), "cluster", "app");
+ }
+
+ private void createSubclusterIfAbsent(SubClusterId scId) throws
YarnException, IOException {
+if (subClusters.containsKey(scId)) {
+ return;
+}
+SubClusterInfo subClusterInfo = federationFacade.getSubCluster(scId);
+Configuration subClusterConf = new Configuration(getConf());
+FederationProxyProviderUtil
+.updateConfForFederation(subClusterConf,
subClusterInfo.getSubClusterId().toString());
Review Comment:
Of course not. but explicit call is not wrong.
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter:
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17582561#comment-17582561
]
ASF GitHub Bot commented on YARN-11153:
---
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r950851572
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import
org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+super(conf);
+subClusters = new ConcurrentHashMap<>();
+federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+createSubclusterIfAbsent(scid);
+ApplicationClientProtocol applicationsManager =
subClusters.get(scid).getRight();
+
+return super.getApplicationReport(applicationsManager, appId);
+ }
+
+ @Override
+ public String getRmAppPageUrlBase(ApplicationId appId)
+ throws IOException, YarnException {
+SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+createSubclusterIfAbsent(scid);
+
+SubClusterInfo subClusterInfo = subClusters.get(scid).getLeft();
+String scheme = WebAppUtils.getHttpSchemePrefix(getConf());
+return StringHelper.pjoin(scheme +
subClusterInfo.getRMWebServiceAddress(), "cluster", "app");
+ }
+
+ private void createSubclusterIfAbsent(SubClusterId scId) throws
YarnException, IOException {
+if (subClusters.containsKey(scId)) {
+ return;
+}
+SubClusterInfo subClusterInfo = federationFacade.getSubCluster(scId);
+Configuration subClusterConf = new Configuration(getConf());
+FederationProxyProviderUtil
+.updateConfForFederation(subClusterConf,
subClusterInfo.getSubClusterId().toString());
Review Comment:
subClusterInfo.getSubClusterId() should be enough
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter:
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17582560#comment-17582560
]
ASF GitHub Bot commented on YARN-11153:
---
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r950851516
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/FedAppReportFetcher.java:
##
@@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import
org.apache.hadoop.yarn.server.federation.failover.FederationProxyProviderUtil;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class FedAppReportFetcher extends AppReportFetcher {
+
+ private final Map> subClusters;
+ private FederationStateStoreFacade federationFacade;
+
+ /**
+ * Create a new Connection to the RM/Application History Server to fetch
+ * Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public FedAppReportFetcher(Configuration conf) {
+super(conf);
+subClusters = new ConcurrentHashMap<>();
+federationFacade = FederationStateStoreFacade.getInstance();
+ }
+
+ /**
+ * Get an application report for the specified application id from the RM and
+ * fall back to the Application History Server if not found in RM.
+ *
+ * @param appId id of the application to get.
+ * @return the ApplicationReport for the appId.
+ * @throws YarnException on any error.
+ * @throws IOException connection exception.
+ */
+ @Override
+ public FetchedAppReport getApplicationReport(ApplicationId appId)
+ throws YarnException, IOException {
+SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+createSubclusterIfAbsent(scid);
+ApplicationClientProtocol applicationsManager =
subClusters.get(scid).getRight();
+
+return super.getApplicationReport(applicationsManager, appId);
+ }
+
+ @Override
+ public String getRmAppPageUrlBase(ApplicationId appId)
+ throws IOException, YarnException {
+SubClusterId scid = federationFacade.getApplicationHomeSubCluster(appId);
+createSubclusterIfAbsent(scid);
+
+SubClusterInfo subClusterInfo = subClusters.get(scid).getLeft();
+String scheme = WebAppUtils.getHttpSchemePrefix(getConf());
+return StringHelper.pjoin(scheme +
subClusterInfo.getRMWebServiceAddress(), "cluster", "app");
+ }
+
+ private void createSubclusterIfAbsent(SubClusterId scId) throws
YarnException, IOException {
+if (subClusters.containsKey(scId)) {
+ return;
+}
+SubClusterInfo subClusterInfo = federationFacade.getSubCluster(scId);
+Configuration subClusterConf = new Configuration(getConf());
+FederationProxyProviderUtil
+.updateConfForFederation(subClusterConf,
subClusterInfo.getSubClusterId().toString());
Review Comment:
Is toString necessary?
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579566#comment-17579566
]
ASF GitHub Bot commented on YARN-11153:
---
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1214700469
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 57s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 17s | | trunk passed |
| +1 :green_heart: | compile | 4m 7s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 32s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 56s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 45s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 45s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 12s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 3m 47s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 47s | | the patch passed |
| +1 :green_heart: | compile | 3m 18s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 13s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new +
74 unchanged - 24 fixed = 74 total (was 98) |
| +1 :green_heart: | mvnsite | 1m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 15s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 19s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 28s | | hadoop-yarn-server-web-proxy in
the patch passed. |
| +1 :green_heart: | unit | 102m 45s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 226m 1s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/23/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b7888fa7415e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 75d7994581c9bf7e62b468b0c8f21fe86460ecf2 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579508#comment-17579508
]
ASF GitHub Bot commented on YARN-11153:
---
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945404189
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java:
##
@@ -134,16 +132,19 @@ public WebAppProxyServlet() {
this.trackingUriPlugins =
conf.getInstances(YarnConfiguration.YARN_TRACKING_URL_GENERATOR,
TrackingUriPlugin.class);
-this.rmAppPageUrlBase =
-StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
- "cluster", "app");
this.failurePageUrlBase =
StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "failure");
-this.ahsAppPageUrlBase =
-StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
- + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
- "applicationhistory", "app");
+ }
+
+ private String getRmAppPageUrlBase(ApplicationId id) throws YarnException,
IOException {
+return ((AppReportFetcher)
getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
Review Comment:
This code should show like this. If refactor is only for clean code. I think
it is not very clean. Maybe I did not catch your idea clearly.
```
ServletContext context = getServletContext();
AppReportFetcher report =
((AppReportFetcher)
context.getAttribute(WebAppProxy.FETCHER_ATTRIBUTE));
return report.getRmAppPageUrlBase(id);
```
or
```
AppReportFetcher report =
((AppReportFetcher)
getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE));
return report.getRmAppPageUrlBase(id);
```
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579506#comment-17579506
]
ASF GitHub Bot commented on YARN-11153:
---
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945404189
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java:
##
@@ -134,16 +132,19 @@ public WebAppProxyServlet() {
this.trackingUriPlugins =
conf.getInstances(YarnConfiguration.YARN_TRACKING_URL_GENERATOR,
TrackingUriPlugin.class);
-this.rmAppPageUrlBase =
-StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
- "cluster", "app");
this.failurePageUrlBase =
StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "failure");
-this.ahsAppPageUrlBase =
-StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
- + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
- "applicationhistory", "app");
+ }
+
+ private String getRmAppPageUrlBase(ApplicationId id) throws YarnException,
IOException {
+return ((AppReportFetcher)
getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
Review Comment:
This code should show like this. If refactor is only for clean code. I think
it is not very clean. Maybe I did not catch your idea clearly.
```
ServletContext context = getServletContext();
AppReportFetcher report =
((AppReportFetcher)
context.getAttribute(WebAppProxy.FETCHER_ATTRIBUTE));
return report.getRmAppPageUrlBase(id);
```
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579505#comment-17579505
]
ASF GitHub Bot commented on YARN-11153:
---
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945404189
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java:
##
@@ -134,16 +132,19 @@ public WebAppProxyServlet() {
this.trackingUriPlugins =
conf.getInstances(YarnConfiguration.YARN_TRACKING_URL_GENERATOR,
TrackingUriPlugin.class);
-this.rmAppPageUrlBase =
-StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
- "cluster", "app");
this.failurePageUrlBase =
StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "failure");
-this.ahsAppPageUrlBase =
-StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
- + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
- "applicationhistory", "app");
+ }
+
+ private String getRmAppPageUrlBase(ApplicationId id) throws YarnException,
IOException {
+return ((AppReportFetcher)
getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
Review Comment:
This code should show like this. If refactor is only for clean code. I think
it is not very clean.
```
ServletContext context = getServletContext();
AppReportFetcher report =
((AppReportFetcher)
context.getAttribute(WebAppProxy.FETCHER_ATTRIBUTE));
return report.getRmAppPageUrlBase(id);
```
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579480#comment-17579480
]
ASF GitHub Bot commented on YARN-11153:
---
goiri commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945385056
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/utils/FederationStateStoreFacade.java:
##
@@ -610,7 +610,6 @@ public boolean equals(Object obj) {
TResult invoke(T input) throws Exception;
}
-
Review Comment:
Avoid this change
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579422#comment-17579422
]
ASF GitHub Bot commented on YARN-11153:
---
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945302648
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/WebAppProxyServlet.java:
##
@@ -134,16 +132,19 @@ public WebAppProxyServlet() {
this.trackingUriPlugins =
conf.getInstances(YarnConfiguration.YARN_TRACKING_URL_GENERATOR,
TrackingUriPlugin.class);
-this.rmAppPageUrlBase =
-StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
- "cluster", "app");
this.failurePageUrlBase =
StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "failure");
-this.ahsAppPageUrlBase =
-StringHelper.pjoin(WebAppUtils.getHttpSchemePrefix(conf)
- + WebAppUtils.getAHSWebAppURLWithoutScheme(conf),
- "applicationhistory", "app");
+ }
+
+ private String getRmAppPageUrlBase(ApplicationId id) throws YarnException,
IOException {
+return ((AppReportFetcher)
getServletContext().getAttribute(WebAppProxy.FETCHER_ATTRIBUTE))
Review Comment:
This line of code needs to be refactored, too many layers and not easy to
read.
This way of writing might be better.
```
AppReportFetcher appReportFetcher = (AppReportFetcher) getServletContext();
Attribute attr1 =
appReportFetcher.getAttribute(WebAppProxy.FETCHER_ATTRIBUTE);
return attr1.getAhsAppPageUrlBase();
```
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579395#comment-17579395
]
ASF GitHub Bot commented on YARN-11153:
---
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1214385755
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 34s | | trunk passed |
| +1 :green_heart: | compile | 4m 8s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 36s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 35s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 8s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 53s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 37s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 55s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 23s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 8s | | the patch passed |
| +1 :green_heart: | compile | 3m 53s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 53s | | the patch passed |
| +1 :green_heart: | compile | 3m 17s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 14s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new +
84 unchanged - 24 fixed = 84 total (was 108) |
| +1 :green_heart: | mvnsite | 2m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 4s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 57s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 5s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 1m 28s | | hadoop-yarn-server-web-proxy in
the patch passed. |
| +1 :green_heart: | unit | 103m 0s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 238m 44s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/22/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 6503ca331e17 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 22d57d78a676cfdfbb10d220fa50ac9c85ad5fa6 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579350#comment-17579350
]
ASF GitHub Bot commented on YARN-11153:
---
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945259806
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import
org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import
org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import
org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import static org.junit.Assert.fail;
+
+public class TestFedAppReportFetcher {
+
+ private Configuration conf;
+ private static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 =
SubClusterInfo.newInstance(subClusterId1, "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", "10.0.0.1:1000",
SubClusterState.SC_RUNNING, 0L, "");
+ private SubClusterInfo clusterInfo2 =
SubClusterInfo.newInstance(subClusterId2, "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", "10.0.0.2:1000",
SubClusterState.SC_RUNNING, 0L, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
Review Comment:
Of couse could. But is it necessary? Here just construct a easy object!
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579346#comment-17579346
]
ASF GitHub Bot commented on YARN-11153:
---
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1214309067
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 47s | | trunk passed |
| +1 :green_heart: | compile | 4m 5s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 33s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 54s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 38s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 59s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 24s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 51s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 25s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt)
| hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | mvninstall | 0m 40s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 2m 3s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| hadoop-yarn-server in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 2m 3s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| hadoop-yarn-server in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 1m 51s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| hadoop-yarn-server in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 1m 51s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/21/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| hadoop-yarn-server in
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[ https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579342#comment-17579342 ] ASF GitHub Bot commented on YARN-11153: --- slfan1989 commented on PR #4314: URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1214305205 @zhengchenyu Thanks a lot for your contribution, need to fix some checkstyle issues. > Make proxy server support YARN federation. > -- > > Key: YARN-11153 > URL: https://issues.apache.org/jira/browse/YARN-11153 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 8h 40m > Remaining Estimate: 0h > > Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and > YARN-10775-design-doc.001.pdf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579341#comment-17579341
]
ASF GitHub Bot commented on YARN-11153:
---
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945245127
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import
org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import
org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import
org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import static org.junit.Assert.fail;
+
+public class TestFedAppReportFetcher {
+
+ private Configuration conf;
+ private static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 =
SubClusterInfo.newInstance(subClusterId1, "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", "10.0.0.1:1000",
SubClusterState.SC_RUNNING, 0L, "");
+ private SubClusterInfo clusterInfo2 =
SubClusterInfo.newInstance(subClusterId2, "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", "10.0.0.2:1000",
SubClusterState.SC_RUNNING, 0L, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
+ private static FedAppReportFetcher fetcher;
+ private final String appNotFoundExceptionMsg = "APP NOT FOUND";
+
+ @After
+ public void cleanUp() {
+history = null;
+fetcher = null;
+ }
+
+ private void testHelper(boolean isAHSEnabled)
+ throws YarnException, IOException {
+conf = new YarnConfiguration();
+conf.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
isAHSEnabled);
+
+FederationStateStoreFacade fedFacade =
FederationStateStoreFacade.getInstance();
+FederationStateStore fss = new MemoryFederationStateStore();
+fss.init(conf);
+fedFacade.reinitialize(fss, conf);
+
+
fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo1));
+
fss.registerSubCluster(SubClusterRegisterRequest.newInstance(clusterInfo2));
+fss.addApplicationHomeSubCluster(AddApplicationHomeSubClusterRequest
+.newInstance(ApplicationHomeSubCluster.newInstance(appId1,
subClusterId1)));
+
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579340#comment-17579340
]
ASF GitHub Bot commented on YARN-11153:
---
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945245040
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestFedAppReportFetcher.java:
##
@@ -0,0 +1,170 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.ApplicationHistoryProtocol;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportRequest;
+import org.apache.hadoop.yarn.api.protocolrecords.GetApplicationReportResponse;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.server.federation.store.FederationStateStore;
+import
org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import
org.apache.hadoop.yarn.server.federation.store.records.AddApplicationHomeSubClusterRequest;
+import
org.apache.hadoop.yarn.server.federation.store.records.ApplicationHomeSubCluster;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+import org.mockito.Mockito;
+
+import static org.junit.Assert.fail;
+
+public class TestFedAppReportFetcher {
+
+ private Configuration conf;
+ private static ApplicationHistoryProtocol history;
+
+ private SubClusterId subClusterId1 = SubClusterId.newInstance("subCluster1");
+ private SubClusterId subClusterId2 = SubClusterId.newInstance("subCluster2");
+ private SubClusterInfo clusterInfo1 =
SubClusterInfo.newInstance(subClusterId1, "10.0.0.1:1000",
+ "10.0.0.1:1000", "10.0.0.1:1000", "10.0.0.1:1000",
SubClusterState.SC_RUNNING, 0L, "");
+ private SubClusterInfo clusterInfo2 =
SubClusterInfo.newInstance(subClusterId2, "10.0.0.2:1000",
+ "10.0.0.2:1000", "10.0.0.2:1000", "10.0.0.2:1000",
SubClusterState.SC_RUNNING, 0L, "");
+ private ApplicationClientProtocol appManager1;
+ private ApplicationClientProtocol appManager2;
+ private ApplicationId appId1 = ApplicationId.newInstance(0, 1);
+ private ApplicationId appId2 = ApplicationId.newInstance(0, 2);
+
Review Comment:
Can these parameters be initialized in setUp?
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579339#comment-17579339
]
ASF GitHub Bot commented on YARN-11153:
---
slfan1989 commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945244909
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/DefaultAppReportFetcher.java:
##
@@ -0,0 +1,95 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with this
+ * work for additional information regarding copyright ownership. The ASF
+ * licenses this file to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
+ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
+ * License for the specific language governing permissions and limitations
under
+ * the License.
+ */
+
+package org.apache.hadoop.yarn.server.webproxy;
+
+import java.io.IOException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.ipc.RPC;
+import org.apache.hadoop.yarn.api.ApplicationClientProtocol;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.hadoop.yarn.client.ClientRMProxy;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
+
+public class DefaultAppReportFetcher extends AppReportFetcher {
+
+ private final ApplicationClientProtocol applicationsManager;
+ private String rmAppPageUrlBase;
+
+ /**
+ * Create a new Connection to the RM/Application History Server
+ * to fetch Application reports.
+ *
+ * @param conf the conf to use to know where the RM is.
+ */
+ public DefaultAppReportFetcher(Configuration conf) {
+super(conf);
+this.rmAppPageUrlBase =
+StringHelper.pjoin(WebAppUtils.getResolvedRMWebAppURLWithScheme(conf),
"cluster", "app");
+try {
+ this.applicationsManager = ClientRMProxy.createRMProxy(conf,
ApplicationClientProtocol.class);
+} catch (IOException e) {
+ throw new YarnRuntimeException(e);
+}
+ }
+
+ /**
+ * Create a direct connection to RM instead of a remote connection when
+ * the proxy is running as part of the RM. Also create a remote connection to
+ * Application History Server if it is enabled.
+ *
+ * @param confthe configuration to use
+ * @param applicationsManager what to use to get the RM reports.
+ */
+ public DefaultAppReportFetcher(Configuration conf,
+ ApplicationClientProtocol
applicationsManager) {
Review Comment:
indentation.
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579303#comment-17579303
]
ASF GitHub Bot commented on YARN-11153:
---
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1214265428
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 22s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 9s | | trunk passed |
| +1 :green_heart: | compile | 3m 46s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 14s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 14s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 3s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 6s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 56s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 15s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 17s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-web-proxy.txt)
| hadoop-yarn-server-web-proxy in the patch failed. |
| -1 :x: | mvninstall | 0m 31s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 1m 44s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| hadoop-yarn-server in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 1m 44s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| hadoop-yarn-server in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 1m 36s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| hadoop-yarn-server in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 1m 36s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/20/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| hadoop-yarn-server in
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579264#comment-17579264
]
ASF GitHub Bot commented on YARN-11153:
---
goiri commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945172390
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java:
##
@@ -27,81 +29,54 @@
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.client.AHSProxy;
-import org.apache.hadoop.yarn.client.ClientRMProxy;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
import org.apache.hadoop.yarn.factories.RecordFactory;
import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
/**
* This class abstracts away how ApplicationReports are fetched.
*/
-public class AppReportFetcher {
- enum AppReportSource { RM, AHS }
+public abstract class AppReportFetcher {
+
+ protected enum AppReportSource {RM, AHS}
+
private final Configuration conf;
- private final ApplicationClientProtocol applicationsManager;
- private final ApplicationHistoryProtocol historyManager;
+ private ApplicationHistoryProtocol historyManager;
+ private String ahsAppPageUrlBase;
private final RecordFactory recordFactory =
RecordFactoryProvider.getRecordFactory(null);
private boolean isAHSEnabled;
- /**
- * Create a new Connection to the RM/Application History Server
- * to fetch Application reports.
- * @param conf the conf to use to know where the RM is.
- */
public AppReportFetcher(Configuration conf) {
+this.conf = conf;
if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
+ this.isAHSEnabled = true;
+ String scheme = WebAppUtils.getHttpSchemePrefix(conf);
+ String historyUrl = WebAppUtils.getAHSWebAppURLWithoutScheme(conf);
+ this.ahsAppPageUrlBase = StringHelper.pjoin(scheme + historyUrl,
"applicationhistory", "app");
}
-this.conf = conf;
try {
- applicationsManager = ClientRMProxy.createRMProxy(conf,
- ApplicationClientProtocol.class);
- if (isAHSEnabled) {
-historyManager = getAHSProxy(conf);
+ if (this.isAHSEnabled) {
+this.historyManager = getAHSProxy(conf);
} else {
this.historyManager = null;
}
} catch (IOException e) {
throw new YarnRuntimeException(e);
}
}
-
- /**
- * Create a direct connection to RM instead of a remote connection when
- * the proxy is running as part of the RM. Also create a remote connection to
- * Application History Server if it is enabled.
- * @param conf the configuration to use
- * @param applicationsManager what to use to get the RM reports.
- */
- public AppReportFetcher(Configuration conf, ApplicationClientProtocol
applicationsManager) {
-if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
-YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
-}
-this.conf = conf;
-this.applicationsManager = applicationsManager;
-if (isAHSEnabled) {
- try {
-historyManager = getAHSProxy(conf);
- } catch (IOException e) {
-throw new YarnRuntimeException(e);
- }
-} else {
- this.historyManager = null;
-}
- }
protected ApplicationHistoryProtocol getAHSProxy(Configuration configuration)
throws IOException {
-return AHSProxy.createAHSProxy(configuration,
- ApplicationHistoryProtocol.class,
- configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
-YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
-YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT));
+return AHSProxy.createAHSProxy(configuration,
ApplicationHistoryProtocol.class,
+configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
+YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
+YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT));
Review Comment:
I think the cleanest is to do:
```
InetSocketAddress addr =
configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT)
return AHSProxy.createAHSProxy(configuration,
ApplicationHistoryProtocol.class, addr);
```
>
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579249#comment-17579249
]
ASF GitHub Bot commented on YARN-11153:
---
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1214164427
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 25s | | trunk passed |
| +1 :green_heart: | compile | 4m 4s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 31s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 34s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 53s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 37s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 0s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 58s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 25s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 32s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 8s | | the patch passed |
| +1 :green_heart: | compile | 3m 47s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 47s | | the patch passed |
| +1 :green_heart: | compile | 3m 21s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 15s | |
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 0 new +
85 unchanged - 24 fixed = 85 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 19s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 1s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 56s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 37s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 4s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 1m 29s | | hadoop-yarn-server-web-proxy in
the patch passed. |
| +1 :green_heart: | unit | 102m 29s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 238m 16s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/19/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3425f047a000 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 05854d89656769840416945c719509ac13ceeddd |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579232#comment-17579232
]
ASF GitHub Bot commented on YARN-11153:
---
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r945120827
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java:
##
@@ -361,8 +362,8 @@ public void testAppReportForEmptyTrackingUrl() throws
Exception {
proxyConn.getURL().toString().equals(appAddressInRm));
//set AHS_ENBALED = true to simulate getting the app report from AHS
-configuration.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
-true);
+configuration.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
true);
+proxy.proxy.appReportFetcher.setAhsAppPageUrlBase(configuration);
Review Comment:
Because the code style of original code is wrong, I will fix all.
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579033#comment-17579033
]
ASF GitHub Bot commented on YARN-11153:
---
goiri commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r944610371
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java:
##
@@ -361,8 +362,8 @@ public void testAppReportForEmptyTrackingUrl() throws
Exception {
proxyConn.getURL().toString().equals(appAddressInRm));
//set AHS_ENBALED = true to simulate getting the app report from AHS
-configuration.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
-true);
+configuration.setBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
true);
+proxy.proxy.appReportFetcher.setAhsAppPageUrlBase(configuration);
Review Comment:
Not sure why checkstyle complains about this.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/test/java/org/apache/hadoop/yarn/server/webproxy/TestWebAppProxyServlet.java:
##
@@ -375,8 +376,8 @@ public void testAppReportForEmptyTrackingUrl() throws
Exception {
String appAddressInAhs = WebAppUtils.getHttpSchemePrefix(configuration) +
WebAppUtils.getAHSWebAppURLWithoutScheme(configuration) +
"/applicationhistory" + "/app/" + app.toString();
-assertTrue("Webapp proxy servlet should have redirected to AHS",
-proxyConn.getURL().toString().equals(appAddressInAhs));
+ assertEquals("Webapp proxy servlet should have redirected to AHS",
appAddressInAhs,
Review Comment:
I think the indentation is broken here.
> Make proxy server support YARN federation.
> --
>
> Key: YARN-11153
> URL: https://issues.apache.org/jira/browse/YARN-11153
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.2.1
>Reporter: zhengchenyu
>Assignee: zhengchenyu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 8h 40m
> Remaining Estimate: 0h
>
> Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and
> YARN-10775-design-doc.001.pdf
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[ https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578950#comment-17578950 ] ASF GitHub Bot commented on YARN-11153: --- zhengchenyu commented on PR #4314: URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1213071121 @goiri How about merge to trunk? > Make proxy server support YARN federation. > -- > > Key: YARN-11153 > URL: https://issues.apache.org/jira/browse/YARN-11153 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.2.1 >Reporter: zhengchenyu >Assignee: zhengchenyu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 8h 40m > Remaining Estimate: 0h > > Detail message see: https://issues.apache.org/jira/browse/YARN-10775 and > YARN-10775-design-doc.001.pdf -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17578935#comment-17578935
]
ASF GitHub Bot commented on YARN-11153:
---
hadoop-yetus commented on PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#issuecomment-1213025113
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 0s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 2s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 14s | | trunk passed |
| +1 :green_heart: | compile | 4m 8s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 32s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 32s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 10s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 47s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 39s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 30s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 55s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 3s | | the patch passed |
| +1 :green_heart: | compile | 3m 49s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 3m 49s | | the patch passed |
| +1 :green_heart: | compile | 3m 19s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 13s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/18/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 2
new + 105 unchanged - 4 fixed = 107 total (was 109) |
| +1 :green_heart: | mvnsite | 2m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 2s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 54s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 36s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 37s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 3m 5s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 1m 27s | | hadoop-yarn-server-web-proxy in
the patch passed. |
| +1 :green_heart: | unit | 102m 52s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 237m 57s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4314/18/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4314 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 28d010b9761e 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (YARN-11153) Make proxy server support YARN federation.
[
https://issues.apache.org/jira/browse/YARN-11153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17577077#comment-17577077
]
ASF GitHub Bot commented on YARN-11153:
---
zhengchenyu commented on code in PR #4314:
URL: https://github.com/apache/hadoop/pull/4314#discussion_r940821669
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-web-proxy/src/main/java/org/apache/hadoop/yarn/server/webproxy/AppReportFetcher.java:
##
@@ -27,81 +29,54 @@
import org.apache.hadoop.yarn.api.records.ApplicationId;
import org.apache.hadoop.yarn.api.records.ApplicationReport;
import org.apache.hadoop.yarn.client.AHSProxy;
-import org.apache.hadoop.yarn.client.ClientRMProxy;
import org.apache.hadoop.yarn.conf.YarnConfiguration;
import org.apache.hadoop.yarn.exceptions.ApplicationNotFoundException;
import org.apache.hadoop.yarn.exceptions.YarnException;
import org.apache.hadoop.yarn.exceptions.YarnRuntimeException;
import org.apache.hadoop.yarn.factories.RecordFactory;
import org.apache.hadoop.yarn.factory.providers.RecordFactoryProvider;
+import org.apache.hadoop.yarn.util.StringHelper;
+import org.apache.hadoop.yarn.webapp.util.WebAppUtils;
/**
* This class abstracts away how ApplicationReports are fetched.
*/
-public class AppReportFetcher {
- enum AppReportSource { RM, AHS }
+public abstract class AppReportFetcher {
+
+ protected enum AppReportSource {RM, AHS}
+
private final Configuration conf;
- private final ApplicationClientProtocol applicationsManager;
- private final ApplicationHistoryProtocol historyManager;
+ private ApplicationHistoryProtocol historyManager;
+ private String ahsAppPageUrlBase;
private final RecordFactory recordFactory =
RecordFactoryProvider.getRecordFactory(null);
private boolean isAHSEnabled;
- /**
- * Create a new Connection to the RM/Application History Server
- * to fetch Application reports.
- * @param conf the conf to use to know where the RM is.
- */
public AppReportFetcher(Configuration conf) {
+this.conf = conf;
if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
+ this.isAHSEnabled = true;
+ String scheme = WebAppUtils.getHttpSchemePrefix(conf);
+ String historyUrl = WebAppUtils.getAHSWebAppURLWithoutScheme(conf);
+ this.ahsAppPageUrlBase = StringHelper.pjoin(scheme + historyUrl,
"applicationhistory", "app");
}
-this.conf = conf;
try {
- applicationsManager = ClientRMProxy.createRMProxy(conf,
- ApplicationClientProtocol.class);
- if (isAHSEnabled) {
-historyManager = getAHSProxy(conf);
+ if (this.isAHSEnabled) {
+this.historyManager = getAHSProxy(conf);
} else {
this.historyManager = null;
}
} catch (IOException e) {
throw new YarnRuntimeException(e);
}
}
-
- /**
- * Create a direct connection to RM instead of a remote connection when
- * the proxy is running as part of the RM. Also create a remote connection to
- * Application History Server if it is enabled.
- * @param conf the configuration to use
- * @param applicationsManager what to use to get the RM reports.
- */
- public AppReportFetcher(Configuration conf, ApplicationClientProtocol
applicationsManager) {
-if (conf.getBoolean(YarnConfiguration.APPLICATION_HISTORY_ENABLED,
-YarnConfiguration.DEFAULT_APPLICATION_HISTORY_ENABLED)) {
- isAHSEnabled = true;
-}
-this.conf = conf;
-this.applicationsManager = applicationsManager;
-if (isAHSEnabled) {
- try {
-historyManager = getAHSProxy(conf);
- } catch (IOException e) {
-throw new YarnRuntimeException(e);
- }
-} else {
- this.historyManager = null;
-}
- }
protected ApplicationHistoryProtocol getAHSProxy(Configuration configuration)
throws IOException {
-return AHSProxy.createAHSProxy(configuration,
- ApplicationHistoryProtocol.class,
- configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
-YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
-YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT));
+return AHSProxy.createAHSProxy(configuration,
ApplicationHistoryProtocol.class,
+configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
+YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
+YarnConfiguration.DEFAULT_TIMELINE_SERVICE_PORT));
Review Comment:
> I think the indentation is correct in both cases. TBH, I think what would
help here is to extract,
Should I need to extract below code to a single function?
```
configuration.getSocketAddr(YarnConfiguration.TIMELINE_SERVICE_ADDRESS,
YarnConfiguration.DEFAULT_TIMELINE_SERVICE_ADDRESS,
