[GitHub] [hadoop] hadoop-yetus commented on pull request #4793: YARN-11276. Add lru cache for RMWebServices.getApps
hadoop-yetus commented on PR #4793:
URL: https://github.com/apache/hadoop/pull/4793#issuecomment-1225209958
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 21s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 12s | | trunk passed |
| +1 :green_heart: | compile | 10m 5s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 9m 9s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 19s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 21s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 3m 49s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 51s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/5/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | compile | 3m 51s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/5/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| hadoop-yarn in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 3m 51s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/5/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| hadoop-yarn in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 3m 30s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/5/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| hadoop-yarn in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 3m 30s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/5/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| hadoop-yarn in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 31s | | the patch passed |
| -1 :x: | mvnsite | 0m 57s |
[/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/5/artifact/out/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| +1 :green_heart: | javadoc | 2m 22s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 9s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | spotbugs
[jira] [Commented] (HADOOP-18302) Remove WhiteBox in hadoop-common module.
[
https://issues.apache.org/jira/browse/HADOOP-18302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583999#comment-17583999
]
ASF GitHub Bot commented on HADOOP-18302:
-
hadoop-yetus commented on PR #4457:
URL: https://github.com/apache/hadoop/pull/4457#issuecomment-1225207531
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 11 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 48s | | trunk passed |
| +1 :green_heart: | compile | 23m 20s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 47s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 0s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 32s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 45s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 15s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 35s | | the patch passed |
| +1 :green_heart: | compile | 22m 25s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 25s | |
root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new + 2833 unchanged - 17
fixed = 2833 total (was 2850) |
| +1 :green_heart: | compile | 20m 38s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 38s | |
root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 2629 unchanged -
17 fixed = 2629 total (was 2646) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 48s | | hadoop-common-project:
The patch generated 0 new + 377 unchanged - 233 fixed = 377 total (was 610) |
| +1 :green_heart: | mvnsite | 3m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 39s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 12s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 35s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 1m 28s | | hadoop-nfs in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 15s | | The patch does not
generate ASF License warnings. |
| | | 230m 24s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4457/25/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4457 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9f5883da13fb 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk /
[GitHub] [hadoop] hadoop-yetus commented on pull request #4457: HADOOP-18302. Remove WhiteBox in hadoop-common module.
hadoop-yetus commented on PR #4457:
URL: https://github.com/apache/hadoop/pull/4457#issuecomment-1225207531
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 11 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 48s | | trunk passed |
| +1 :green_heart: | compile | 23m 20s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 47s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 58s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 0s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 32s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 45s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 15s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 35s | | the patch passed |
| +1 :green_heart: | compile | 22m 25s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 25s | |
root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0 new + 2833 unchanged - 17
fixed = 2833 total (was 2850) |
| +1 :green_heart: | compile | 20m 38s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 38s | |
root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 0 new + 2629 unchanged -
17 fixed = 2629 total (was 2646) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 48s | | hadoop-common-project:
The patch generated 0 new + 377 unchanged - 233 fixed = 377 total (was 610) |
| +1 :green_heart: | mvnsite | 3m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 39s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 12s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 35s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 1m 28s | | hadoop-nfs in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 15s | | The patch does not
generate ASF License warnings. |
| | | 230m 24s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4457/25/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4457 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 9f5883da13fb 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 57ecde40aa26ca7b72d84c7b63da17e1adf331a9 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
[GitHub] [hadoop] hadoop-yetus commented on pull request #4792: YARN-11275. [Federation] Add batchFinishApplicationMaster in UAMPoolManager.
hadoop-yetus commented on PR #4792:
URL: https://github.com/apache/hadoop/pull/4792#issuecomment-1225198391
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 44s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 40s | | trunk passed |
| +1 :green_heart: | compile | 9m 55s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 8m 57s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 17s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 5s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 33s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 10s | | the patch passed |
| +1 :green_heart: | compile | 9m 7s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 9m 7s | | the patch passed |
| +1 :green_heart: | compile | 8m 43s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 8m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 2s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 53s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 41s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 3m 35s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 16s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 11s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 16s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 24m 18s | | hadoop-yarn-server-nodemanager
in the patch passed. |
| +1 :green_heart: | asflicense | 1m 15s | | The patch does not
generate ASF License warnings. |
| | | 204m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4792/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4792 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux c76fa740686d 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 46f8154ebba3818f17517f5405fab37d18303bba |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4792/3/testReport/ |
| Max. process+thread count | 731 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
U:
[jira] [Assigned] (HADOOP-18189) S3PrefetchingInputStream to support status probes when closed
[ https://issues.apache.org/jira/browse/HADOOP-18189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani reassigned HADOOP-18189: - Assignee: Viraj Jasani > S3PrefetchingInputStream to support status probes when closed > - > > Key: HADOOP-18189 > URL: https://issues.apache.org/jira/browse/HADOOP-18189 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Minor > > S3PrefetchingInputStream is a bit over aggressive on raising > exceptions/downgrading responses after a stream is closed > * MUST: getPos() to return last read location, or a least 0 (maybe we should > add this to filesystem spec) > * MUST: getIOStatistics(). critical for collecting stats in processes > * MUST: seekToNewSource() (it's a no op anyway) > * MAY S3AInputStreamStatistics and getStreamStatistics() though that is only > in used in testing...ioStatistics have given us a stable stats api. we may > want to tag the method @VisibleForTesting to discourage use. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4746: YARN-9708. Yarn Federation Router Support DelegationToken.
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953335382
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/secure/TestRouterDelegationTokenSecretManager.java:
##
@@ -0,0 +1,198 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.secure;
+
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.apache.hadoop.util.Time;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+import org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService;
+import
org.apache.hadoop.yarn.server.router.security.RouterDelegationTokenSecretManager;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+public class TestRouterDelegationTokenSecretManager extends
AbstractSecureRouterTest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TestRouterDelegationTokenSecretManager.class);
+
+ @Test
+ public void testRouterStoreNewMasterKey() throws Exception {
+LOG.info("Test RouterDelegationTokenSecretManager: StoreNewMasterKey.");
+
+// Start the Router in Secure Mode
+startSecureRouter();
+
+// Store NewMasterKey
+RouterClientRMService routerClientRMService =
this.getRouter().getClientRMProxyService();
+RouterDelegationTokenSecretManager secretManager =
+routerClientRMService.getRouterDTSecretManager();
+DelegationKey storeKey = new DelegationKey(1234, 4321,
"keyBytes".getBytes());
+secretManager.storeNewMasterKey(storeKey);
+
+// Get DelegationKey
+DelegationKey paramKey = new DelegationKey(1234, 4321,
"keyBytes".getBytes());
+DelegationKey responseKey =
secretManager.getMasterKeyByDelegationKey(paramKey);
+
+Assert.assertNotNull(paramKey);
+Assert.assertEquals(storeKey.getExpiryDate(), responseKey.getExpiryDate());
+Assert.assertEquals(storeKey.getKeyId(), responseKey.getKeyId());
+Assert.assertTrue(Arrays.equals(storeKey.getEncodedKey(),
responseKey.getEncodedKey()));
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18186) s3a prefetching to use SemaphoredDelegatingExecutor for submitting work
[ https://issues.apache.org/jira/browse/HADOOP-18186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583990#comment-17583990 ] ASF GitHub Bot commented on HADOOP-18186: - virajjasani commented on PR #4796: URL: https://github.com/apache/hadoop/pull/4796#issuecomment-1225182580 With `scale`: ``` $ mvn -Dparallel-tests -DtestsThreadCount=10 -Dscale clean verify [INFO] - > s3a prefetching to use SemaphoredDelegatingExecutor for submitting work > --- > > Key: HADOOP-18186 > URL: https://issues.apache.org/jira/browse/HADOOP-18186 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Use SemaphoredDelegatingExecutor for each to stream to submit work, if > possible, for better fairness in processes with many streams. > this also takes a DurationTrackerFactory to count how long was spent in the > queue, something we would want to know -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4796: HADOOP-18186. s3a prefetching to use SemaphoredDelegatingExecutor for submitting work
virajjasani commented on PR #4796: URL: https://github.com/apache/hadoop/pull/4796#issuecomment-1225182580 With `scale`: ``` $ mvn -Dparallel-tests -DtestsThreadCount=10 -Dscale clean verify [INFO] --- maven-surefire-plugin:3.0.0-M1:test (default-test) @ hadoop-aws --- [INFO] [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Results: [INFO] [WARNING] Tests run: 426, Failures: 0, Errors: 0, Skipped: 4 [INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (default-integration-test) @ hadoop-aws --- [INFO] [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Results: [INFO] [WARNING] Tests run: 1143, Failures: 0, Errors: 0, Skipped: 146 [INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (sequential-integration-tests) @ hadoop-aws --- [INFO] [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Results: [INFO] [ERROR] Errors: [ERROR] ITestS3AContractRootDir>AbstractContractRootDirectoryTest.testRecursiveRootListing:267 » TestTimedOut [INFO] [ERROR] Tests run: 124, Failures: 0, Errors: 1, Skipped: 10 [ERROR] testRecursiveRootListing(org.apache.hadoop.fs.contract.s3a.ITestS3AContractRootDir) Time elapsed: 180.127 s <<< ERROR! org.junit.runners.model.TestTimedOutException: test timed out after 18 milliseconds ``` When ran individually, `ITestS3AContractRootDir` tests passed without any issues. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18186) s3a prefetching to use SemaphoredDelegatingExecutor for submitting work
[ https://issues.apache.org/jira/browse/HADOOP-18186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583988#comment-17583988 ] ASF GitHub Bot commented on HADOOP-18186: - virajjasani commented on PR #4796: URL: https://github.com/apache/hadoop/pull/4796#issuecomment-1225181844 Endpoint `us-west-2`: ``` $ mvn -Dparallel-tests -DtestsThreadCount=8 clean verify [INFO] - > s3a prefetching to use SemaphoredDelegatingExecutor for submitting work > --- > > Key: HADOOP-18186 > URL: https://issues.apache.org/jira/browse/HADOOP-18186 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Use SemaphoredDelegatingExecutor for each to stream to submit work, if > possible, for better fairness in processes with many streams. > this also takes a DurationTrackerFactory to count how long was spent in the > queue, something we would want to know -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4746: YARN-9708. Yarn Federation Router Support DelegationToken.
hadoop-yetus commented on PR #4746: URL: https://github.com/apache/hadoop/pull/4746#issuecomment-1225181917 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 19s | | https://github.com/apache/hadoop/pull/4746 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/4746 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4746/7/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4796: HADOOP-18186. s3a prefetching to use SemaphoredDelegatingExecutor for submitting work
virajjasani commented on PR #4796: URL: https://github.com/apache/hadoop/pull/4796#issuecomment-1225181844 Endpoint `us-west-2`: ``` $ mvn -Dparallel-tests -DtestsThreadCount=8 clean verify [INFO] --- maven-surefire-plugin:3.0.0-M1:test (default-test) @ hadoop-aws --- [INFO] [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Results: [INFO] [WARNING] Tests run: 426, Failures: 0, Errors: 0, Skipped: 4 [INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (default-integration-test) @ hadoop-aws --- [INFO] [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Results: [INFO] [WARNING] Tests run: 1143, Failures: 0, Errors: 0, Skipped: 186 [INFO] --- maven-failsafe-plugin:3.0.0-M1:integration-test (sequential-integration-tests) @ hadoop-aws --- [INFO] [INFO] --- [INFO] T E S T S [INFO] --- [INFO] Results: [INFO] [WARNING] Tests run: 124, Failures: 0, Errors: 0, Skipped: 84 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18186) s3a prefetching to use SemaphoredDelegatingExecutor for submitting work
[
https://issues.apache.org/jira/browse/HADOOP-18186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583985#comment-17583985
]
ASF GitHub Bot commented on HADOOP-18186:
-
hadoop-yetus commented on PR #4796:
URL: https://github.com/apache/hadoop/pull/4796#issuecomment-1225178955
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 24s | | trunk passed |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 46s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 105m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4796/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4796 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 78f4d719d52a 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c94267a96e6c6ed152bf5265625dd2c32d8141ea |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4796/1/testReport/ |
| Max. process+thread count | 633 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4796/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus
[jira] [Updated] (HADOOP-18186) s3a prefetching to use SemaphoredDelegatingExecutor for submitting work
[ https://issues.apache.org/jira/browse/HADOOP-18186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18186: Labels: pull-request-available (was: ) > s3a prefetching to use SemaphoredDelegatingExecutor for submitting work > --- > > Key: HADOOP-18186 > URL: https://issues.apache.org/jira/browse/HADOOP-18186 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Use SemaphoredDelegatingExecutor for each to stream to submit work, if > possible, for better fairness in processes with many streams. > this also takes a DurationTrackerFactory to count how long was spent in the > queue, something we would want to know -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4746: YARN-9708. Yarn Federation Router Support DelegationToken.
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953331074
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -558,4 +571,32 @@ protected void finalize() {
public Map getUserPipelineMap() {
return userPipelineMap;
}
+
+ /**
+ * Create RouterRMDelegationTokenSecretManager.
+ * In the YARN federation, the Router will replace the RM to
+ * manage the RMDelegationToken (generate, update, cancel),
+ * so the relevant configuration parameters still obtain the configuration
parameters of the RM.
+ *
+ * @param conf Configuration
+ * @return RouterDelegationTokenSecretManager.
+ */
+ protected RouterDelegationTokenSecretManager
createRouterRMDelegationTokenSecretManager(
+ Configuration conf) {
+
+long secretKeyInterval =
conf.getLong(YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_KEY,
+YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
+long tokenMaxLifetime =
conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY,
+YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
+long tokenRenewInterval =
conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
+YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
+
+return new RouterDelegationTokenSecretManager(secretKeyInterval,
+tokenMaxLifetime, tokenRenewInterval, 360);
Review Comment:
> 360 make it 10 hours: 10 * 60 * 60 * 1000
I double checked, it should be 1 hour, 60 * 60 * 1000
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4796: HADOOP-18186. s3a prefetching to use SemaphoredDelegatingExecutor for submitting work
hadoop-yetus commented on PR #4796:
URL: https://github.com/apache/hadoop/pull/4796#issuecomment-1225178955
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 41m 24s | | trunk passed |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 46s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 27s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 38s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 46s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 105m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4796/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4796 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 78f4d719d52a 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c94267a96e6c6ed152bf5265625dd2c32d8141ea |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4796/1/testReport/ |
| Max. process+thread count | 633 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4796/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4746: YARN-9708. Yarn Federation Router Support DelegationToken.
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953317155
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -558,4 +571,32 @@ protected void finalize() {
public Map getUserPipelineMap() {
return userPipelineMap;
}
+
+ /**
+ * Create RouterRMDelegationTokenSecretManager.
+ * In the YARN federation, the Router will replace the RM to
+ * manage the RMDelegationToken (generate, update, cancel),
+ * so the relevant configuration parameters still obtain the configuration
parameters of the RM.
+ *
+ * @param conf Configuration
+ * @return RouterDelegationTokenSecretManager.
+ */
+ protected RouterDelegationTokenSecretManager
createRouterRMDelegationTokenSecretManager(
+ Configuration conf) {
+
+long secretKeyInterval =
conf.getLong(YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_KEY,
+YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
+long tokenMaxLifetime =
conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY,
+YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
+long tokenRenewInterval =
conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
+YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
+
+return new RouterDelegationTokenSecretManager(secretKeyInterval,
+tokenMaxLifetime, tokenRenewInterval, 360);
Review Comment:
Thanks for your suggestion. I will fix it.
I think hardcoding 360 should not be a reasonable way, the parameters
should be extracted into the configuration file, because this part is
consistent with RM, it will use RM-related parameter names.
It is not very reasonable to extract this parameter in this pr into the
configuration file, I submitted a separate pr for extracting this parameter
into the configuration file.
JIRA: YARN-11253. Add Configuration to delegationToken RemoverScanInterval.
PR: https://github.com/apache/hadoop/pull/4751
Can you help review this pr? Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4746: YARN-9708. Yarn Federation Router Support DelegationToken.
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953317427
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/secure/TestRouterDelegationTokenSecretManager.java:
##
@@ -0,0 +1,198 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.secure;
+
+import org.apache.hadoop.io.Text;
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.test.LambdaTestUtils;
+import org.apache.hadoop.util.Time;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+import org.apache.hadoop.yarn.server.router.clientrm.RouterClientRMService;
+import
org.apache.hadoop.yarn.server.router.security.RouterDelegationTokenSecretManager;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.IOException;
+import java.util.Arrays;
+
+public class TestRouterDelegationTokenSecretManager extends
AbstractSecureRouterTest {
+
+ private static final Logger LOG =
+ LoggerFactory.getLogger(TestRouterDelegationTokenSecretManager.class);
+
+ @Test
+ public void testRouterStoreNewMasterKey() throws Exception {
+LOG.info("Test RouterDelegationTokenSecretManager: StoreNewMasterKey.");
+
+// Start the Router in Secure Mode
+startSecureRouter();
+
+// Store NewMasterKey
+RouterClientRMService routerClientRMService =
this.getRouter().getClientRMProxyService();
+RouterDelegationTokenSecretManager secretManager =
+routerClientRMService.getRouterDTSecretManager();
+DelegationKey storeKey = new DelegationKey(1234, 4321,
"keyBytes".getBytes());
+secretManager.storeNewMasterKey(storeKey);
+
+// Get DelegationKey
+DelegationKey paramKey = new DelegationKey(1234, 4321,
"keyBytes".getBytes());
+DelegationKey responseKey =
secretManager.getMasterKeyByDelegationKey(paramKey);
+
+Assert.assertNotNull(paramKey);
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4746: YARN-9708. Yarn Federation Router Support DelegationToken.
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953317155
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -558,4 +571,32 @@ protected void finalize() {
public Map getUserPipelineMap() {
return userPipelineMap;
}
+
+ /**
+ * Create RouterRMDelegationTokenSecretManager.
+ * In the YARN federation, the Router will replace the RM to
+ * manage the RMDelegationToken (generate, update, cancel),
+ * so the relevant configuration parameters still obtain the configuration
parameters of the RM.
+ *
+ * @param conf Configuration
+ * @return RouterDelegationTokenSecretManager.
+ */
+ protected RouterDelegationTokenSecretManager
createRouterRMDelegationTokenSecretManager(
+ Configuration conf) {
+
+long secretKeyInterval =
conf.getLong(YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_KEY,
+YarnConfiguration.RM_DELEGATION_KEY_UPDATE_INTERVAL_DEFAULT);
+long tokenMaxLifetime =
conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_KEY,
+YarnConfiguration.RM_DELEGATION_TOKEN_MAX_LIFETIME_DEFAULT);
+long tokenRenewInterval =
conf.getLong(YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_KEY,
+YarnConfiguration.RM_DELEGATION_TOKEN_RENEW_INTERVAL_DEFAULT);
+
+return new RouterDelegationTokenSecretManager(secretKeyInterval,
+tokenMaxLifetime, tokenRenewInterval, 360);
Review Comment:
Thanks for your suggestion.
I think hardcoding 360 should not be a reasonable way, the parameters
should be extracted into the configuration file, because this part is
consistent with RM, it will use RM-related parameter names.
It is not very reasonable to extract this parameter in this pr into the
configuration file, I submitted a separate pr for extracting this parameter
into the configuration file.
JIRA: YARN-11253. Add Configuration to delegationToken RemoverScanInterval.
PR: https://github.com/apache/hadoop/pull/4751
Can you help review this pr? Thank you very much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4746: YARN-9708. Yarn Federation Router Support DelegationToken.
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953295360
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/TestFederationStateStoreFacade.java:
##
@@ -232,4 +239,111 @@ public void testGetApplicationHomeSubClusterCache()
throws YarnException {
}
}
+ @Test
+ public void testStoreNewMasterKey() throws YarnException, IOException {
+// store delegation key;
+DelegationKey key = new DelegationKey(1234, 4321, "keyBytes".getBytes());
+HashSet keySet = new HashSet();
+keySet.add(key);
+facade.storeNewMasterKey(key);
+
+MemoryFederationStateStore federationStateStore =
+(MemoryFederationStateStore) facade.getStateStore();
+RouterRMDTSecretManagerState secretManagerState =
+federationStateStore.getRouterRMSecretManagerState();
+Assert.assertEquals(keySet, secretManagerState.getMasterKeyState());
+ }
+
+ @Test
+ public void testRemoveStoredMasterKey() throws YarnException, IOException {
+// store delegation key;
+DelegationKey key = new DelegationKey(4567, 7654, "keyBytes".getBytes());
+HashSet keySet = new HashSet();
+keySet.add(key);
+facade.storeNewMasterKey(key);
+
+// check to delete delegationKey
+facade.removeStoredMasterKey(key);
+keySet.clear();
+
+MemoryFederationStateStore federationStateStore =
+(MemoryFederationStateStore) facade.getStateStore();
+RouterRMDTSecretManagerState secretManagerState =
+federationStateStore.getRouterRMSecretManagerState();
+Assert.assertEquals(keySet, secretManagerState.getMasterKeyState());
+ }
+
+ @Test
+ public void testStoreNewToken() throws YarnException, IOException {
+// store new rm-token
+RMDelegationTokenIdentifier dtId1 = new RMDelegationTokenIdentifier(
+new Text("owner1"), new Text("renewer1"), new Text("realuser1"));
+int sequenceNumber = 1;
+dtId1.setSequenceNumber(sequenceNumber);
+Long renewDate1 = Time.now();
+facade.storeNewToken(dtId1, renewDate1);
+
+Map token1 =
+ new HashMap();
+token1.put(dtId1, renewDate1);
+
+MemoryFederationStateStore federationStateStore =
+(MemoryFederationStateStore) facade.getStateStore();
+RouterRMDTSecretManagerState storeSecretManagerState =
+federationStateStore.getRouterRMSecretManagerState();
+Assert.assertEquals(token1, storeSecretManagerState.getTokenState());
+ }
+
+ @Test
+ public void testUpdateNewToken() throws YarnException, IOException {
+// store new rm-token
+RMDelegationTokenIdentifier dtId1 = new RMDelegationTokenIdentifier(
+new Text("owner2"), new Text("renewer2"), new Text("realuser2"));
+int sequenceNumber = 2;
+dtId1.setSequenceNumber(sequenceNumber);
+Long renewDate1 = Time.now();
+facade.storeNewToken(dtId1, renewDate1);
+
+Map token1 =
+ new HashMap();
+token1.put(dtId1, renewDate1);
+
+renewDate1 = Time.now();
+facade.updateStoredToken(dtId1, renewDate1);
+token1.put(dtId1, renewDate1);
+
+MemoryFederationStateStore federationStateStore =
+(MemoryFederationStateStore) facade.getStateStore();
+RouterRMDTSecretManagerState updateSecretManagerState =
+federationStateStore.getRouterRMSecretManagerState();
+Assert.assertEquals(token1, updateSecretManagerState.getTokenState());
+Assert.assertEquals(sequenceNumber,
updateSecretManagerState.getDTSequenceNumber());
+ }
+
+ @Test
+ public void testRemoveStoredToken() throws YarnException, IOException {
+// store new rm-token
+RMDelegationTokenIdentifier dtId1 = new RMDelegationTokenIdentifier(
+new Text("owner3"), new Text("renewer3"), new Text("realuser3"));
+int sequenceNumber = 3;
+dtId1.setSequenceNumber(sequenceNumber);
+Long renewDate1 = Time.now();
+facade.storeNewToken(dtId1, renewDate1);
+
+Map token1 =
+ new HashMap();
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4746: YARN-9708. Yarn Federation Router Support DelegationToken.
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953295280
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/TestFederationStateStoreFacade.java:
##
@@ -232,4 +239,111 @@ public void testGetApplicationHomeSubClusterCache()
throws YarnException {
}
}
+ @Test
+ public void testStoreNewMasterKey() throws YarnException, IOException {
+// store delegation key;
+DelegationKey key = new DelegationKey(1234, 4321, "keyBytes".getBytes());
+HashSet keySet = new HashSet();
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4746: YARN-9708. Yarn Federation Router Support DelegationToken.
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953294793
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/test/java/org/apache/hadoop/yarn/server/federation/utils/TestFederationStateStoreFacade.java:
##
@@ -232,4 +239,111 @@ public void testGetApplicationHomeSubClusterCache()
throws YarnException {
}
}
+ @Test
+ public void testStoreNewMasterKey() throws YarnException, IOException {
+// store delegation key;
+DelegationKey key = new DelegationKey(1234, 4321, "keyBytes".getBytes());
+HashSet keySet = new HashSet();
+keySet.add(key);
+facade.storeNewMasterKey(key);
+
+MemoryFederationStateStore federationStateStore =
Review Comment:
This part of the code is written because only MemoryFederationStateStore is
currently implemented, but it is not very clean to write this way, I will
refactor this part of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4746: YARN-9708. Yarn Federation Router Support DelegationToken.
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953291856
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterRMDTSecretManagerState.java:
##
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+public class RouterRMDTSecretManagerState {
+
+ // DTIdentifier -> renewDate
+ private Map delegationTokenState =
+ new HashMap();
Review Comment:
I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4746: YARN-9708. Yarn Federation Router Support DelegationToken.
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953290103
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterMasterKey.java:
##
@@ -0,0 +1,128 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.yarn.util.Records;
+
+import java.nio.ByteBuffer;
+import java.util.Arrays;
+
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public abstract class RouterMasterKey {
+
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public static RouterMasterKey newInstance(Integer keyId, ByteBuffer
keyBytes, Long expiryDate) {
+RouterMasterKey policy = Records.newRecord(RouterMasterKey.class);
+policy.setKeyId(keyId);
+policy.setKeyBytes(keyBytes);
+policy.setExpiryDate(expiryDate);
+return policy;
+ }
+
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public static RouterMasterKey newInstance(RouterMasterKey masterKey) {
+RouterMasterKey routerMasterKey = Records.newRecord(RouterMasterKey.class);
+routerMasterKey.setKeyId(masterKey.getKeyId());
+routerMasterKey.setKeyBytes(masterKey.getKeyBytes());
+routerMasterKey.setExpiryDate(masterKey.getExpiryDate());
+return routerMasterKey;
+ }
+
+ /**
+ * Get the keyId of the MasterKey.
+ *
+ * @return MasterKeyId.
+ */
+ @InterfaceAudience.Public
+ @InterfaceStability.Unstable
+ public abstract Integer getKeyId();
+
+ /**
+ * Set the keyId of the MasterKey.
+ *
+ * @param keyId MasterKeyId.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract void setKeyId(Integer keyId);
+
+ /**
+ * Get the keyBytes of the DelegationKey.
+ *
+ * @return KeyBytes of the DelegationKey.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract ByteBuffer getKeyBytes();
+
+ /**
+ * Set the keyBytes of the DelegationKey.
+ *
+ * @param keyBytes KeyBytes of the DelegationKey.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract void setKeyBytes(ByteBuffer keyBytes);
+
+ /**
+ * Get the ExpiryDate of the DelegationKey.
+ *
+ * @return ExpiryDate of the DelegationKey.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract Long getExpiryDate();
+
+ /**
+ * Set the expiryDate of the DelegationKey.
+ *
+ * @param expiryDate expiryDate of the DelegationKey.
+ */
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public abstract void setExpiryDate(Long expiryDate);
+
+ @Override
+ public int hashCode() {
+final int prime = 31;
+int result = 1;
+result = prime * result + (int) (getExpiryDate() ^ (getExpiryDate() >>>
32));
+result = prime * result + Arrays.hashCode(getKeyBytes().array());
+result = prime * result + getKeyId();
+return result;
+ }
+
+ @Override
+ public boolean equals(Object right) {
Review Comment:
Thanks for your suggestion, I will refactor this part of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4746: YARN-9708. Yarn Federation Router Support DelegationToken.
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953289930
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/YARNDelegationTokenIdentifier.java:
##
@@ -112,4 +115,14 @@ public YARNDelegationTokenIdentifierProto getProto() {
setBuilderFields();
return builder.build();
}
+
+ @InterfaceAudience.Private
Review Comment:
I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #4369: HDFS-16601. DataTransfer should throw IOException to client
ZanderXu commented on PR #4369: URL: https://github.com/apache/hadoop/pull/4369#issuecomment-1225131216 @jojochuang Master, I have rebased this patch based on the latest trunk. I'm looking forward to getting some of your thoughts on this issue. BTW, Remote copy of HDFS-2139(FastCopy) will use DataTransfer and it also requires DataTransfer to throw IOException to Client to retry. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4678: YARN-11240. Fix incorrect placeholder in yarn-module.
slfan1989 commented on PR #4678: URL: https://github.com/apache/hadoop/pull/4678#issuecomment-1225113830 @ayushtkn Can you help review the code again? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4746: YARN-9708. Yarn Federation Router Support DelegationToken.
slfan1989 commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953266280
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/impl/MemoryFederationStateStore.java:
##
@@ -370,6 +385,142 @@ public GetReservationsHomeSubClusterResponse
getReservationsHomeSubCluster(
return GetReservationsHomeSubClusterResponse.newInstance(result);
}
+ @Override
+ public RouterMasterKeyResponse storeNewMasterKey(RouterMasterKeyRequest
request)
+ throws YarnException, IOException {
+
+// Restore the DelegationKey from the request
+RouterMasterKey masterKey = request.getRouterMasterKey();
+ByteBuffer keyByteBuf = masterKey.getKeyBytes();
Review Comment:
Thanks for your suggestion, I will refactor this part of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4734: HDFS-16728. RBF throw IndexOutOfBoundsException with disableNameServices
hadoop-yetus commented on PR #4734:
URL: https://github.com/apache/hadoop/pull/4734#issuecomment-1225103180
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 10s | | trunk passed |
| +1 :green_heart: | compile | 1m 18s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 49s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 53s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 22s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 40s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 38s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 40m 18s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 147m 1s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4734/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4734 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux bf6d75d84fc6 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 06c83a49fb9a22dbef0513356809ff7d95347dd4 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4734/4/testReport/ |
| Max. process+thread count | 2286 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4734/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4764: YARN-11177. Support getNewReservation, submit / update/ Reservation API's for Federation.
slfan1989 commented on code in PR #4764:
URL: https://github.com/apache/hadoop/pull/4764#discussion_r953258995
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -888,13 +890,120 @@ public MoveApplicationAcrossQueuesResponse
moveApplicationAcrossQueues(
@Override
public GetNewReservationResponse getNewReservation(
GetNewReservationRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null) {
+ routerMetrics.incrGetNewReservationFailedRetrieved();
+ String errMsg = "Missing getNewReservation request.";
+ RouterServerUtil.logAndThrowException(errMsg, null);
+}
+
+long startTime = clock.getTime();
+Map subClustersActive =
+federationFacade.getSubClusters(true);
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+ SubClusterId subClusterId = getRandomActiveSubCluster(subClustersActive);
+ LOG.info("getNewReservation try #{} on SubCluster {}.", i, subClusterId);
+ ApplicationClientProtocol clientRMProxy =
getClientRMProxyForSubCluster(subClusterId);
+ GetNewReservationResponse response = null;
+ try {
+response = clientRMProxy.getNewReservation(request);
+ } catch (Exception e) {
+LOG.warn("Unable to create a new Reservation in SubCluster {}.",
subClusterId.getId(), e);
+subClustersActive.remove(subClusterId);
+ }
+
+ if (response != null) {
+long stopTime = clock.getTime();
+routerMetrics.succeededGetNewReservationRetrieved(stopTime -
startTime);
+return response;
+ }
+}
+
+routerMetrics.incrGetNewReservationFailedRetrieved();
+String errMsg = "Failed to create a new reservation.";
+throw new YarnException(errMsg);
}
@Override
public ReservationSubmissionResponse submitReservation(
ReservationSubmissionRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null || request.getReservationId() == null
+|| request.getReservationDefinition() == null ||
request.getQueue() == null) {
+ routerMetrics.incrSubmitReservationFailedRetrieved();
+ RouterServerUtil.logAndThrowException(
+ "Missing submitReservation request or reservationId " +
+ "or reservation definition or queue.", null);
+}
+
+ReservationSubmissionResponse response = null;
+long startTime = clock.getTime();
+ReservationId reservationId = request.getReservationId();
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+
+ SubClusterId subClusterId =
policyFacade.getReservationHomeSubCluster(request);
+
+ LOG.info("submitReservation reservationId {} try #{} on SubCluster {}.",
+ reservationId, i, subClusterId);
+
+ ReservationHomeSubCluster reservationHomeSubCluster =
+ ReservationHomeSubCluster.newInstance(reservationId, subClusterId);
+
+ if (i == 0) {
Review Comment:
I tried to refactor this part of the code, please help to review the code
again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4764: YARN-11177. Support getNewReservation, submit / update/ Reservation API's for Federation.
slfan1989 commented on code in PR #4764:
URL: https://github.com/apache/hadoop/pull/4764#discussion_r953253625
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -888,13 +890,120 @@ public MoveApplicationAcrossQueuesResponse
moveApplicationAcrossQueues(
@Override
public GetNewReservationResponse getNewReservation(
GetNewReservationRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null) {
+ routerMetrics.incrGetNewReservationFailedRetrieved();
+ String errMsg = "Missing getNewReservation request.";
+ RouterServerUtil.logAndThrowException(errMsg, null);
+}
+
+long startTime = clock.getTime();
+Map subClustersActive =
+federationFacade.getSubClusters(true);
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+ SubClusterId subClusterId = getRandomActiveSubCluster(subClustersActive);
+ LOG.info("getNewReservation try #{} on SubCluster {}.", i, subClusterId);
+ ApplicationClientProtocol clientRMProxy =
getClientRMProxyForSubCluster(subClusterId);
+ GetNewReservationResponse response = null;
+ try {
+response = clientRMProxy.getNewReservation(request);
+ } catch (Exception e) {
+LOG.warn("Unable to create a new Reservation in SubCluster {}.",
subClusterId.getId(), e);
+subClustersActive.remove(subClusterId);
+ }
+
+ if (response != null) {
+long stopTime = clock.getTime();
+routerMetrics.succeededGetNewReservationRetrieved(stopTime -
startTime);
+return response;
+ }
+}
+
+routerMetrics.incrGetNewReservationFailedRetrieved();
+String errMsg = "Failed to create a new reservation.";
+throw new YarnException(errMsg);
}
@Override
public ReservationSubmissionResponse submitReservation(
ReservationSubmissionRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null || request.getReservationId() == null
+|| request.getReservationDefinition() == null ||
request.getQueue() == null) {
+ routerMetrics.incrSubmitReservationFailedRetrieved();
+ RouterServerUtil.logAndThrowException(
+ "Missing submitReservation request or reservationId " +
+ "or reservation definition or queue.", null);
+}
+
+ReservationSubmissionResponse response = null;
Review Comment:
I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jianghuazhu commented on a diff in pull request #4761: HDFS-16733.Improve INode#isRoot().
jianghuazhu commented on code in PR #4761:
URL: https://github.com/apache/hadoop/pull/4761#discussion_r953248876
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java:
##
@@ -73,7 +73,8 @@ public abstract class INode implements INodeAttributes,
Diff.Element {
* Check whether this is the root inode.
*/
final boolean isRoot() {
-return getLocalNameBytes().length == 0;
+// Note: There is no restriction that id must be equal to
INodeId#ROOT_INODE_ID
+return getLocalNameBytes() == null || getLocalNameBytes().length == 0;
Review Comment:
Thanks @Hexiaoqiao .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4764: YARN-11177. Support getNewReservation, submit / update/ Reservation API's for Federation.
slfan1989 commented on code in PR #4764:
URL: https://github.com/apache/hadoop/pull/4764#discussion_r953247032
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -888,13 +890,120 @@ public MoveApplicationAcrossQueuesResponse
moveApplicationAcrossQueues(
@Override
public GetNewReservationResponse getNewReservation(
GetNewReservationRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null) {
+ routerMetrics.incrGetNewReservationFailedRetrieved();
+ String errMsg = "Missing getNewReservation request.";
+ RouterServerUtil.logAndThrowException(errMsg, null);
+}
+
+long startTime = clock.getTime();
+Map subClustersActive =
+federationFacade.getSubClusters(true);
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+ SubClusterId subClusterId = getRandomActiveSubCluster(subClustersActive);
+ LOG.info("getNewReservation try #{} on SubCluster {}.", i, subClusterId);
+ ApplicationClientProtocol clientRMProxy =
getClientRMProxyForSubCluster(subClusterId);
+ GetNewReservationResponse response = null;
+ try {
+response = clientRMProxy.getNewReservation(request);
+ } catch (Exception e) {
+LOG.warn("Unable to create a new Reservation in SubCluster {}.",
subClusterId.getId(), e);
+subClustersActive.remove(subClusterId);
+ }
+
+ if (response != null) {
+long stopTime = clock.getTime();
+routerMetrics.succeededGetNewReservationRetrieved(stopTime -
startTime);
+return response;
+ }
+}
+
+routerMetrics.incrGetNewReservationFailedRetrieved();
+String errMsg = "Failed to create a new reservation.";
+throw new YarnException(errMsg);
}
@Override
public ReservationSubmissionResponse submitReservation(
ReservationSubmissionRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null || request.getReservationId() == null
+|| request.getReservationDefinition() == null ||
request.getQueue() == null) {
+ routerMetrics.incrSubmitReservationFailedRetrieved();
+ RouterServerUtil.logAndThrowException(
+ "Missing submitReservation request or reservationId " +
+ "or reservation definition or queue.", null);
+}
+
+ReservationSubmissionResponse response = null;
+long startTime = clock.getTime();
+ReservationId reservationId = request.getReservationId();
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+
+ SubClusterId subClusterId =
policyFacade.getReservationHomeSubCluster(request);
+
+ LOG.info("submitReservation reservationId {} try #{} on SubCluster {}.",
+ reservationId, i, subClusterId);
+
+ ReservationHomeSubCluster reservationHomeSubCluster =
+ ReservationHomeSubCluster.newInstance(reservationId, subClusterId);
+
+ if (i == 0) {
+try {
+ // persist the mapping of reservationId and the subClusterId which
has
+ // been selected as its home
+ subClusterId =
federationFacade.addReservationHomeSubCluster(reservationHomeSubCluster);
+} catch (YarnException e) {
+ routerMetrics.incrSubmitReservationFailedRetrieved();
+ String message =
+ String.format("Unable to insert the ReservationId %s into the
FederationStateStore.",
+ reservationId);
+ RouterServerUtil.logAndThrowException(message, e);
+}
+ } else {
+try {
+ // update the mapping of reservationId and the home subClusterId to
+ // the new subClusterId we have selected
+
federationFacade.updateReservationHomeSubCluster(reservationHomeSubCluster);
+} catch (YarnException e) {
+ String message =
+ String.format("Unable to update the ReservationId %s into the
FederationStateStore.",
Review Comment:
I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4764: YARN-11177. Support getNewReservation, submit / update/ Reservation API's for Federation.
slfan1989 commented on code in PR #4764:
URL: https://github.com/apache/hadoop/pull/4764#discussion_r953245747
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -888,13 +890,120 @@ public MoveApplicationAcrossQueuesResponse
moveApplicationAcrossQueues(
@Override
public GetNewReservationResponse getNewReservation(
GetNewReservationRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null) {
+ routerMetrics.incrGetNewReservationFailedRetrieved();
+ String errMsg = "Missing getNewReservation request.";
+ RouterServerUtil.logAndThrowException(errMsg, null);
+}
+
+long startTime = clock.getTime();
+Map subClustersActive =
+federationFacade.getSubClusters(true);
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+ SubClusterId subClusterId = getRandomActiveSubCluster(subClustersActive);
+ LOG.info("getNewReservation try #{} on SubCluster {}.", i, subClusterId);
+ ApplicationClientProtocol clientRMProxy =
getClientRMProxyForSubCluster(subClusterId);
+ GetNewReservationResponse response = null;
+ try {
+response = clientRMProxy.getNewReservation(request);
+ } catch (Exception e) {
+LOG.warn("Unable to create a new Reservation in SubCluster {}.",
subClusterId.getId(), e);
+subClustersActive.remove(subClusterId);
+ }
+
+ if (response != null) {
+long stopTime = clock.getTime();
Review Comment:
Thanks for your suggestion, I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4792: YARN-11275. [Federation] Add batchFinishApplicationMaster in UAMPoolManager.
slfan1989 commented on code in PR #4792:
URL: https://github.com/apache/hadoop/pull/4792#discussion_r953243985
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/uam/UnmanagedAMPoolManager.java:
##
@@ -450,4 +452,44 @@ public void drainUAMHeartbeats() {
uam.drainHeartbeatThread();
}
}
+
+ public Map
batchFinishApplicationMaster(
+ FinishApplicationMasterRequest request, String appId) {
+
+Map responseMap = new HashMap<>();
+Set subClusterIds = this.unmanagedAppMasterMap.keySet();
+
+if (subClusterIds.size() > 0) {
Review Comment:
Thank you very much for helping to review the code, I will modify the code.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestFederationInterceptor.java:
##
@@ -969,4 +969,58 @@ private PreemptionMessage createDummyPreemptionMessage(
preemptionMessage.setContract(contract);
return preemptionMessage;
}
+
+ @Test
+ public void testBatchFinishApplicationMaster() throws IOException,
InterruptedException {
+
+final RegisterApplicationMasterRequest registerReq =
+Records.newRecord(RegisterApplicationMasterRequest.class);
+registerReq.setHost(Integer.toString(testAppId));
+registerReq.setRpcPort(testAppId);
+registerReq.setTrackingUrl("");
+
+UserGroupInformation ugi =
interceptor.getUGIWithToken(interceptor.getAttemptId());
+
+ugi.doAs((PrivilegedExceptionAction) () -> {
+
+ // Register the application
+ RegisterApplicationMasterRequest registerReq1 =
+ Records.newRecord(RegisterApplicationMasterRequest.class);
+ registerReq1.setHost(Integer.toString(testAppId));
+ registerReq1.setRpcPort(0);
+ registerReq1.setTrackingUrl("");
+
+ // Register ApplicationMaster
+ RegisterApplicationMasterResponse registerResponse =
+ interceptor.registerApplicationMaster(registerReq1);
+ Assert.assertNotNull(registerResponse);
+ lastResponseId = 0;
+
+ Assert.assertEquals(0, interceptor.getUnmanagedAMPoolSize());
+
+ // Allocate the first batch of containers, with sc1 and sc2 active
+ registerSubCluster(SubClusterId.newInstance("SC-1"));
+ registerSubCluster(SubClusterId.newInstance("SC-2"));
+
+ int numberOfContainers = 3;
+ List containers =
+ getContainersAndAssert(numberOfContainers, numberOfContainers * 2);
+ Assert.assertEquals(2, interceptor.getUnmanagedAMPoolSize());
+ Assert.assertEquals(numberOfContainers, containers.size());
+
+ // Finish the application
+ FinishApplicationMasterRequest finishReq =
+ Records.newRecord(FinishApplicationMasterRequest.class);
+ finishReq.setDiagnostics("");
+ finishReq.setTrackingUrl("");
+ finishReq.setFinalApplicationStatus(FinalApplicationStatus.SUCCEEDED);
+
+ FinishApplicationMasterResponse finshResponse =
+ interceptor.finishApplicationMaster(finishReq);
+ Assert.assertNotNull(finshResponse);
+ Assert.assertEquals(true, finshResponse.getIsUnregistered());
Review Comment:
I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snmvaughan commented on pull request #4786: HDFS-16684. Exclude the current JournalNode
snmvaughan commented on PR #4786: URL: https://github.com/apache/hadoop/pull/4786#issuecomment-1225057314 I've run all of the failed test classes locally without an issue. I've opened a ticket specifically for this problem. https://issues.apache.org/jira/browse/HDFS-16740 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18406) Adds alignment context to call path for creating RPC proxy with multiple connections per user.
[
https://issues.apache.org/jira/browse/HADOOP-18406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583928#comment-17583928
]
ASF GitHub Bot commented on HADOOP-18406:
-
hadoop-yetus commented on PR #4748:
URL: https://github.com/apache/hadoop/pull/4748#issuecomment-1225048110
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 24s | | trunk passed |
| +1 :green_heart: | compile | 23m 10s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 44s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 9s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 46s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 6s | | the patch passed |
| +1 :green_heart: | compile | 22m 25s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 25s | | the patch passed |
| +1 :green_heart: | compile | 20m 39s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 43s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 8s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 37s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 52s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 33s | | The patch does not
generate ASF License warnings. |
| | | 216m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4748/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4748 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 224c91da7a97 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 330df2d2de00d6a56648d0e184881ad7cbbb634d |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4748/3/testReport/ |
| Max. process+thread count | 3149 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4748/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
[GitHub] [hadoop] hadoop-yetus commented on pull request #4748: HADOOP-18406: Adds alignment context to call path for creating RPC proxy with multiple connections per user.
hadoop-yetus commented on PR #4748:
URL: https://github.com/apache/hadoop/pull/4748#issuecomment-1225048110
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 34s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 24s | | trunk passed |
| +1 :green_heart: | compile | 23m 10s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 44s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 46s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 9s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 46s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 13s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 45s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 6s | | the patch passed |
| +1 :green_heart: | compile | 22m 25s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 25s | | the patch passed |
| +1 :green_heart: | compile | 20m 39s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 43s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 8s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 37s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 24s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 8s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 52s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 33s | | The patch does not
generate ASF License warnings. |
| | | 216m 43s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4748/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4748 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 224c91da7a97 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 330df2d2de00d6a56648d0e184881ad7cbbb634d |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4748/3/testReport/ |
| Max. process+thread count | 3149 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4748/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For
[GitHub] [hadoop] hadoop-yetus commented on pull request #4794: HDFS-16686. GetJournalEditServlet fails to authorize valid Kerberos request (#4724)
hadoop-yetus commented on PR #4794:
URL: https://github.com/apache/hadoop/pull/4794#issuecomment-1225032086
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 10m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 18s | | branch-3.3 passed |
| +1 :green_heart: | compile | 1m 27s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 1m 4s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 1m 35s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 1m 45s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 3m 36s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 28m 51s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 47s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4794/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 4 new + 17 unchanged -
16 fixed = 21 total (was 33) |
| +1 :green_heart: | mvnsite | 1m 25s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed |
| +1 :green_heart: | spotbugs | 3m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 50s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 218m 56s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4794/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 59s | | The patch does not
generate ASF License warnings. |
| | | 343m 13s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestReconstructStripedFileWithValidator |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFSStriped |
| | hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport |
| | hadoop.hdfs.server.balancer.TestBalancer |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4794/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4794 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b18814100895 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / b9732a282a06f94e7007b7b783cb7015fbe1f838 |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~18.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4794/1/testReport/ |
| Max. process+thread count | 2246 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4794/1/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail:
[jira] [Commented] (HADOOP-18333) hadoop-client-runtime impact by CVE-2022-2047 CVE-2022-2048 due to shaded jetty
[ https://issues.apache.org/jira/browse/HADOOP-18333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583913#comment-17583913 ] ASF GitHub Bot commented on HADOOP-18333: - jojochuang commented on PR #4600: URL: https://github.com/apache/hadoop/pull/4600#issuecomment-1225015487 Thanks for verifying the test for me. Merged. > hadoop-client-runtime impact by CVE-2022-2047 CVE-2022-2048 due to shaded > jetty > --- > > Key: HADOOP-18333 > URL: https://issues.apache.org/jira/browse/HADOOP-18333 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.3.3 >Reporter: phoebe chen >Assignee: groot >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > CVE-2022-2047 and CVE-2022-2048 is recently found for Eclipse Jetty, and > impacts 9.4.0 thru 9.4.46. > In latest 3.3.3 of hadoop-client-runtime, it shaded 9.4.43.v20210629 version > jetty which is impacted. > In Trunk, Jetty is in version 9.4.44.v20210927, which is still impacted. > Need to upgrade Jetty Version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18333) hadoop-client-runtime impact by CVE-2022-2047 CVE-2022-2048 due to shaded jetty
[ https://issues.apache.org/jira/browse/HADOOP-18333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang resolved HADOOP-18333. -- Fix Version/s: 3.4.0 3.3.9 Resolution: Fixed > hadoop-client-runtime impact by CVE-2022-2047 CVE-2022-2048 due to shaded > jetty > --- > > Key: HADOOP-18333 > URL: https://issues.apache.org/jira/browse/HADOOP-18333 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.3.3 >Reporter: phoebe chen >Assignee: groot >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > Time Spent: 1h 50m > Remaining Estimate: 0h > > CVE-2022-2047 and CVE-2022-2048 is recently found for Eclipse Jetty, and > impacts 9.4.0 thru 9.4.46. > In latest 3.3.3 of hadoop-client-runtime, it shaded 9.4.43.v20210629 version > jetty which is impacted. > In Trunk, Jetty is in version 9.4.44.v20210927, which is still impacted. > Need to upgrade Jetty Version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang commented on pull request #4600: HADOOP-18333. Upgrade jetty version to 9.4.48.v20220622
jojochuang commented on PR #4600: URL: https://github.com/apache/hadoop/pull/4600#issuecomment-1225015487 Thanks for verifying the test for me. Merged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18333) hadoop-client-runtime impact by CVE-2022-2047 CVE-2022-2048 due to shaded jetty
[ https://issues.apache.org/jira/browse/HADOOP-18333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583912#comment-17583912 ] ASF GitHub Bot commented on HADOOP-18333: - jojochuang merged PR #4600: URL: https://github.com/apache/hadoop/pull/4600 > hadoop-client-runtime impact by CVE-2022-2047 CVE-2022-2048 due to shaded > jetty > --- > > Key: HADOOP-18333 > URL: https://issues.apache.org/jira/browse/HADOOP-18333 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.3.3 >Reporter: phoebe chen >Assignee: groot >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > CVE-2022-2047 and CVE-2022-2048 is recently found for Eclipse Jetty, and > impacts 9.4.0 thru 9.4.46. > In latest 3.3.3 of hadoop-client-runtime, it shaded 9.4.43.v20210629 version > jetty which is impacted. > In Trunk, Jetty is in version 9.4.44.v20210927, which is still impacted. > Need to upgrade Jetty Version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang merged pull request #4600: HADOOP-18333. Upgrade jetty version to 9.4.48.v20220622
jojochuang merged PR #4600: URL: https://github.com/apache/hadoop/pull/4600 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #4784: HDFS-16737. Fix number of threads in FsDatasetAsyncDiskService#addExecutorForVolume
ZanderXu commented on PR #4784: URL: https://github.com/apache/hadoop/pull/4784#issuecomment-1225011225 @ayushtkn Sir, can you help me review this patch? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #4734: HDFS-16728. RBF throw IndexOutOfBoundsException with disableNameServices
ZanderXu commented on PR #4734: URL: https://github.com/apache/hadoop/pull/4734#issuecomment-1225010431 @goiri @Hexiaoqiao I have rebased this patch based on the latest trunk. Please help me review and merge it into trunk. Then I will rebase [HDFS-16734](https://issues.apache.org/jira/browse/HDFS-16734) on the latest trunk. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18406) Adds alignment context to call path for creating RPC proxy with multiple connections per user.
[ https://issues.apache.org/jira/browse/HADOOP-18406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583907#comment-17583907 ] ASF GitHub Bot commented on HADOOP-18406: - omalley commented on PR #4748: URL: https://github.com/apache/hadoop/pull/4748#issuecomment-1225006293 +1 LGTM. > Adds alignment context to call path for creating RPC proxy with multiple > connections per user. > -- > > Key: HADOOP-18406 > URL: https://issues.apache.org/jira/browse/HADOOP-18406 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > Labels: pull-request-available > > HDFS-13274 (RBF: Extend RouterRpcClient to use multiple sockets) gets the RPC > proxy using methods which do not allow using an alignment context. These > methods were added in HADOOP-13144 (Enhancing IPC client throughput via > multiple connections per user). > This change adds an alignment context as an argument for methods in the call > path for creating the proxy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] omalley commented on pull request #4748: HADOOP-18406: Adds alignment context to call path for creating RPC proxy with multiple connections per user.
omalley commented on PR #4748: URL: https://github.com/apache/hadoop/pull/4748#issuecomment-1225006293 +1 LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18406) Adds alignment context to call path for creating RPC proxy with multiple connections per user.
[ https://issues.apache.org/jira/browse/HADOOP-18406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583906#comment-17583906 ] ASF GitHub Bot commented on HADOOP-18406: - omalley closed pull request #4748: HADOOP-18406: Adds alignment context to call path for creating RPC proxy with multiple connections per user. URL: https://github.com/apache/hadoop/pull/4748 > Adds alignment context to call path for creating RPC proxy with multiple > connections per user. > -- > > Key: HADOOP-18406 > URL: https://issues.apache.org/jira/browse/HADOOP-18406 > Project: Hadoop Common > Issue Type: Improvement > Components: ipc >Reporter: Simbarashe Dzinamarira >Assignee: Simbarashe Dzinamarira >Priority: Major > Labels: pull-request-available > > HDFS-13274 (RBF: Extend RouterRpcClient to use multiple sockets) gets the RPC > proxy using methods which do not allow using an alignment context. These > methods were added in HADOOP-13144 (Enhancing IPC client throughput via > multiple connections per user). > This change adds an alignment context as an argument for methods in the call > path for creating the proxy. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] omalley closed pull request #4748: HADOOP-18406: Adds alignment context to call path for creating RPC proxy with multiple connections per user.
omalley closed pull request #4748: HADOOP-18406: Adds alignment context to call path for creating RPC proxy with multiple connections per user. URL: https://github.com/apache/hadoop/pull/4748 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18410) S3AInputStream.unbuffer() async drain not releasing http connections
[ https://issues.apache.org/jira/browse/HADOOP-18410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583890#comment-17583890 ] ASF GitHub Bot commented on HADOOP-18410: - virajjasani commented on PR #4766: URL: https://github.com/apache/hadoop/pull/4766#issuecomment-1224965093 > a way to do some retirement of long-lived input streams. Hmm might need some sort of "self expiring cache" kind of thing > S3AInputStream.unbuffer() async drain not releasing http connections > > > Key: HADOOP-18410 > URL: https://issues.apache.org/jira/browse/HADOOP-18410 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.9 >Reporter: Steve Loughran >Assignee: Steve Loughran >Priority: Blocker > Labels: pull-request-available > > Impala tcp-ds setup to s3 is hitting problems with timeout fetching http > connections from the s3a fs pool. Disabling s3a async drain makes this > problem *go away*. assumption, either those async ops are blocking, or they > are not releasing references properly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #4766: HADOOP-18410. S3AInputStream.unbuffer() not releasing http connections
virajjasani commented on PR #4766: URL: https://github.com/apache/hadoop/pull/4766#issuecomment-1224965093 > a way to do some retirement of long-lived input streams. Hmm might need some sort of "self expiring cache" kind of thing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4793: YARN-11276. Add lru cache for RMWebServices.getApps
hadoop-yetus commented on PR #4793:
URL: https://github.com/apache/hadoop/pull/4793#issuecomment-1224924118
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 40s | | trunk passed |
| +1 :green_heart: | compile | 10m 0s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 8m 59s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 50s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 32s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 16s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 34s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 32s | | the patch passed |
| +1 :green_heart: | compile | 9m 9s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 9m 9s | | the patch passed |
| +1 :green_heart: | compile | 8m 45s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 8m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 2m 4s | | the patch passed |
| +1 :green_heart: | mvnsite | 4m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 42s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 3m 31s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 30s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 21s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 36s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 21s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 98m 51s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 1m 19s | | The patch does not
generate ASF License warnings. |
| | | 282m 14s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4793 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 0c9cd5142c56 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c84fa9842cf15cb599a3e5727bb937ed61c39bd7 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4793/4/testReport/ |
| Max. process+thread count | 979 (vs. ulimit of 5500) |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common
[GitHub] [hadoop] hadoop-yetus commented on pull request #4786: HDFS-16684. Exclude the current JournalNode
hadoop-yetus commented on PR #4786:
URL: https://github.com/apache/hadoop/pull/4786#issuecomment-1224923563
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 4s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ branch-3.3 Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 37s | | branch-3.3 passed |
| +1 :green_heart: | compile | 1m 29s | | branch-3.3 passed |
| +1 :green_heart: | checkstyle | 1m 5s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 1m 37s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 1m 44s | | branch-3.3 passed |
| +1 :green_heart: | spotbugs | 3m 40s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 28m 46s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 22s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 47s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed |
| +1 :green_heart: | spotbugs | 3m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 28m 6s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 217m 17s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4786/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 1m 3s | | The patch does not
generate ASF License warnings. |
| | | 332m 12s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade |
| | hadoop.hdfs.server.namenode.TestNameNodeMXBean |
| | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4786/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4786 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d85ed4270070 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | branch-3.3 / f332c1efed5d0603506285d25a2bfb473eeeb0af |
| Default Java | Private Build-1.8.0_342-8u342-b07-0ubuntu1~18.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4786/4/testReport/ |
| Max. process+thread count | 2291 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4786/4/console |
| versions | git=2.17.1 maven=3.6.0 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4746: YARN-9708. Yarn Federation Router Support DelegationToken.
goiri commented on code in PR #4746:
URL: https://github.com/apache/hadoop/pull/4746#discussion_r953096007
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/client/YARNDelegationTokenIdentifier.java:
##
@@ -112,4 +115,14 @@ public YARNDelegationTokenIdentifierProto getProto() {
setBuilderFields();
return builder.build();
}
+
+ @InterfaceAudience.Private
Review Comment:
Can we import the InterfaceAudience.Private as others?
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterRMDTSecretManagerState.java:
##
@@ -0,0 +1,53 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.hadoop.security.token.delegation.DelegationKey;
+import org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+public class RouterRMDTSecretManagerState {
+
+ // DTIdentifier -> renewDate
+ private Map delegationTokenState =
+ new HashMap();
Review Comment:
`new HashMap<>();` and single line
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/federation/store/records/RouterRMTokenRequest.java:
##
@@ -0,0 +1,43 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.federation.store.records;
+
+import org.apache.hadoop.classification.InterfaceAudience;
+import org.apache.hadoop.classification.InterfaceStability;
+import org.apache.hadoop.yarn.util.Records;
+
+@InterfaceAudience.Private
+@InterfaceStability.Unstable
+public abstract class RouterRMTokenRequest {
+
+ @InterfaceAudience.Private
+ @InterfaceStability.Unstable
+ public static RouterRMTokenRequest newInstance(RouterStoreToken
routerStoreToken) {
+RouterRMTokenRequest request =
Records.newRecord(RouterRMTokenRequest.class);
+request.setRouterStoreToken(routerStoreToken);
+return request;
+ }
+
+ @InterfaceAudience.Public
+ @InterfaceStability.Unstable
+ public abstract RouterStoreToken getRouterStoreToken();
+
+ @InterfaceAudience.Private
Review Comment:
Import statically this.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/secure/TestRouterDelegationTokenSecretManager.java:
##
@@ -0,0 +1,198 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the
[jira] [Commented] (HADOOP-18417) Upgrade Maven Surefire plugin
[
https://issues.apache.org/jira/browse/HADOOP-18417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583839#comment-17583839
]
ASF GitHub Bot commented on HADOOP-18417:
-
hadoop-yetus commented on PR #4795:
URL: https://github.com/apache/hadoop/pull/4795#issuecomment-1224832973
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 1s | | trunk passed |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 63m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | shadedclient | 3m 30s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 20s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 71m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4795/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux c58eeba1aab5 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a75bbb35b6e130f3166b1ae471ee082cc3bc8101 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4795/1/testReport/ |
| Max. process+thread count | 693 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4795/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Upgrade Maven Surefire plugin
> -
>
> Key: HADOOP-18417
> URL:
[GitHub] [hadoop] hadoop-yetus commented on pull request #4795: HADOOP-18417. Upgrade to M7 of surefire plugin
hadoop-yetus commented on PR #4795:
URL: https://github.com/apache/hadoop/pull/4795#issuecomment-1224832973
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 1s | | trunk passed |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 0m 37s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 63m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | shadedclient | 3m 30s | | patch has errors when building
and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 20s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 38s | | The patch does not
generate ASF License warnings. |
| | | 71m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4795/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux c58eeba1aab5 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a75bbb35b6e130f3166b1ae471ee082cc3bc8101 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4795/1/testReport/ |
| Max. process+thread count | 693 (vs. ulimit of 5500) |
| modules | C: hadoop-project U: hadoop-project |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4795/1/console |
| versions | git=2.25.1 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe,
[GitHub] [hadoop] goiri merged pull request #4719: HDFS-16724. RBF should support get the information about ancestor mount points
goiri merged PR #4719: URL: https://github.com/apache/hadoop/pull/4719 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18345) Enhance client protocol to propagate last seen state IDs for multiple nameservices.
[
https://issues.apache.org/jira/browse/HADOOP-18345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583827#comment-17583827
]
ASF GitHub Bot commented on HADOOP-18345:
-
goiri commented on PR #4584:
URL: https://github.com/apache/hadoop/pull/4584#issuecomment-1224788447
No worries, you merged manually instead of merging the existing PR.
It is fine.
> Enhance client protocol to propagate last seen state IDs for multiple
> nameservices.
> ---
>
> Key: HADOOP-18345
> URL: https://issues.apache.org/jira/browse/HADOOP-18345
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The RPCHeader in the client protocol currently contains a single value to
> indicate the last seen state ID for a namenode.
> {noformat}
> optional int64 stateId = 8; // The last seen Global State ID
> {noformat}
> When there are multiple namenodes, such as in router based federation, the
> headers need to carry the state IDs for each of these nameservices that are
> part of the federation.
> This change is a prerequisite for HDFS-13522: RBF: Support observer node from
> Router-Based Federation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on pull request #4584: HADOOP-18345: Enhance client protocol to propagate last seen state IDs for multiple nameservices.
goiri commented on PR #4584: URL: https://github.com/apache/hadoop/pull/4584#issuecomment-1224788447 No worries, you merged manually instead of merging the existing PR. It is fine. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18345) Enhance client protocol to propagate last seen state IDs for multiple nameservices.
[
https://issues.apache.org/jira/browse/HADOOP-18345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Owen O'Malley resolved HADOOP-18345.
Fix Version/s: 3.4.0
3.3.9
Resolution: Fixed
> Enhance client protocol to propagate last seen state IDs for multiple
> nameservices.
> ---
>
> Key: HADOOP-18345
> URL: https://issues.apache.org/jira/browse/HADOOP-18345
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.9
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The RPCHeader in the client protocol currently contains a single value to
> indicate the last seen state ID for a namenode.
> {noformat}
> optional int64 stateId = 8; // The last seen Global State ID
> {noformat}
> When there are multiple namenodes, such as in router based federation, the
> headers need to carry the state IDs for each of these nameservices that are
> part of the federation.
> This change is a prerequisite for HDFS-13522: RBF: Support observer node from
> Router-Based Federation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18345) Enhance client protocol to propagate last seen state IDs for multiple nameservices.
[
https://issues.apache.org/jira/browse/HADOOP-18345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583819#comment-17583819
]
ASF GitHub Bot commented on HADOOP-18345:
-
omalley commented on PR #4584:
URL: https://github.com/apache/hadoop/pull/4584#issuecomment-1224730686
@goiri No, I committed the change to trunk and branch-3.3.
> Enhance client protocol to propagate last seen state IDs for multiple
> nameservices.
> ---
>
> Key: HADOOP-18345
> URL: https://issues.apache.org/jira/browse/HADOOP-18345
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The RPCHeader in the client protocol currently contains a single value to
> indicate the last seen state ID for a namenode.
> {noformat}
> optional int64 stateId = 8; // The last seen Global State ID
> {noformat}
> When there are multiple namenodes, such as in router based federation, the
> headers need to carry the state IDs for each of these nameservices that are
> part of the federation.
> This change is a prerequisite for HDFS-13522: RBF: Support observer node from
> Router-Based Federation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] omalley commented on pull request #4584: HADOOP-18345: Enhance client protocol to propagate last seen state IDs for multiple nameservices.
omalley commented on PR #4584: URL: https://github.com/apache/hadoop/pull/4584#issuecomment-1224730686 @goiri No, I committed the change to trunk and branch-3.3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snmvaughan commented on a diff in pull request #4415: MAPREDUCE-7386. Maven parallel builds (skipping tests) fail
snmvaughan commented on code in PR #4415: URL: https://github.com/apache/hadoop/pull/4415#discussion_r953020342 ## hadoop-project/pom.xml: ## @@ -167,7 +167,7 @@ -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -3.0.0-M1 +3.0.0-M7 Review Comment: I've cherry-picked just the upgrade to 3.0.0-M7. https://github.com/apache/hadoop/pull/4795 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18417) Upgrade Maven Surefire plugin
[ https://issues.apache.org/jira/browse/HADOOP-18417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583815#comment-17583815 ] ASF GitHub Bot commented on HADOOP-18417: - snmvaughan opened a new pull request, #4795: URL: https://github.com/apache/hadoop/pull/4795 This addresses an issue where the plugin's default classpath for executing tests fails to include org.junit.platform.launcher.core.LauncherFactory. This commit was originally part of MAPREDUCE-7386, but has been pulled out so that it can be tracked independently. It has already been tested as part of that pull request. - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Upgrade Maven Surefire plugin > - > > Key: HADOOP-18417 > URL: https://issues.apache.org/jira/browse/HADOOP-18417 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0, 3.3.9 >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Blocker > > The Maven Surefire plugin 3.0.0-M1 doesn't always including the launcher as > part of it's setup, which can cause problems with Yarn tests. Some of the > Yarn modules use Jupiter, which may be a complicating factor. Switching to > 3.0.0-M7 fixes the issue. > This is currently blocking MAPREDUCE-7386 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18417) Upgrade Maven Surefire plugin
[ https://issues.apache.org/jira/browse/HADOOP-18417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18417: Labels: pull-request-available (was: ) > Upgrade Maven Surefire plugin > - > > Key: HADOOP-18417 > URL: https://issues.apache.org/jira/browse/HADOOP-18417 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.4.0, 3.3.9 >Reporter: Steve Vaughan >Assignee: Steve Vaughan >Priority: Blocker > Labels: pull-request-available > > The Maven Surefire plugin 3.0.0-M1 doesn't always including the launcher as > part of it's setup, which can cause problems with Yarn tests. Some of the > Yarn modules use Jupiter, which may be a complicating factor. Switching to > 3.0.0-M7 fixes the issue. > This is currently blocking MAPREDUCE-7386 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] snmvaughan opened a new pull request, #4795: HADOOP-18417. Upgrade to M7 of surefire plugin
snmvaughan opened a new pull request, #4795: URL: https://github.com/apache/hadoop/pull/4795 This addresses an issue where the plugin's default classpath for executing tests fails to include org.junit.platform.launcher.core.LauncherFactory. This commit was originally part of MAPREDUCE-7386, but has been pulled out so that it can be tracked independently. It has already been tested as part of that pull request. - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583814#comment-17583814
]
ASF GitHub Bot commented on HADOOP-18073:
-
hadoop-yetus commented on PR #4706:
URL: https://github.com/apache/hadoop/pull/4706#issuecomment-1224664703
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ feature-HADOOP-18073-s3a-sdk-upgrade Compile Tests _ |
| +0 :ok: | mvndep | 15m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 42s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | compile | 23m 11s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 31s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 34s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | mvnsite | 2m 41s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | javadoc | 2m 26s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 26s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 1m 25s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 22m 23s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 54s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 50s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 4s | | the patch passed |
| +1 :green_heart: | compile | 22m 31s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 31s | | the patch passed |
| +1 :green_heart: | compile | 20m 46s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 11s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 21s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 29s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 1m 10s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 22m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 9s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 3m 34s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 35s | | The patch does not
generate ASF License warnings. |
| | | 218m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4706/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4706 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 2d06080f231b 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18073-s3a-sdk-upgrade /
[GitHub] [hadoop] hadoop-yetus commented on pull request #4706: HADOOP-18073. Adds in s3 client config and updates list.
hadoop-yetus commented on PR #4706:
URL: https://github.com/apache/hadoop/pull/4706#issuecomment-1224664703
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ feature-HADOOP-18073-s3a-sdk-upgrade Compile Tests _ |
| +0 :ok: | mvndep | 15m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 42s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | compile | 23m 11s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 31s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 34s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | mvnsite | 2m 41s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed |
| +1 :green_heart: | javadoc | 2m 26s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 26s | |
feature-HADOOP-18073-s3a-sdk-upgrade passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 1m 25s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 22m 23s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 22m 54s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 50s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 4s | | the patch passed |
| +1 :green_heart: | compile | 22m 31s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 31s | | the patch passed |
| +1 :green_heart: | compile | 20m 46s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 11s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 21s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 29s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +0 :ok: | spotbugs | 1m 10s | | hadoop-project has no data from
spotbugs |
| +1 :green_heart: | shadedclient | 22m 33s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 9s | | hadoop-project in the patch
passed. |
| +1 :green_heart: | unit | 3m 34s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 35s | | The patch does not
generate ASF License warnings. |
| | | 218m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4706/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4706 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint spotbugs checkstyle |
| uname | Linux 2d06080f231b 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18073-s3a-sdk-upgrade /
a1e8bedcbcc61d493bcb7dc025af05677144e19a |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
[jira] [Created] (HADOOP-18417) Upgrade Maven Surefire plugin
Steve Vaughan created HADOOP-18417: -- Summary: Upgrade Maven Surefire plugin Key: HADOOP-18417 URL: https://issues.apache.org/jira/browse/HADOOP-18417 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 3.4.0, 3.3.9 Reporter: Steve Vaughan Assignee: Steve Vaughan The Maven Surefire plugin 3.0.0-M1 doesn't always including the launcher as part of it's setup, which can cause problems with Yarn tests. Some of the Yarn modules use Jupiter, which may be a complicating factor. Switching to 3.0.0-M7 fixes the issue. This is currently blocking MAPREDUCE-7386 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4764: YARN-11177. Support getNewReservation, submit / update/ Reservation API's for Federation.
goiri commented on code in PR #4764:
URL: https://github.com/apache/hadoop/pull/4764#discussion_r953004392
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -888,13 +890,120 @@ public MoveApplicationAcrossQueuesResponse
moveApplicationAcrossQueues(
@Override
public GetNewReservationResponse getNewReservation(
GetNewReservationRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null) {
+ routerMetrics.incrGetNewReservationFailedRetrieved();
+ String errMsg = "Missing getNewReservation request.";
+ RouterServerUtil.logAndThrowException(errMsg, null);
+}
+
+long startTime = clock.getTime();
+Map subClustersActive =
+federationFacade.getSubClusters(true);
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+ SubClusterId subClusterId = getRandomActiveSubCluster(subClustersActive);
+ LOG.info("getNewReservation try #{} on SubCluster {}.", i, subClusterId);
+ ApplicationClientProtocol clientRMProxy =
getClientRMProxyForSubCluster(subClusterId);
+ GetNewReservationResponse response = null;
+ try {
+response = clientRMProxy.getNewReservation(request);
+ } catch (Exception e) {
+LOG.warn("Unable to create a new Reservation in SubCluster {}.",
subClusterId.getId(), e);
+subClustersActive.remove(subClusterId);
+ }
+
+ if (response != null) {
+long stopTime = clock.getTime();
+routerMetrics.succeededGetNewReservationRetrieved(stopTime -
startTime);
+return response;
+ }
+}
+
+routerMetrics.incrGetNewReservationFailedRetrieved();
+String errMsg = "Failed to create a new reservation.";
+throw new YarnException(errMsg);
}
@Override
public ReservationSubmissionResponse submitReservation(
ReservationSubmissionRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null || request.getReservationId() == null
+|| request.getReservationDefinition() == null ||
request.getQueue() == null) {
+ routerMetrics.incrSubmitReservationFailedRetrieved();
+ RouterServerUtil.logAndThrowException(
+ "Missing submitReservation request or reservationId " +
+ "or reservation definition or queue.", null);
+}
+
+ReservationSubmissionResponse response = null;
Review Comment:
Declare this where we use it.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/FederationClientInterceptor.java:
##
@@ -888,13 +890,120 @@ public MoveApplicationAcrossQueuesResponse
moveApplicationAcrossQueues(
@Override
public GetNewReservationResponse getNewReservation(
GetNewReservationRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null) {
+ routerMetrics.incrGetNewReservationFailedRetrieved();
+ String errMsg = "Missing getNewReservation request.";
+ RouterServerUtil.logAndThrowException(errMsg, null);
+}
+
+long startTime = clock.getTime();
+Map subClustersActive =
+federationFacade.getSubClusters(true);
+
+for (int i = 0; i < numSubmitRetries; ++i) {
+ SubClusterId subClusterId = getRandomActiveSubCluster(subClustersActive);
+ LOG.info("getNewReservation try #{} on SubCluster {}.", i, subClusterId);
+ ApplicationClientProtocol clientRMProxy =
getClientRMProxyForSubCluster(subClusterId);
+ GetNewReservationResponse response = null;
+ try {
+response = clientRMProxy.getNewReservation(request);
+ } catch (Exception e) {
+LOG.warn("Unable to create a new Reservation in SubCluster {}.",
subClusterId.getId(), e);
+subClustersActive.remove(subClusterId);
+ }
+
+ if (response != null) {
+long stopTime = clock.getTime();
+routerMetrics.succeededGetNewReservationRetrieved(stopTime -
startTime);
+return response;
+ }
+}
+
+routerMetrics.incrGetNewReservationFailedRetrieved();
+String errMsg = "Failed to create a new reservation.";
+throw new YarnException(errMsg);
}
@Override
public ReservationSubmissionResponse submitReservation(
ReservationSubmissionRequest request) throws YarnException, IOException {
-throw new NotImplementedException("Code is not implemented");
+
+if (request == null || request.getReservationId() == null
+|| request.getReservationDefinition() == null ||
request.getQueue() == null) {
+ routerMetrics.incrSubmitReservationFailedRetrieved();
+ RouterServerUtil.logAndThrowException(
+
[GitHub] [hadoop] snmvaughan opened a new pull request, #4794: HDFS-16686. GetJournalEditServlet fails to authorize valid Kerberos request (#4724)
snmvaughan opened a new pull request, #4794: URL: https://github.com/apache/hadoop/pull/4794 Backport from trunk. GetJournalEditServlet uses request.getRemoteuser() to determine the remoteShortName for Kerberos authorization, which fails to match when the JournalNode uses its own Kerberos principal (e.g. jn/@). This can be fixed by using the UserGroupInformation provided by the base DfsServlet class using the getUGI(request, conf) call. - [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4792: YARN-11275. [Federation] Add batchFinishApplicationMaster in UAMPoolManager.
goiri commented on code in PR #4792:
URL: https://github.com/apache/hadoop/pull/4792#discussion_r953000614
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/uam/UnmanagedAMPoolManager.java:
##
@@ -450,4 +452,44 @@ public void drainUAMHeartbeats() {
uam.drainHeartbeatThread();
}
}
+
+ public Map
batchFinishApplicationMaster(
+ FinishApplicationMasterRequest request, String appId) {
+
+Map responseMap = new HashMap<>();
+Set subClusterIds = this.unmanagedAppMasterMap.keySet();
+
+if (subClusterIds.size() > 0) {
Review Comment:
!isEmpty()
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/amrmproxy/TestFederationInterceptor.java:
##
@@ -969,4 +969,58 @@ private PreemptionMessage createDummyPreemptionMessage(
preemptionMessage.setContract(contract);
return preemptionMessage;
}
+
+ @Test
+ public void testBatchFinishApplicationMaster() throws IOException,
InterruptedException {
+
+final RegisterApplicationMasterRequest registerReq =
+Records.newRecord(RegisterApplicationMasterRequest.class);
+registerReq.setHost(Integer.toString(testAppId));
+registerReq.setRpcPort(testAppId);
+registerReq.setTrackingUrl("");
+
+UserGroupInformation ugi =
interceptor.getUGIWithToken(interceptor.getAttemptId());
+
+ugi.doAs((PrivilegedExceptionAction) () -> {
+
+ // Register the application
+ RegisterApplicationMasterRequest registerReq1 =
+ Records.newRecord(RegisterApplicationMasterRequest.class);
+ registerReq1.setHost(Integer.toString(testAppId));
+ registerReq1.setRpcPort(0);
+ registerReq1.setTrackingUrl("");
+
+ // Register ApplicationMaster
+ RegisterApplicationMasterResponse registerResponse =
+ interceptor.registerApplicationMaster(registerReq1);
+ Assert.assertNotNull(registerResponse);
+ lastResponseId = 0;
+
+ Assert.assertEquals(0, interceptor.getUnmanagedAMPoolSize());
+
+ // Allocate the first batch of containers, with sc1 and sc2 active
+ registerSubCluster(SubClusterId.newInstance("SC-1"));
+ registerSubCluster(SubClusterId.newInstance("SC-2"));
+
+ int numberOfContainers = 3;
+ List containers =
+ getContainersAndAssert(numberOfContainers, numberOfContainers * 2);
+ Assert.assertEquals(2, interceptor.getUnmanagedAMPoolSize());
+ Assert.assertEquals(numberOfContainers, containers.size());
+
+ // Finish the application
+ FinishApplicationMasterRequest finishReq =
+ Records.newRecord(FinishApplicationMasterRequest.class);
+ finishReq.setDiagnostics("");
+ finishReq.setTrackingUrl("");
+ finishReq.setFinalApplicationStatus(FinalApplicationStatus.SUCCEEDED);
+
+ FinishApplicationMasterResponse finshResponse =
+ interceptor.finishApplicationMaster(finishReq);
+ Assert.assertNotNull(finshResponse);
+ Assert.assertEquals(true, finshResponse.getIsUnregistered());
Review Comment:
assertTrue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4780: HDFS-16735. Reduce the number of HeartbeatManager loops.
goiri commented on code in PR #4780:
URL: https://github.com/apache/hadoop/pull/4780#discussion_r952993440
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java:
##
@@ -492,12 +498,12 @@ void heartbeatCheck() {
// log nodes detected as stale since last heartBeat
dumpStaleNodes(staleNodes);
- allAlive = dead == null && failedStorage == null;
+ allAlive = deadDatanodes.size() == 0 && failedStorages.size() == 0;
Review Comment:
isEmpty()
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java:
##
@@ -96,6 +97,9 @@ class HeartbeatManager implements DatanodeStatistics {
enableLogStaleNodes = conf.getBoolean(
DFSConfigKeys.DFS_NAMENODE_ENABLE_LOG_STALE_DATANODE_KEY,
DFSConfigKeys.DFS_NAMENODE_ENABLE_LOG_STALE_DATANODE_DEFAULT);
+this.removeBatchNum =
Review Comment:
```
this.removeBatchNum = conf.getInt(
DFSConfigKeys.DFS_NAMENODE_REMOVE_BAD_BATCH_NUM,
DFSConfigKeys.DFS_NAMENODE_REMOVE_BAD_BATCH_NUM_DEFAULT);
```
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/HeartbeatManager.java:
##
@@ -436,12 +440,14 @@ void heartbeatCheck() {
return;
}
boolean allAlive = false;
+// Locate limited dead nodes.
+List deadDatanodes = new ArrayList<>(removeBatchNum);
+// Locate limited failed storages that isn't on a dead node.
+List failedStorages = new ArrayList<>(removeBatchNum);
while (!allAlive) {
Review Comment:
break line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18345) Enhance client protocol to propagate last seen state IDs for multiple nameservices.
[
https://issues.apache.org/jira/browse/HADOOP-18345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583799#comment-17583799
]
ASF GitHub Bot commented on HADOOP-18345:
-
goiri commented on PR #4584:
URL: https://github.com/apache/hadoop/pull/4584#issuecomment-1224567895
@omalley I'm guessing the close was accidental?
> Enhance client protocol to propagate last seen state IDs for multiple
> nameservices.
> ---
>
> Key: HADOOP-18345
> URL: https://issues.apache.org/jira/browse/HADOOP-18345
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The RPCHeader in the client protocol currently contains a single value to
> indicate the last seen state ID for a namenode.
> {noformat}
> optional int64 stateId = 8; // The last seen Global State ID
> {noformat}
> When there are multiple namenodes, such as in router based federation, the
> headers need to carry the state IDs for each of these nameservices that are
> part of the federation.
> This change is a prerequisite for HDFS-13522: RBF: Support observer node from
> Router-Based Federation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on pull request #4584: HADOOP-18345: Enhance client protocol to propagate last seen state IDs for multiple nameservices.
goiri commented on PR #4584: URL: https://github.com/apache/hadoop/pull/4584#issuecomment-1224567895 @omalley I'm guessing the close was accidental? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] abhishekkarigar commented on pull request #4744: HDFS-16689. Standby NameNode crashes when transitioning to Active with in-progress tailer
abhishekkarigar commented on PR #4744:
URL: https://github.com/apache/hadoop/pull/4744#issuecomment-1224559583
@ZanderXu
hi one more question,
i am setting up HA namenode on kubernetes
on the standby namenode , i triggered the bootstrapStandby
$ hdfs namenode -bootstrapStandby
and i get the following error
}{"name":"org.apache.hadoop.hdfs.server.namenode.NameNode","time":1661279164420,"date":"2022-08-23
18:26:04,420","level":"INFO","thread":"main","message":"registered UNIX signal
handlers for [TERM, HUP,
INT]"}{"name":"org.apache.hadoop.hdfs.server.namenode.NameNode","time":1661279164532,"date":"2022-08-23
18:26:04,532","level":"INFO","thread":"main","message":"createNameNode
[-bootstrapStandby]"}{"name":"org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby","time":1661279164846,"date":"2022-08-23
18:26:04,846","level":"INFO","thread":"main","message":"Found nn:
apache-hadoop-namenode-0.apache-hadoop-namenode.nom-backend.svc.cluster.local,
ipc:
hdfs:8020"}{"name":"org.apache.hadoop.hdfs.server.namenode.NameNode","time":1661279164847,"date":"2022-08-23
18:26:04,847","level":"ERROR","thread":"main","message":"Failed to start
namenode.","exceptionclass":"java.io.IOException","stack":["java.io.IOException:
java.lang.NullPointerException","\tat org.apache.hadoop.hdfs.server.nameno
de.ha.BootstrapStandby.run(BootstrapStandby.java:549)","\tat
org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1741)","\tat
org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1834)","Caused
by: java.lang.NullPointerException","\tat
org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.parseConfAndFindOtherNN(BootstrapStandby.java:435)","\tat
org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:114)","\tat
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:81)","\tat
org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:95)","\tat
org.apache.hadoop.hdfs.server.namenode.ha.BootstrapStandby.run(BootstrapStandby.java:544)","\t...
2
more"]}{"name":"org.apache.hadoop.util.ExitUtil","time":1661279164850,"date":"2022-08-23
18:26:04,850","level":"INFO","thread":"main","message":"Exiting with status 1:
java.io.IOException:
java.lang.NullPointerException"}{"name":"org.apache.hadoop.hdfs.server.namenode.NameNode"
,"time":1661279164852,"date":"2022-08-23
18:26:04,852","level":"INFO","thread":"shutdown-hook-0","message":"SHUTDOWN_MSG:
\n/\nSHUTDOWN_MSG:
Shutting down NameNode at
apache-hadoop-namenode-1.apache-hadoop-namenode.nom-backend.svc.cluster.local/10.129.2.45\n
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18345) Enhance client protocol to propagate last seen state IDs for multiple nameservices.
[
https://issues.apache.org/jira/browse/HADOOP-18345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583795#comment-17583795
]
ASF GitHub Bot commented on HADOOP-18345:
-
omalley closed pull request #4584: HADOOP-18345: Enhance client protocol to
propagate last seen state IDs for multiple nameservices.
URL: https://github.com/apache/hadoop/pull/4584
> Enhance client protocol to propagate last seen state IDs for multiple
> nameservices.
> ---
>
> Key: HADOOP-18345
> URL: https://issues.apache.org/jira/browse/HADOOP-18345
> Project: Hadoop Common
> Issue Type: New Feature
>Reporter: Simbarashe Dzinamarira
>Assignee: Simbarashe Dzinamarira
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> The RPCHeader in the client protocol currently contains a single value to
> indicate the last seen state ID for a namenode.
> {noformat}
> optional int64 stateId = 8; // The last seen Global State ID
> {noformat}
> When there are multiple namenodes, such as in router based federation, the
> headers need to carry the state IDs for each of these nameservices that are
> part of the federation.
> This change is a prerequisite for HDFS-13522: RBF: Support observer node from
> Router-Based Federation
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] omalley closed pull request #4584: HADOOP-18345: Enhance client protocol to propagate last seen state IDs for multiple nameservices.
omalley closed pull request #4584: HADOOP-18345: Enhance client protocol to propagate last seen state IDs for multiple nameservices. URL: https://github.com/apache/hadoop/pull/4584 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16806) AWS AssumedRoleCredentialProvider needs ExternalId add
[
https://issues.apache.org/jira/browse/HADOOP-16806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583772#comment-17583772
]
ASF GitHub Bot commented on HADOOP-16806:
-
jmahonin commented on PR #4753:
URL: https://github.com/apache/hadoop/pull/4753#issuecomment-1224436307
Thanks @dannycjones, that did it!
All integration tests have now passed in this PR.
I was indeed using `us-east-1` in `fs.s3a.endpoint.region`, which would
explain the issue. Might make sense to add some additional instructions in
`hadoop-aws/testing.md` under `Testing against different regions`. I can
include that as part of this PR, or file a separate one.
> AWS AssumedRoleCredentialProvider needs ExternalId add
> --
>
> Key: HADOOP-16806
> URL: https://issues.apache.org/jira/browse/HADOOP-16806
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.1
>Reporter: Jon Hartlaub
>Priority: Minor
> Labels: pull-request-available
>
> AWS has added a security feature to the assume-role function in the form of
> the "ExternalId" key in the AWS Java SDK
> {{STSAssumeRoleSessionCredentialsProvider.Builder}} class. To support this
> security feature, the hadoop aws {{AssumedRoleCredentialProvider}} needs a
> patch to include this value from the configuration as well as an added
> Constant to the {{org.apache.hadoop.fs.s3a.Constants}} file.
> The ExternalId is not a required security feature, it is an augmentation of
> the current assume role configuration.
> Proposed:
> * Get the assume-role ExternalId token from the configuration for the
> configuration key {{fs.s3a.assumed.role.externalid}}
> * Use the configured ExternalId value in the
> {{STSAssumeRoleSessionCredentialsProvider.Builder}}
> e.g.
> {{if (StringUtils.isNotEmpty(externalId)) {}}
> {{ builder.withExternalId(externalId); // include the token for
> cross-account assume role}}
> {{}}}
> Tests:
> * +Unit test+ which verifies the ExternalId state value of the
> {{AssumedRoleCredentialProvider}} is consistent with the configured value -
> either empty or populated
> * Question: not sure about how to write the +integration test+ for this
> feature. We have an account configured for this use-case that verifies this
> feature but I don't have much context on the Hadoop project AWS S3
> integration tests, perhaps a pointer could help.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jmahonin commented on pull request #4753: HADOOP-16806: AWS AssumedRoleCredentialProvider needs ExternalId add
jmahonin commented on PR #4753: URL: https://github.com/apache/hadoop/pull/4753#issuecomment-1224436307 Thanks @dannycjones, that did it! All integration tests have now passed in this PR. I was indeed using `us-east-1` in `fs.s3a.endpoint.region`, which would explain the issue. Might make sense to add some additional instructions in `hadoop-aws/testing.md` under `Testing against different regions`. I can include that as part of this PR, or file a separate one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18416) ITestS3AIOStatisticsContext failure
[
https://issues.apache.org/jira/browse/HADOOP-18416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran reassigned HADOOP-18416:
---
Assignee: Mehakmeet Singh
> ITestS3AIOStatisticsContext failure
> ---
>
> Key: HADOOP-18416
> URL: https://issues.apache.org/jira/browse/HADOOP-18416
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.9
>Reporter: Steve Loughran
>Assignee: Mehakmeet Singh
>Priority: Major
> Attachments:
> org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext-output.txt
>
>
> test failure running the new ITestS3AIOStatisticsContext. attaching the stack
> and log file.
> This happened on a large (12 thread) test run, but i can get it to come back
> intermittently on repeated runs of the whole suite, but never when i just run
> the single test case.
> {code}
> [ERROR]
> testThreadIOStatisticsForDifferentThreads(org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext)
> Time elapsed: 3.616 s <<< FAILURE!
> java.lang.AssertionError:
> [Counter named stream_write_bytes]
> Expecting actual not to be null
> at
> org.apache.hadoop.fs.statistics.IOStatisticAssertions.lookupStatistic(IOStatisticAssertions.java:160)
> at
> org.apache.hadoop.fs.statistics.IOStatisticAssertions.assertThatStatisticLong(IOStatisticAssertions.java:291)
> at
> org.apache.hadoop.fs.statistics.IOStatisticAssertions.assertThatStatisticCounter(IOStatisticAssertions.java:306)
> at
> org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext.assertThreadStatisticsForThread(ITestS3AIOStatisticsContext.java:367)
> at
> org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext.testThreadIOStatisticsForDifferentThreads(ITestS3AIOStatisticsContext.java:260)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:750)
> {code}
> I'm suspecting some race condition *or* gc pressure is releasing that
> reference in the worker thread.
> proposed test changes
> * worker thread changes its thread ID for the logs
> * stores its thread context into a field, so there's guarantee of no GC
> * logs more as it goes along.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18416) ITestS3AIOStatisticsContext failure
[
https://issues.apache.org/jira/browse/HADOOP-18416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18416:
Attachment: org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext-output.txt
> ITestS3AIOStatisticsContext failure
> ---
>
> Key: HADOOP-18416
> URL: https://issues.apache.org/jira/browse/HADOOP-18416
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3, test
>Affects Versions: 3.3.9
>Reporter: Steve Loughran
>Priority: Major
> Attachments:
> org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext-output.txt
>
>
> test failure running the new ITestS3AIOStatisticsContext. attaching the stack
> and log file.
> This happened on a large (12 thread) test run, but i can get it to come back
> intermittently on repeated runs of the whole suite, but never when i just run
> the single test case.
> {code}
> [ERROR]
> testThreadIOStatisticsForDifferentThreads(org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext)
> Time elapsed: 3.616 s <<< FAILURE!
> java.lang.AssertionError:
> [Counter named stream_write_bytes]
> Expecting actual not to be null
> at
> org.apache.hadoop.fs.statistics.IOStatisticAssertions.lookupStatistic(IOStatisticAssertions.java:160)
> at
> org.apache.hadoop.fs.statistics.IOStatisticAssertions.assertThatStatisticLong(IOStatisticAssertions.java:291)
> at
> org.apache.hadoop.fs.statistics.IOStatisticAssertions.assertThatStatisticCounter(IOStatisticAssertions.java:306)
> at
> org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext.assertThreadStatisticsForThread(ITestS3AIOStatisticsContext.java:367)
> at
> org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext.testThreadIOStatisticsForDifferentThreads(ITestS3AIOStatisticsContext.java:260)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:750)
> {code}
> I'm suspecting some race condition *or* gc pressure is releasing that
> reference in the worker thread.
> proposed test changes
> * worker thread changes its thread ID for the logs
> * stores its thread context into a field, so there's guarantee of no GC
> * logs more as it goes along.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18416) ITestS3AIOStatisticsContext failure
Steve Loughran created HADOOP-18416:
---
Summary: ITestS3AIOStatisticsContext failure
Key: HADOOP-18416
URL: https://issues.apache.org/jira/browse/HADOOP-18416
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3, test
Affects Versions: 3.3.9
Reporter: Steve Loughran
test failure running the new ITestS3AIOStatisticsContext. attaching the stack
and log file.
This happened on a large (12 thread) test run, but i can get it to come back
intermittently on repeated runs of the whole suite, but never when i just run
the single test case.
{code}
[ERROR]
testThreadIOStatisticsForDifferentThreads(org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext)
Time elapsed: 3.616 s <<< FAILURE!
java.lang.AssertionError:
[Counter named stream_write_bytes]
Expecting actual not to be null
at
org.apache.hadoop.fs.statistics.IOStatisticAssertions.lookupStatistic(IOStatisticAssertions.java:160)
at
org.apache.hadoop.fs.statistics.IOStatisticAssertions.assertThatStatisticLong(IOStatisticAssertions.java:291)
at
org.apache.hadoop.fs.statistics.IOStatisticAssertions.assertThatStatisticCounter(IOStatisticAssertions.java:306)
at
org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext.assertThreadStatisticsForThread(ITestS3AIOStatisticsContext.java:367)
at
org.apache.hadoop.fs.s3a.ITestS3AIOStatisticsContext.testThreadIOStatisticsForDifferentThreads(ITestS3AIOStatisticsContext.java:260)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
{code}
I'm suspecting some race condition *or* gc pressure is releasing that reference
in the worker thread.
proposed test changes
* worker thread changes its thread ID for the logs
* stores its thread context into a field, so there's guarantee of no GC
* logs more as it goes along.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-18388) Allow dynamic groupSearchFilter in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-18388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Larry McCay reassigned HADOOP-18388:
Assignee: Ayush Saxena
> Allow dynamic groupSearchFilter in LdapGroupsMapping
>
>
> Key: HADOOP-18388
> URL: https://issues.apache.org/jira/browse/HADOOP-18388
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: dynamic-filter-idea.patch
>
>
> As of now the lookupGroup() method doesn't allow to have placeholders in
> groupSearchFilter, so that can not be dynamically adjusted.
> If we have placeholders for groupSearchFilter like:
> (&(|(XYZ=\{0})(ABC=\{1}))(objectClass=posixGroup))
> This fails here:
>
> {code:java}
> groupResults =
> c.search(groupbaseDN,
> "(&" + groupSearchFilter + "(" + groupMemberAttr + "={0}))",
> new Object[]{userDn},
> SEARCH_CONTROLS); {code}
> With
>
>
> {noformat}
> javax.naming.directory.InvalidSearchFilterException: number exceeds argument
> list: 1; remaining name {noformat}
>
> >>Dropped off or changed the details above which I thought won't be safe to
> >>disclose.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18388) Allow dynamic groupSearchFilter in LdapGroupsMapping
[
https://issues.apache.org/jira/browse/HADOOP-18388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583753#comment-17583753
]
Larry McCay commented on HADOOP-18388:
--
[~ayushtkn] - thank you for the patch.
I'd like to see a unit test added as well to tease out the expected behavior.
Have you actually built and tried a patched version of LdapGroupsMapping?
> Allow dynamic groupSearchFilter in LdapGroupsMapping
>
>
> Key: HADOOP-18388
> URL: https://issues.apache.org/jira/browse/HADOOP-18388
> Project: Hadoop Common
> Issue Type: Improvement
> Components: security
>Reporter: Ayush Saxena
>Priority: Major
> Attachments: dynamic-filter-idea.patch
>
>
> As of now the lookupGroup() method doesn't allow to have placeholders in
> groupSearchFilter, so that can not be dynamically adjusted.
> If we have placeholders for groupSearchFilter like:
> (&(|(XYZ=\{0})(ABC=\{1}))(objectClass=posixGroup))
> This fails here:
>
> {code:java}
> groupResults =
> c.search(groupbaseDN,
> "(&" + groupSearchFilter + "(" + groupMemberAttr + "={0}))",
> new Object[]{userDn},
> SEARCH_CONTROLS); {code}
> With
>
>
> {noformat}
> javax.naming.directory.InvalidSearchFilterException: number exceeds argument
> list: 1; remaining name {noformat}
>
> >>Dropped off or changed the details above which I thought won't be safe to
> >>disclose.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-16806) AWS AssumedRoleCredentialProvider needs ExternalId add
[
https://issues.apache.org/jira/browse/HADOOP-16806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583748#comment-17583748
]
ASF GitHub Bot commented on HADOOP-16806:
-
dannycjones commented on PR #4753:
URL: https://github.com/apache/hadoop/pull/4753#issuecomment-1224331509
Unfortunately, the errors can be a bit vague for S3 HeadObject (used by
getFileStatus). There was a similar issue when the credentials had expired in
HADOOP-18353.
```
...
Authorization: AWS4-HMAC-SHA256
Credential=/20220823/us-east-1/s3/aws4_request
...
```
The signature is signed for us-east-1, but the `landsat-pds` bucket is in
us-west-2. Do you have `fs.s3a.endpoint.region` set? If so, try adding the
following to your _auth-keys.xml_:
```
fs.s3a.bucket.landsat-pds.endpoint.region
us-west-2
```
> AWS AssumedRoleCredentialProvider needs ExternalId add
> --
>
> Key: HADOOP-16806
> URL: https://issues.apache.org/jira/browse/HADOOP-16806
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.2.1
>Reporter: Jon Hartlaub
>Priority: Minor
> Labels: pull-request-available
>
> AWS has added a security feature to the assume-role function in the form of
> the "ExternalId" key in the AWS Java SDK
> {{STSAssumeRoleSessionCredentialsProvider.Builder}} class. To support this
> security feature, the hadoop aws {{AssumedRoleCredentialProvider}} needs a
> patch to include this value from the configuration as well as an added
> Constant to the {{org.apache.hadoop.fs.s3a.Constants}} file.
> The ExternalId is not a required security feature, it is an augmentation of
> the current assume role configuration.
> Proposed:
> * Get the assume-role ExternalId token from the configuration for the
> configuration key {{fs.s3a.assumed.role.externalid}}
> * Use the configured ExternalId value in the
> {{STSAssumeRoleSessionCredentialsProvider.Builder}}
> e.g.
> {{if (StringUtils.isNotEmpty(externalId)) {}}
> {{ builder.withExternalId(externalId); // include the token for
> cross-account assume role}}
> {{}}}
> Tests:
> * +Unit test+ which verifies the ExternalId state value of the
> {{AssumedRoleCredentialProvider}} is consistent with the configured value -
> either empty or populated
> * Question: not sure about how to write the +integration test+ for this
> feature. We have an account configured for this use-case that verifies this
> feature but I don't have much context on the Hadoop project AWS S3
> integration tests, perhaps a pointer could help.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dannycjones commented on pull request #4753: HADOOP-16806: AWS AssumedRoleCredentialProvider needs ExternalId add
dannycjones commented on PR #4753: URL: https://github.com/apache/hadoop/pull/4753#issuecomment-1224331509 Unfortunately, the errors can be a bit vague for S3 HeadObject (used by getFileStatus). There was a similar issue when the credentials had expired in HADOOP-18353. ``` ... Authorization: AWS4-HMAC-SHA256 Credential=/20220823/us-east-1/s3/aws4_request ... ``` The signature is signed for us-east-1, but the `landsat-pds` bucket is in us-west-2. Do you have `fs.s3a.endpoint.region` set? If so, try adding the following to your _auth-keys.xml_: ``` fs.s3a.bucket.landsat-pds.endpoint.region us-west-2 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4728: MAPREDUCE-7403. manifest-committer dynamic partitioning support.
hadoop-yetus commented on PR #4728:
URL: https://github.com/apache/hadoop/pull/4728#issuecomment-1224309966
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 31s | | trunk passed |
| +1 :green_heart: | compile | 25m 15s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 52s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 56s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 57s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 42s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 28s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 24m 35s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 35s | | the patch passed |
| +1 :green_heart: | compile | 22m 4s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 4s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 25s | | the patch passed |
| +1 :green_heart: | mvnsite | 2m 27s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 49s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 57s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 7m 30s | | hadoop-mapreduce-client-core in
the patch passed. |
| +1 :green_heart: | unit | 3m 5s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 17s | | The patch does not
generate ASF License warnings. |
| | | 233m 58s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4728/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4728 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 2b93cafc2124 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bc9dfc96a9471fa384a2bab0273b7fee8a325e05 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4728/6/testReport/ |
| Max. process+thread count | 1082 (vs. ulimit of 5500) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
hadoop-tools/hadoop-aws U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4728/6/console |
| versions | git=2.25.1 maven=3.6.3
[jira] [Commented] (HADOOP-16806) AWS AssumedRoleCredentialProvider needs ExternalId add
[ https://issues.apache.org/jira/browse/HADOOP-16806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583733#comment-17583733 ] ASF GitHub Bot commented on HADOOP-16806: - jmahonin commented on PR #4753: URL: https://github.com/apache/hadoop/pull/4753#issuecomment-1224304355 It looks like the issue is in retrieving `s3a://landsat-pds/scene_list.gz`. The corresponding role should have full S3 access at this point. I've enabled the low-level request tracing. Here's what it looks like for the first parameterized test: `testJobSubmissionCollectsTokens[0](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob)` ``` 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "HEAD /scene_list.gz HTTP/1.1[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Host: landsat-pds.s3.amazonaws.com[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "amz-sdk-invocation-id: [\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "amz-sdk-request: attempt=1;max=21[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "amz-sdk-retry: 0/0/500[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Authorization: AWS4-HMAC-SHA256 Credential=/20220823/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;referer;user-agent;x-amz-content-sha256;x-amz-date, Signature=[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Content-Type: application/octet-stream[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Referer: https://audit.example.org/hadoop/1/op_get_file_status//?op=op_get_file_status=scene_list.gz=jmahonin/localh...@example.com===15==15=1661271325482[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "User-Agent: Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.262 Linux/5.10.104-linuxkit OpenJDK_64-Bit_Server_VM/25.342-b07 java/1.8.0_342 kotlin/1.4.10 vendor/Private_Build cfg/retry-mode/legacy[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "X-Amz-Date: 20220823T161525Z[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Connection: Keep-Alive[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "HTTP/1.1 400 Bad Request[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "x-amz-request-id: ZMY1169S2RC7NEHP[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "x-amz-id-2: ENtHr5DR7HX+qkh5FfYQublaU82ykB/SD5fAvR5kC6JZJibFVLH6Rq+F/EutrE3dAL1uTz6yad8=[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Content-Type: application/xml[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Date: Tue, 23 Aug 2022 16:15:25 GMT[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Server: AmazonS3[\r][\n]" 2022-08-23 16:15:25,937 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Connection: close[\r][\n]" 2022-08-23 16:15:25,937 [JUnit-tes
[GitHub] [hadoop] jmahonin commented on pull request #4753: HADOOP-16806: AWS AssumedRoleCredentialProvider needs ExternalId add
jmahonin commented on PR #4753: URL: https://github.com/apache/hadoop/pull/4753#issuecomment-1224304355 It looks like the issue is in retrieving `s3a://landsat-pds/scene_list.gz`. The corresponding role should have full S3 access at this point. I've enabled the low-level request tracing. Here's what it looks like for the first parameterized test: `testJobSubmissionCollectsTokens[0](org.apache.hadoop.fs.s3a.auth.delegation.ITestDelegatedMRJob)` ``` 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "HEAD /scene_list.gz HTTP/1.1[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Host: landsat-pds.s3.amazonaws.com[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "amz-sdk-invocation-id: [\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "amz-sdk-request: attempt=1;max=21[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "amz-sdk-retry: 0/0/500[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Authorization: AWS4-HMAC-SHA256 Credential=/20220823/us-east-1/s3/aws4_request, SignedHeaders=amz-sdk-invocation-id;amz-sdk-request;amz-sdk-retry;content-type;host;referer;user-agent;x-amz-content-sha256;x-amz-date, Signature=[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Content-Type: application/octet-stream[\r][\n]" 2022-08-23 16:15:25,838 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Referer: https://audit.example.org/hadoop/1/op_get_file_status//?op=op_get_file_status=scene_list.gz=jmahonin/localh...@example.com===15==15=1661271325482[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "User-Agent: Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.262 Linux/5.10.104-linuxkit OpenJDK_64-Bit_Server_VM/25.342-b07 java/1.8.0_342 kotlin/1.4.10 vendor/Private_Build cfg/retry-mode/legacy[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "x-amz-content-sha256: UNSIGNED-PAYLOAD[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "X-Amz-Date: 20220823T161525Z[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "Connection: Keep-Alive[\r][\n]" 2022-08-23 16:15:25,839 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 >> "[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "HTTP/1.1 400 Bad Request[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "x-amz-request-id: ZMY1169S2RC7NEHP[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "x-amz-id-2: ENtHr5DR7HX+qkh5FfYQublaU82ykB/SD5fAvR5kC6JZJibFVLH6Rq+F/EutrE3dAL1uTz6yad8=[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Content-Type: application/xml[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Date: Tue, 23 Aug 2022 16:15:25 GMT[\r][\n]" 2022-08-23 16:15:25,936 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Server: AmazonS3[\r][\n]" 2022-08-23 16:15:25,937 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "Connection: close[\r][\n]" 2022-08-23 16:15:25,937 [JUnit-testJobSubmissionCollectsTokens[0]] DEBUG http.wire (Wire.java:wire(73)) - http-outgoing-7 << "[\r][\n]" ``` I've also tried adding this to my auth-keys, which likewise does not work: ``` fs.s3a.bucket.landsat-pds.aws.credentials
[GitHub] [hadoop] hadoop-yetus commented on pull request #4791: HDFS-16738. Invalid CallerContext caused NullPointerException
hadoop-yetus commented on PR #4791:
URL: https://github.com/apache/hadoop/pull/4791#issuecomment-1224301825
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 38s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 31s | | trunk passed |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 1m 33s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 17s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 40s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 19s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 46s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 1m 30s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 1m 30s | | the patch passed |
| +1 :green_heart: | compile | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 7s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 32s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 47s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 250m 9s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 9s | | The patch does not
generate ASF License warnings. |
| | | 361m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4791/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4791 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5ba28308f245 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5be62a40b4cbc6b6ca27ff8044545d24664c3a85 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4791/2/testReport/ |
| Max. process+thread count | 3465 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4791/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about
[jira] [Commented] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583699#comment-17583699
]
ASF GitHub Bot commented on HADOOP-18073:
-
ahmarsuhail commented on code in PR #4706:
URL: https://github.com/apache/hadoop/pull/4706#discussion_r952789775
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestRequestFactory.java:
##
@@ -170,9 +170,11 @@ private void createFactoryObjects(RequestFactory factory) {
a(factory.newGetObjectRequest(path));
a(factory.newGetObjectMetadataRequest(path));
a(factory.newListMultipartUploadsRequest(path));
-a(factory.newListObjectsV1Request(path, "/", 1));
+//TODO: Commenting out for now, new request extends AwsRequest, this can
be updated once all
+// request factory operations are updated.
Review Comment:
have done requestPreparer, this one is slightly more complicated to do as it
needs to return `AWSRequestAnalyzer.RequestInfo`.
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Assignee: Ahmar Suhail
>Priority: Major
> Labels: pull-request-available
> Attachments: Upgrading S3A to SDKV2.pdf
>
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ahmarsuhail commented on a diff in pull request #4706: HADOOP-18073. Adds in s3 client config and updates list.
ahmarsuhail commented on code in PR #4706:
URL: https://github.com/apache/hadoop/pull/4706#discussion_r952789775
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/impl/TestRequestFactory.java:
##
@@ -170,9 +170,11 @@ private void createFactoryObjects(RequestFactory factory) {
a(factory.newGetObjectRequest(path));
a(factory.newGetObjectMetadataRequest(path));
a(factory.newListMultipartUploadsRequest(path));
-a(factory.newListObjectsV1Request(path, "/", 1));
+//TODO: Commenting out for now, new request extends AwsRequest, this can
be updated once all
+// request factory operations are updated.
Review Comment:
have done requestPreparer, this one is slightly more complicated to do as it
needs to return `AWSRequestAnalyzer.RequestInfo`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18333) hadoop-client-runtime impact by CVE-2022-2047 CVE-2022-2048 due to shaded jetty
[ https://issues.apache.org/jira/browse/HADOOP-18333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17583668#comment-17583668 ] ASF GitHub Bot commented on HADOOP-18333: - ashutoshcipher commented on PR #4600: URL: https://github.com/apache/hadoop/pull/4600#issuecomment-1224208228 @jojochuang I ran the following tests which are failing in Yetus Run and they are passing in local. ``` hadoop.mapred.TestMRTimelineEventHandling hadoop.mapred.TestMiniMRClasspath hadoop.yarn.client.api.impl.TestAMRMClient hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade hadoop.hdfs.tools.TestDFSAdmin hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery ``` > hadoop-client-runtime impact by CVE-2022-2047 CVE-2022-2048 due to shaded > jetty > --- > > Key: HADOOP-18333 > URL: https://issues.apache.org/jira/browse/HADOOP-18333 > Project: Hadoop Common > Issue Type: Improvement > Components: build >Affects Versions: 3.3.3 >Reporter: phoebe chen >Assignee: groot >Priority: Major > Labels: pull-request-available > Time Spent: 1h 50m > Remaining Estimate: 0h > > CVE-2022-2047 and CVE-2022-2048 is recently found for Eclipse Jetty, and > impacts 9.4.0 thru 9.4.46. > In latest 3.3.3 of hadoop-client-runtime, it shaded 9.4.43.v20210629 version > jetty which is impacted. > In Trunk, Jetty is in version 9.4.44.v20210927, which is still impacted. > Need to upgrade Jetty Version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ashutoshcipher commented on pull request #4600: HADOOP-18333. Upgrade jetty version to 9.4.48.v20220622
ashutoshcipher commented on PR #4600: URL: https://github.com/apache/hadoop/pull/4600#issuecomment-1224208228 @jojochuang I ran the following tests which are failing in Yetus Run and they are passing in local. ``` hadoop.mapred.TestMRTimelineEventHandling hadoop.mapred.TestMiniMRClasspath hadoop.yarn.client.api.impl.TestAMRMClient hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade hadoop.hdfs.tools.TestDFSAdmin hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaRecovery ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #4724: HDFS-16686. GetJournalEditServlet fails to authorize valid Kerberos request
sunchao commented on PR #4724: URL: https://github.com/apache/hadoop/pull/4724#issuecomment-1224203561 Merged, thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao merged pull request #4724: HDFS-16686. GetJournalEditServlet fails to authorize valid Kerberos request
sunchao merged PR #4724: URL: https://github.com/apache/hadoop/pull/4724 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4792: YARN-11275. [Federation] Add batchFinishApplicationMaster in UAMPoolManager.
hadoop-yetus commented on PR #4792:
URL: https://github.com/apache/hadoop/pull/4792#issuecomment-1224181213
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 23s | | trunk passed |
| +1 :green_heart: | compile | 10m 40s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 9m 9s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 2m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 3m 29s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 3m 16s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 59s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 1s | | the patch passed |
| +1 :green_heart: | compile | 9m 52s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 9m 52s | | the patch passed |
| +1 :green_heart: | compile | 9m 2s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 9m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 3m 2s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 56s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 6m 4s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 17s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 5m 6s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 10s | | hadoop-yarn-server-common in
the patch passed. |
| -1 :x: | unit | 23m 55s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4792/2/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch passed. |
| +1 :green_heart: | asflicense | 1m 8s | | The patch does not
generate ASF License warnings. |
| | | 207m 19s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.nodemanager.amrmproxy.TestFederationInterceptor |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4792/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4792 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3d9357c034ea 4.15.0-191-generic #202-Ubuntu SMP Thu Aug 4
01:49:29 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / e7fdf271ece3c64c2410600100c4460a0235f705 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
|
[GitHub] [hadoop] slfan1989 commented on pull request #4793: YARN-11276. Add lru cache for RMWebServices.getApps
slfan1989 commented on PR #4793: URL: https://github.com/apache/hadoop/pull/4793#issuecomment-1224136306 @leixm Modify CheckStyle and add relevant parameters in yarn-site.xml. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on a diff in pull request #4761: HDFS-16733.Improve INode#isRoot().
Hexiaoqiao commented on code in PR #4761:
URL: https://github.com/apache/hadoop/pull/4761#discussion_r952681531
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/INode.java:
##
@@ -73,7 +73,8 @@ public abstract class INode implements INodeAttributes,
Diff.Element {
* Check whether this is the root inode.
*/
final boolean isRoot() {
-return getLocalNameBytes().length == 0;
+// Note: There is no restriction that id must be equal to
INodeId#ROOT_INODE_ID
+return getLocalNameBytes() == null || getLocalNameBytes().length == 0;
Review Comment:
Got it. If that is true, I think we should submit PR to branch FGL first?
Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
