[GitHub] [hadoop] hadoop-yetus commented on pull request #5333: HDFS-16815. Error occurred in processing CacheManagerSection for xml parsing fsimage
hadoop-yetus commented on PR #5333: URL: https://github.com/apache/hadoop/pull/5333#issuecomment-1409797883 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 24s | | https://github.com/apache/hadoop/pull/5333 does not apply to trunk. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/5333 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5333/1/console | | versions | git=2.17.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5328: YARN-11222. [Federation] Add addToClusterNodeLabels, removeFromClusterNodeLabels REST APIs for Router.
hadoop-yetus commented on PR #5328:
URL: https://github.com/apache/hadoop/pull/5328#issuecomment-1409797397
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 2m 6s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 1s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 18s | | trunk passed |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 34s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 38s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 43s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 33s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 29s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 112m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5328/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5328 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 36635898aa7a 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 2c5e0fb73048175ffdda0b21f6c78e60f42eee95 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5328/6/testReport/ |
| Max. process+thread count | 630 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5328/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[GitHub] [hadoop] meijing123 opened a new pull request, #5333: HDFS-16815. Error occurred in processing CacheManagerSection for xml parsing fsimage
meijing123 opened a new pull request, #5333: URL: https://github.com/apache/hadoop/pull/5333 1. We added a test method to test the CacheManagerSectionProcessor. 2. We fixed two bugs in the processor of CacheManagerSectionProcessor to make sure the xml parsing fsimage. ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5272: YARN-11217. [Federation] Add dumpSchedulerLogs REST APIs for Router.
hadoop-yetus commented on PR #5272:
URL: https://github.com/apache/hadoop/pull/5272#issuecomment-1409697250
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 40s | | trunk passed |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 59s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 26m 16s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 8s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 28s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 109m 21s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5272/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5272 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 8689bd765f2b 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 11900833fe0e4a50db154a682a81509bb2c338ca |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5272/16/testReport/ |
| Max. process+thread count | 536 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5272/16/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from
[GitHub] [hadoop] hadoop-yetus commented on pull request #5328: YARN-11222. [Federation] Add addToClusterNodeLabels, removeFromClusterNodeLabels REST APIs for Router.
hadoop-yetus commented on PR #5328:
URL: https://github.com/apache/hadoop/pull/5328#issuecomment-1409685217
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 28s | | trunk passed |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 37s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 29s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 28s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 112m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5328/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5328 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 592f96b58d29 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 977303772825f5bf3c9acde9855e0aafeab919f6 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5328/5/testReport/ |
| Max. process+thread count | 544 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5328/5/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[GitHub] [hadoop] hadoop-yetus commented on pull request #5324: HDFS-16895. NamenodeHeartbeatService should use credentials of logged…
hadoop-yetus commented on PR #5324:
URL: https://github.com/apache/hadoop/pull/5324#issuecomment-1409679730
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 31s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 30s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 16s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 49s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 41m 9s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5324/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 154m 10s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver
|
| | hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5324/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5324 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 3b14d27ed9fa 4.15.0-197-generic #208-Ubuntu SMP Tue Nov 1
17:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d4e8bfedb7c9f3a9da2a28acf849c29ee0bb89d1 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5324/4/testReport/ |
| Max. process+thread count | 2107 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5324/4/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus
[GitHub] [hadoop] haiyang1987 commented on pull request #5296: HDFS-16888. BlockManager#maxReplicationStreams, replicationStreamsHardLimit, blocksReplWorkMultiplier and PendingReconstructionBlocks#tim
haiyang1987 commented on PR #5296: URL: https://github.com/apache/hadoop/pull/5296#issuecomment-1409649554 Hi @tomscut @tasanuma the PR already rebase，please help review it again. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] meijing123 closed pull request #5069: HDFS-16815. Error occurred in processing CacheManagerSection for xml parsing fsimage
meijing123 closed pull request #5069: HDFS-16815. Error occurred in processing CacheManagerSection for xml parsing fsimage URL: https://github.com/apache/hadoop/pull/5069 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[
https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682340#comment-17682340
]
ASF GitHub Bot commented on HADOOP-18206:
-
hadoop-yetus commented on PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1409627535
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 28 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 30s | | trunk passed |
| +1 :green_heart: | compile | 23m 9s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 20m 27s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 18m 59s | | trunk passed |
| -1 :x: | javadoc | 1m 22s |
[/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/18/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 7m 17s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 22s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 3m 51s |
[/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/18/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html)
| hadoop-mapreduce-project/hadoop-mapreduce-client in trunk has 1 extant
spotbugs warnings. |
| -1 :x: | spotbugs | 32m 7s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/18/artifact/out/branch-spotbugs-root-warnings.html)
| root in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 54m 10s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 54m 31s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 41s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 41m 25s | | the patch passed |
| +1 :green_heart: | compile | 22m 36s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 22m 36s | |
root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 0 new + 2813 unchanged - 5
fixed = 2813 total (was 2818) |
| +1 :green_heart: | compile | 20m 31s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 20m 31s | |
root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 0 new + 2613 unchanged - 5 fixed
= 2613 total (was 2618) |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 47s | | root: The patch generated
0 new + 682 unchanged - 28 fixed = 682 total (was 710) |
| +1 :green_heart: | mvnsite | 19m 14s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| -1 :x: | javadoc | 1m 14s |
[/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/18/artifact/out/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 8m 2s | | the patch passed with JDK
Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
hadoop-yetus commented on PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1409627535
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 35s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 28 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 52s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 30s | | trunk passed |
| +1 :green_heart: | compile | 23m 9s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 20m 27s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 3m 50s | | trunk passed |
| +1 :green_heart: | mvnsite | 18m 59s | | trunk passed |
| -1 :x: | javadoc | 1m 22s |
[/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/18/artifact/out/branch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in trunk failed with JDK Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 7m 17s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 22s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| -1 :x: | spotbugs | 3m 51s |
[/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/18/artifact/out/branch-spotbugs-hadoop-mapreduce-project_hadoop-mapreduce-client-warnings.html)
| hadoop-mapreduce-project/hadoop-mapreduce-client in trunk has 1 extant
spotbugs warnings. |
| -1 :x: | spotbugs | 32m 7s |
[/branch-spotbugs-root-warnings.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/18/artifact/out/branch-spotbugs-root-warnings.html)
| root in trunk has 1 extant spotbugs warnings. |
| +1 :green_heart: | shadedclient | 54m 10s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 54m 31s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 41s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 41m 25s | | the patch passed |
| +1 :green_heart: | compile | 22m 36s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 22m 36s | |
root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 generated 0 new + 2813 unchanged - 5
fixed = 2813 total (was 2818) |
| +1 :green_heart: | compile | 20m 31s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 20m 31s | |
root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08 with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08 generated 0 new + 2613 unchanged - 5 fixed
= 2613 total (was 2618) |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 3m 47s | | root: The patch generated
0 new + 682 unchanged - 28 fixed = 682 total (was 710) |
| +1 :green_heart: | mvnsite | 19m 14s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 1s | | No new issues. |
| -1 :x: | javadoc | 1m 14s |
[/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5315/18/artifact/out/patch-javadoc-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 8m 2s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +0 :ok: | spotbugs | 0m 19s | | hadoop-project has no data from
spotbugs |
| -1 :x: | shadedclient | 55m 39s | | patch has errors when building
and testing our client artifacts. |
[GitHub] [hadoop] hchaverri commented on a diff in pull request #5324: HDFS-16895. NamenodeHeartbeatService should use credentials of logged…
hchaverri commented on code in PR #5324:
URL: https://github.com/apache/hadoop/pull/5324#discussion_r1091307738
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/NamenodeHeartbeatService.java:
##
@@ -339,8 +341,11 @@ protected NamenodeStatusReport getNamenodeStatusReport() {
// should be required at some point for QoS
updateSafeModeParameters(serviceURI, report);
- // Read the stats from JMX (optional)
- updateJMXParameters(webAddress, report);
+ // Read the stats from JMX (optional) using the login user credentials
+ SecurityUtil.doAsLoginUser((PrivilegedExceptionAction) () -> {
Review Comment:
Only the JMX call is having issues authenticating, but I agree we should
expect all requests to be made in the same context. I'll expand the scope of
this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5326: YARN-11425. [Federation] Router Supports SubClusterCleaner.
slfan1989 commented on code in PR #5326:
URL: https://github.com/apache/hadoop/pull/5326#discussion_r1091306165
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java:
##
@@ -4299,6 +4299,26 @@ public static boolean isAclEnabled(Configuration conf) {
ROUTER_PREFIX + "interceptor.allow-partial-result.enable";
public static final boolean
DEFAULT_ROUTER_INTERCEPTOR_ALLOW_PARTIAL_RESULT_ENABLED = false;
+ /** Router SubCluster Cleaner Thread Clean Interval Time. **/
+ public static final String ROUTER_SUBCLUSTER_CLEANER_INTERVAL_TIME =
+ ROUTER_PREFIX + "subcluster.cleaner.interval.time";
+ public static final long DEFAULT_ROUTER_SUBCLUSTER_CLEANER_INTERVAL_TIME =
6;
Review Comment:
Thanks for your suggestion, I will modify the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5321: YARN-11424. [Federation] Router Supports DeregisterSubCluster.
slfan1989 commented on code in PR #5321:
URL: https://github.com/apache/hadoop/pull/5321#discussion_r1091305761
##
hadoop-yarn-project/hadoop-yarn/bin/yarn:
##
@@ -180,6 +180,9 @@ ${HADOOP_COMMON_HOME}/${HADOOP_COMMON_LIB_JARS_DIR}"
HADOOP_HEAPSIZE_MAX="${YARN_ROUTER_HEAPSIZE}"
fi
;;
+routeradmin)
+ HADOOP_CLASSNAME='org.apache.hadoop.yarn.client.cli.RouterCLI'
Review Comment:
Thank you very much for your help in reviewing the code, I agree with your
suggestion, I will split this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] cnauroth commented on pull request #4237: MAPREDUCE-7375 JobSubmissionFiles don't set right permission after mkdirs
cnauroth commented on PR #4237: URL: https://github.com/apache/hadoop/pull/4237#issuecomment-1409558905 @steveloughran , no problem, thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5324: HDFS-16895. NamenodeHeartbeatService should use credentials of logged…
hadoop-yetus commented on PR #5324:
URL: https://github.com/apache/hadoop/pull/5324#issuecomment-1409550402
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 49s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 8s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 35s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 41s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 46s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 52s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 29s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | compile | 0m 30s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 33s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 48s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 41m 18s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5324/3/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 154m 37s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterRPCMultipleDestinationMountTableResolver
|
| | hadoop.hdfs.server.federation.router.TestRouterNamenodeHeartbeat |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5324/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5324 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 018a3b30aebb 4.15.0-197-generic #208-Ubuntu SMP Tue Nov 1
17:23:37 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5e453289e9ecd2811251aa3461006d6ce335f218 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5324/3/testReport/ |
| Max. process+thread count | 2101 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5324/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus
[GitHub] [hadoop] goiri commented on a diff in pull request #5321: YARN-11424. [Federation] Router Supports DeregisterSubCluster.
goiri commented on code in PR #5321:
URL: https://github.com/apache/hadoop/pull/5321#discussion_r1091290660
##
hadoop-yarn-project/hadoop-yarn/bin/yarn:
##
@@ -180,6 +180,9 @@ ${HADOOP_COMMON_HOME}/${HADOOP_COMMON_LIB_JARS_DIR}"
HADOOP_HEAPSIZE_MAX="${YARN_ROUTER_HEAPSIZE}"
fi
;;
+routeradmin)
+ HADOOP_CLASSNAME='org.apache.hadoop.yarn.client.cli.RouterCLI'
Review Comment:
This PR adds a router admin CLI and a method to deregister methods.
Can we split it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] omalley commented on a diff in pull request #5324: HDFS-16895. NamenodeHeartbeatService should use credentials of logged…
omalley commented on code in PR #5324:
URL: https://github.com/apache/hadoop/pull/5324#discussion_r1091288711
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/NamenodeHeartbeatService.java:
##
@@ -339,8 +341,11 @@ protected NamenodeStatusReport getNamenodeStatusReport() {
// should be required at some point for QoS
updateSafeModeParameters(serviceURI, report);
- // Read the stats from JMX (optional)
- updateJMXParameters(webAddress, report);
+ // Read the stats from JMX (optional) using the login user credentials
+ SecurityUtil.doAsLoginUser((PrivilegedExceptionAction) () -> {
Review Comment:
It feels like we should do this at a higher level eg.
NamenodeHeartbeatService.periodicInvoke(). That way the other operations, such
as the rpc call to each NN, also run as the login user.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hchaverri commented on a diff in pull request #5324: HDFS-16895. NamenodeHeartbeatService should use credentials of logged…
hchaverri commented on code in PR #5324:
URL: https://github.com/apache/hadoop/pull/5324#discussion_r1091259025
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterNamenodeHeartbeat.java:
##
@@ -318,4 +320,31 @@ private Configuration generateNamenodeConfiguration(
return conf;
}
+
+ @Test
+ public void testNamendodeHeartbeatWithSecurity() throws Exception {
Review Comment:
Fixed. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] simbadzina commented on a diff in pull request #5324: HDFS-16895. NamenodeHeartbeatService should use credentials of logged…
simbadzina commented on code in PR #5324:
URL: https://github.com/apache/hadoop/pull/5324#discussion_r1091153796
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterNamenodeHeartbeat.java:
##
@@ -318,4 +320,31 @@ private Configuration generateNamenodeConfiguration(
return conf;
}
+
+ @Test
+ public void testNamendodeHeartbeatWithSecurity() throws Exception {
Review Comment:
Nit: Typo "namendode" -> "namenode"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] lnbest0707 commented on pull request #5297: YARN-11391. Support using DNS to resolve yarn services to IP addresses.
lnbest0707 commented on PR #5297: URL: https://github.com/apache/hadoop/pull/5297#issuecomment-1409259382 @slfan1989 style issue already fixed, others issues should not be related and not able to be resolved by rebasing. Could you please review and how can I merge the change? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5326: YARN-11425. [Federation] Router Supports SubClusterCleaner.
goiri commented on code in PR #5326:
URL: https://github.com/apache/hadoop/pull/5326#discussion_r1091037486
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/test/java/org/apache/hadoop/yarn/server/router/cleaner/TestSubClusterCleaner.java:
##
@@ -0,0 +1,98 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.yarn.server.router.cleaner;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.test.GenericTestUtils;
+import org.apache.hadoop.yarn.conf.YarnConfiguration;
+import org.apache.hadoop.yarn.exceptions.YarnException;
+import
org.apache.hadoop.yarn.server.federation.store.impl.MemoryFederationStateStore;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterId;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterInfo;
+import
org.apache.hadoop.yarn.server.federation.store.records.SubClusterRegisterRequest;
+import org.apache.hadoop.yarn.server.federation.store.records.SubClusterState;
+import
org.apache.hadoop.yarn.server.federation.utils.FederationStateStoreFacade;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.Map;
+import java.util.concurrent.TimeoutException;
+
+public class TestSubClusterCleaner {
+
+
+ // Router Constants
+
+ private Configuration conf;
+ private MemoryFederationStateStore stateStore;
+ private FederationStateStoreFacade facade;
+ private SubClusterCleaner cleaner;
+
+ @Before
+ public void setup() throws YarnException {
+conf = new YarnConfiguration();
+conf.setLong(YarnConfiguration.ROUTER_SUBCLUSTER_EXPIRATION_TIME, 1000);
+conf.setInt(YarnConfiguration.FEDERATION_CACHE_TIME_TO_LIVE_SECS, 0);
+
+stateStore = new MemoryFederationStateStore();
+stateStore.init(conf);
+
+facade = FederationStateStoreFacade.getInstance();
+facade.reinitialize(stateStore, conf);
+
+cleaner = new SubClusterCleaner(conf);
+for (int i = 0; i < 4; i++){
+ // Create sub cluster id and info
+ SubClusterId subClusterId = SubClusterId.newInstance("SC-" + i);
+ SubClusterInfo subClusterInfo = SubClusterInfo.newInstance(subClusterId,
+ "127.0.0.1:1", "127.0.0.1:2", "127.0.0.1:3", "127.0.0.1:4",
+ SubClusterState.SC_RUNNING, System.currentTimeMillis(), "");
+ // Register the subCluster
+ stateStore.registerSubCluster(
+ SubClusterRegisterRequest.newInstance(subClusterInfo));
+}
+ }
+
+ @Test
+ public void testSubClusterRegisterHeartBeatTime()
+ throws InterruptedException, TimeoutException, YarnException {
+// We set up such a unit test, We set the status of all subClusters to
RUNNING,
+// and set the SubClusterCleaner Check Heartbeat timeout to 1s.
+// After 1s, the status of all subClusters should be SC_LOST.
+// At this time, the size of the Active SubCluster is 0.
+GenericTestUtils.waitFor(() -> {
+ cleaner.run();
+ try {
+int count = facade.getActiveSubClustersCount();
+if(count == 0) {
+ return true;
+}
+ } catch (YarnException e) {
+ }
+ return false;
+}, 1000, 1 * 1000);
Review Comment:
We are not really waiting for anything here, right?
It should be like 10 seconds top?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #5326: YARN-11425. [Federation] Router Supports SubClusterCleaner.
goiri commented on code in PR #5326:
URL: https://github.com/apache/hadoop/pull/5326#discussion_r1091027952
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java:
##
@@ -4299,6 +4299,26 @@ public static boolean isAclEnabled(Configuration conf) {
ROUTER_PREFIX + "interceptor.allow-partial-result.enable";
public static final boolean
DEFAULT_ROUTER_INTERCEPTOR_ALLOW_PARTIAL_RESULT_ENABLED = false;
+ /** Router SubCluster Cleaner Thread Clean Interval Time. **/
+ public static final String ROUTER_SUBCLUSTER_CLEANER_INTERVAL_TIME =
+ ROUTER_PREFIX + "subcluster.cleaner.interval.time";
+ public static final long DEFAULT_ROUTER_SUBCLUSTER_CLEANER_INTERVAL_TIME =
6;
Review Comment:
If we are using getTimeDuration, let's use something easier to read here
like TimeUnit.Seconds.ToMillis(60) or "60s" might also work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[
https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682247#comment-17682247
]
ASF GitHub Bot commented on HADOOP-18206:
-
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1091026861
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java:
##
@@ -256,7 +255,7 @@ public static void skipFully(InputStream in, long len)
throws IOException {
* instead
*/
@Deprecated
- public static void cleanup(Log log, java.io.Closeable... closeables) {
+ public static void cleanup(Logger log, java.io.Closeable... closeables) {
Review Comment:
Updated the javadoc, will create sub-task for HADOOP-16206 to get this
removed.
> Cleanup the commons-logging references in the code base
> ---
>
> Key: HADOOP-18206
> URL: https://issues.apache.org/jira/browse/HADOOP-18206
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Should always use slf4j
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1091026861
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java:
##
@@ -256,7 +255,7 @@ public static void skipFully(InputStream in, long len)
throws IOException {
* instead
*/
@Deprecated
- public static void cleanup(Log log, java.io.Closeable... closeables) {
+ public static void cleanup(Logger log, java.io.Closeable... closeables) {
Review Comment:
Updated the javadoc, will create sub-task for HADOOP-16206 to get this
removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[ https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682245#comment-17682245 ] ASF GitHub Bot commented on HADOOP-18206: - virajjasani commented on code in PR #5315: URL: https://github.com/apache/hadoop/pull/5315#discussion_r1090918735 ## hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java: ## @@ -223,35 +222,6 @@ public synchronized static void printThreadInfo(PrintStream stream, private static long previousLogTime = 0; - /** - * Log the current thread stacks at INFO level. - * @param log the logger that logs the stack trace - * @param title a descriptive title for the call stacks - * @param minInterval the minimum time from the last - */ - public static void logThreadInfo(Log log, Review Comment: Same as above, this can whole work for moving from log4j1 to log4j2 can only be put in trunk (to be released with 4.0.0) and not on 3.x releases. > Cleanup the commons-logging references in the code base > --- > > Key: HADOOP-18206 > URL: https://issues.apache.org/jira/browse/HADOOP-18206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Should always use slf4j -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
virajjasani commented on code in PR #5315: URL: https://github.com/apache/hadoop/pull/5315#discussion_r1090918735 ## hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java: ## @@ -223,35 +222,6 @@ public synchronized static void printThreadInfo(PrintStream stream, private static long previousLogTime = 0; - /** - * Log the current thread stacks at INFO level. - * @param log the logger that logs the stack trace - * @param title a descriptive title for the call stacks - * @param minInterval the minimum time from the last - */ - public static void logThreadInfo(Log log, Review Comment: Same as above, this can whole work for moving from log4j1 to log4j2 can only be put in trunk (to be released with 4.0.0) and not on 3.x releases. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[
https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682243#comment-17682243
]
ASF GitHub Bot commented on HADOOP-18206:
-
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1090916375
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/ServiceOperations.java:
##
@@ -67,26 +66,6 @@ public static Exception stopQuietly(Service service) {
return stopQuietly(LOG, service);
}
- /**
- * Stop a service; if it is null do nothing. Exceptions are caught and
- * logged at warn level. (but not Throwables). This operation is intended to
- * be used in cleanup operations
- *
- * @param log the log to warn at
- * @param service a service; may be null
- * @return any exception that was caught; null if none was.
- * @see ServiceOperations#stopQuietly(Service)
- */
- public static Exception stopQuietly(Log log, Service service) {
Review Comment:
Unfortunately we can only keep this method on trunk branch and not on any of
branch-3.x, because this method directly takes commons logging Log reference
(which we are removing with this patch) and since this class is IA.Public,
making this incompatible change for 3.x releases.
We can move forward with this overall task of only on 4.0.0 release.
cc @Apache9
> Cleanup the commons-logging references in the code base
> ---
>
> Key: HADOOP-18206
> URL: https://issues.apache.org/jira/browse/HADOOP-18206
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Should always use slf4j
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1090916375
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/ServiceOperations.java:
##
@@ -67,26 +66,6 @@ public static Exception stopQuietly(Service service) {
return stopQuietly(LOG, service);
}
- /**
- * Stop a service; if it is null do nothing. Exceptions are caught and
- * logged at warn level. (but not Throwables). This operation is intended to
- * be used in cleanup operations
- *
- * @param log the log to warn at
- * @param service a service; may be null
- * @return any exception that was caught; null if none was.
- * @see ServiceOperations#stopQuietly(Service)
- */
- public static Exception stopQuietly(Log log, Service service) {
Review Comment:
Unfortunately we can only keep this method on trunk branch and not on any of
branch-3.x, because this method directly takes commons logging Log reference
(which we are removing with this patch) and since this class is IA.Public,
making this incompatible change for 3.x releases.
We can move forward with this overall task of only on 4.0.0 release.
cc @Apache9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[ https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682239#comment-17682239 ] ASF GitHub Bot commented on HADOOP-18206: - virajjasani commented on PR #5315: URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1409146994 > is there a way the log4j refs can be kept inside MetricsLoggerTask Done > Cleanup the commons-logging references in the code base > --- > > Key: HADOOP-18206 > URL: https://issues.apache.org/jira/browse/HADOOP-18206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Should always use slf4j -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
virajjasani commented on PR #5315: URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1409146994 > is there a way the log4j refs can be kept inside MetricsLoggerTask Done -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[
https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682235#comment-17682235
]
ASF GitHub Bot commented on HADOOP-18206:
-
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1091002213
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -949,13 +947,13 @@ protected void startMetricsLogger(Configuration conf) {
return;
}
-MetricsLoggerTask.makeMetricsLoggerAsync(MetricsLog);
+MetricsLoggerTask.makeMetricsLoggerAsync(METRICS_LOG);
Review Comment:
makes sense
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -949,13 +947,13 @@ protected void startMetricsLogger(Configuration conf) {
return;
}
-MetricsLoggerTask.makeMetricsLoggerAsync(MetricsLog);
+MetricsLoggerTask.makeMetricsLoggerAsync(METRICS_LOG);
// Schedule the periodic logging.
metricsLoggerTimer = new ScheduledThreadPoolExecutor(1);
metricsLoggerTimer.setExecuteExistingDelayedTasksAfterShutdownPolicy(
false);
-metricsLoggerTimer.scheduleWithFixedDelay(new MetricsLoggerTask(MetricsLog,
+metricsLoggerTimer.scheduleWithFixedDelay(new
MetricsLoggerTask(METRICS_LOG,
Review Comment:
sounds good.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:
##
@@ -360,7 +358,8 @@ public class DataNode extends ReconfigurableBase
FS_GETSPACEUSED_JITTER_KEY,
FS_GETSPACEUSED_CLASSNAME));
- public static final Log METRICS_LOG =
LogFactory.getLog("DataNodeMetricsLog");
+ public static final org.apache.log4j.Logger METRICS_LOG =
+ org.apache.log4j.Logger.getLogger("DataNodeMetricsLog");
Review Comment:
Yes that sounds good. Let me do that.
> Cleanup the commons-logging references in the code base
> ---
>
> Key: HADOOP-18206
> URL: https://issues.apache.org/jira/browse/HADOOP-18206
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Should always use slf4j
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1091002213
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -949,13 +947,13 @@ protected void startMetricsLogger(Configuration conf) {
return;
}
-MetricsLoggerTask.makeMetricsLoggerAsync(MetricsLog);
+MetricsLoggerTask.makeMetricsLoggerAsync(METRICS_LOG);
Review Comment:
makes sense
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -949,13 +947,13 @@ protected void startMetricsLogger(Configuration conf) {
return;
}
-MetricsLoggerTask.makeMetricsLoggerAsync(MetricsLog);
+MetricsLoggerTask.makeMetricsLoggerAsync(METRICS_LOG);
// Schedule the periodic logging.
metricsLoggerTimer = new ScheduledThreadPoolExecutor(1);
metricsLoggerTimer.setExecuteExistingDelayedTasksAfterShutdownPolicy(
false);
-metricsLoggerTimer.scheduleWithFixedDelay(new MetricsLoggerTask(MetricsLog,
+metricsLoggerTimer.scheduleWithFixedDelay(new
MetricsLoggerTask(METRICS_LOG,
Review Comment:
sounds good.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:
##
@@ -360,7 +358,8 @@ public class DataNode extends ReconfigurableBase
FS_GETSPACEUSED_JITTER_KEY,
FS_GETSPACEUSED_CLASSNAME));
- public static final Log METRICS_LOG =
LogFactory.getLog("DataNodeMetricsLog");
+ public static final org.apache.log4j.Logger METRICS_LOG =
+ org.apache.log4j.Logger.getLogger("DataNodeMetricsLog");
Review Comment:
Yes that sounds good. Let me do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[
https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682229#comment-17682229
]
ASF GitHub Bot commented on HADOOP-18206:
-
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1090989404
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -8853,16 +8849,12 @@ public void logAuditEvent(boolean succeeded, String
userName,
}
public void logAuditMessage(String message) {
- auditLog.info(message);
+ AUDIT_LOG.info(message);
}
}
private static void enableAsyncAuditLog(Configuration conf) {
-if (!(auditLog instanceof Log4JLogger)) {
Review Comment:
I agree for this patch, but we do have sub-tasks coming up that would:
- Move dynamic log4j appenders to log4j.properties and let users update
log4j.properties rather than let the code use log4j appenders
- Replace LOG4J references from production code to SLF4J
Also, if downstream application excludes log4j references, hdfs would not
work because in order for hdfs to add dynamic appenders, it needs log4j in the
classpath at least.
And these references are on the server side only.
> Cleanup the commons-logging references in the code base
> ---
>
> Key: HADOOP-18206
> URL: https://issues.apache.org/jira/browse/HADOOP-18206
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Should always use slf4j
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1090989404
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -8853,16 +8849,12 @@ public void logAuditEvent(boolean succeeded, String
userName,
}
public void logAuditMessage(String message) {
- auditLog.info(message);
+ AUDIT_LOG.info(message);
}
}
private static void enableAsyncAuditLog(Configuration conf) {
-if (!(auditLog instanceof Log4JLogger)) {
Review Comment:
I agree for this patch, but we do have sub-tasks coming up that would:
- Move dynamic log4j appenders to log4j.properties and let users update
log4j.properties rather than let the code use log4j appenders
- Replace LOG4J references from production code to SLF4J
Also, if downstream application excludes log4j references, hdfs would not
work because in order for hdfs to add dynamic appenders, it needs log4j in the
classpath at least.
And these references are on the server side only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[
https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682228#comment-17682228
]
ASF GitHub Bot commented on HADOOP-18206:
-
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1090991860
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -8853,16 +8849,12 @@ public void logAuditEvent(boolean succeeded, String
userName,
}
public void logAuditMessage(String message) {
- auditLog.info(message);
+ AUDIT_LOG.info(message);
}
}
private static void enableAsyncAuditLog(Configuration conf) {
-if (!(auditLog instanceof Log4JLogger)) {
Review Comment:
Also, if downstream application excludes log4j references, hdfs would not
work because in order for hdfs to add dynamic appenders, it needs log4j in the
classpath at least.
And these references are on the server side only.
> Cleanup the commons-logging references in the code base
> ---
>
> Key: HADOOP-18206
> URL: https://issues.apache.org/jira/browse/HADOOP-18206
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Should always use slf4j
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1090991860
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -8853,16 +8849,12 @@ public void logAuditEvent(boolean succeeded, String
userName,
}
public void logAuditMessage(String message) {
- auditLog.info(message);
+ AUDIT_LOG.info(message);
}
}
private static void enableAsyncAuditLog(Configuration conf) {
-if (!(auditLog instanceof Log4JLogger)) {
Review Comment:
Also, if downstream application excludes log4j references, hdfs would not
work because in order for hdfs to add dynamic appenders, it needs log4j in the
classpath at least.
And these references are on the server side only.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[
https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682226#comment-17682226
]
ASF GitHub Bot commented on HADOOP-18206:
-
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1090989404
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -8853,16 +8849,12 @@ public void logAuditEvent(boolean succeeded, String
userName,
}
public void logAuditMessage(String message) {
- auditLog.info(message);
+ AUDIT_LOG.info(message);
}
}
private static void enableAsyncAuditLog(Configuration conf) {
-if (!(auditLog instanceof Log4JLogger)) {
Review Comment:
I agree for this patch, but we do have sub-tasks coming up that would:
- Move dynamic log4j appenders to log4j.properties and let users update
log4j.properties rather than let the code use log4j appenders
- Replace LOG4J references from production code to SLF4J
> Cleanup the commons-logging references in the code base
> ---
>
> Key: HADOOP-18206
> URL: https://issues.apache.org/jira/browse/HADOOP-18206
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Should always use slf4j
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1090989404
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -8853,16 +8849,12 @@ public void logAuditEvent(boolean succeeded, String
userName,
}
public void logAuditMessage(String message) {
- auditLog.info(message);
+ AUDIT_LOG.info(message);
}
}
private static void enableAsyncAuditLog(Configuration conf) {
-if (!(auditLog instanceof Log4JLogger)) {
Review Comment:
I agree for this patch, but we do have sub-tasks coming up that would:
- Move dynamic log4j appenders to log4j.properties and let users update
log4j.properties rather than let the code use log4j appenders
- Replace LOG4J references from production code to SLF4J
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18235) vulnerability: we may leak sensitive information in LocalKeyStoreProvider
[
https://issues.apache.org/jira/browse/HADOOP-18235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682223#comment-17682223
]
ASF GitHub Bot commented on HADOOP-18235:
-
steveloughran commented on code in PR #4998:
URL: https://github.com/apache/hadoop/pull/4998#discussion_r1090918223
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalKeyStoreProvider.java:
##
@@ -142,20 +142,26 @@ protected void initFileSystem(URI uri)
@Override
public void flush() throws IOException {
-super.flush();
-if (LOG.isDebugEnabled()) {
- LOG.debug("Resetting permissions to '" + permissions + "'");
-}
-if (!Shell.WINDOWS) {
- Files.setPosixFilePermissions(Paths.get(file.getCanonicalPath()),
- permissions);
-} else {
- // FsPermission expects a 10-character string because of the leading
- // directory indicator, i.e. "drwx--". The JDK toString method
returns
- // a 9-character string, so prepend a leading character.
- FsPermission fsPermission = FsPermission.valueOf(
- "-" + PosixFilePermissions.toString(permissions));
- FileUtil.setPermission(file, fsPermission);
+super.getWriteLock().lock();
Review Comment:
no need for the super. prefix here
but: we do now require lock() to be reentrant
> vulnerability: we may leak sensitive information in LocalKeyStoreProvider
> --
>
> Key: HADOOP-18235
> URL: https://issues.apache.org/jira/browse/HADOOP-18235
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: lujie
>Assignee: Clay B.
>Priority: Critical
> Labels: pull-request-available
>
> Currently, we implement flush like:
> {code:java}
> // public void flush() throws IOException {
> super.flush();
> if (LOG.isDebugEnabled()) {
> LOG.debug("Resetting permissions to '" + permissions + "'");
> }
> if (!Shell.WINDOWS) {
> Files.setPosixFilePermissions(Paths.get(file.getCanonicalPath()),
> permissions);
> } else {
> // FsPermission expects a 10-character string because of the leading
> // directory indicator, i.e. "drwx--". The JDK toString method
> returns
> // a 9-character string, so prepend a leading character.
> FsPermission fsPermission = FsPermission.valueOf(
> "-" + PosixFilePermissions.toString(permissions));
> FileUtil.setPermission(file, fsPermission);
> }
> } {code}
> we wirite the Credential first, then set permission.
> The correct order is setPermission first, then write Credential .
> Otherswise, we may leak Credential . For example, the origin perms of file is
> 755(default on linux), when the Credential is flushed, Credential can be
> leaked when
>
> 1)between flush and setPermission, others have a chance to access the file.
> 2) CredentialShell(or the machine node ) crash between flush and
> setPermission, the file permission is 755 for ever before we run the
> CredentialShell again.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #4998: HADOOP-18235. vulnerability: we may leak sensitive information in LocalKeyStoreProvider
steveloughran commented on code in PR #4998:
URL: https://github.com/apache/hadoop/pull/4998#discussion_r1090918223
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalKeyStoreProvider.java:
##
@@ -142,20 +142,26 @@ protected void initFileSystem(URI uri)
@Override
public void flush() throws IOException {
-super.flush();
-if (LOG.isDebugEnabled()) {
- LOG.debug("Resetting permissions to '" + permissions + "'");
-}
-if (!Shell.WINDOWS) {
- Files.setPosixFilePermissions(Paths.get(file.getCanonicalPath()),
- permissions);
-} else {
- // FsPermission expects a 10-character string because of the leading
- // directory indicator, i.e. "drwx--". The JDK toString method
returns
- // a 9-character string, so prepend a leading character.
- FsPermission fsPermission = FsPermission.valueOf(
- "-" + PosixFilePermissions.toString(permissions));
- FileUtil.setPermission(file, fsPermission);
+super.getWriteLock().lock();
Review Comment:
no need for the super. prefix here
but: we do now require lock() to be reentrant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[ https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1768#comment-1768 ] ASF GitHub Bot commented on HADOOP-18206: - virajjasani commented on PR #5315: URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1409097037 FWIW, the parent Jira HADOOP-16206 is also marked as "incompatible change" hence I believe we would not be able to backport this whole work to branch-3.x anyways. > Cleanup the commons-logging references in the code base > --- > > Key: HADOOP-18206 > URL: https://issues.apache.org/jira/browse/HADOOP-18206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Should always use slf4j -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
virajjasani commented on PR #5315: URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1409097037 FWIW, the parent Jira HADOOP-16206 is also marked as "incompatible change" hence I believe we would not be able to backport this whole work to branch-3.x anyways. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18592) Sasl connection failure should log remote address
[
https://issues.apache.org/jira/browse/HADOOP-18592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682215#comment-17682215
]
ASF GitHub Bot commented on HADOOP-18592:
-
virajjasani commented on PR #5294:
URL: https://github.com/apache/hadoop/pull/5294#issuecomment-1409075178
Thank you for recent reviews @cnauroth @liuml07
> Sasl connection failure should log remote address
> -
>
> Key: HADOOP-18592
> URL: https://issues.apache.org/jira/browse/HADOOP-18592
> Project: Hadoop Common
> Issue Type: Improvement
>Affects Versions: 3.3.4
>Reporter: Viraj Jasani
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> If Sasl connection fails with some generic error, we miss logging remote
> server that the client was trying to connect to.
> Sample log:
> {code:java}
> 2023-01-12 00:22:28,148 WARN [20%2C1673404849949,1] ipc.Client - Exception
> encountered while connecting to the server
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
> at sun.nio.ch.IOUtil.read(IOUtil.java:197)
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
> at
> org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:141)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
> at java.io.FilterInputStream.read(FilterInputStream.java:133)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at java.io.DataInputStream.readInt(DataInputStream.java:387)
> at org.apache.hadoop.ipc.Client$IpcStreams.readResponse(Client.java:1950)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:367)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:623)
> at org.apache.hadoop.ipc.Client$Connection.access$2300(Client.java:414)
> ...
> ... {code}
> We should log the remote server address.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5294: HADOOP-18592 Sasl connection failure should log remote address
virajjasani commented on PR #5294: URL: https://github.com/apache/hadoop/pull/5294#issuecomment-1409075178 Thank you for recent reviews @cnauroth @liuml07 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[
https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682196#comment-17682196
]
ASF GitHub Bot commented on HADOOP-18206:
-
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1090918735
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java:
##
@@ -223,35 +222,6 @@ public synchronized static void
printThreadInfo(PrintStream stream,
private static long previousLogTime = 0;
- /**
- * Log the current thread stacks at INFO level.
- * @param log the logger that logs the stack trace
- * @param title a descriptive title for the call stacks
- * @param minInterval the minimum time from the last
- */
- public static void logThreadInfo(Log log,
Review Comment:
Same as above, this can whole work behind log4j2 can only be in 4.0.0 if we
follow compatibility guidelines.
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/ServiceOperations.java:
##
@@ -67,26 +66,6 @@ public static Exception stopQuietly(Service service) {
return stopQuietly(LOG, service);
}
- /**
- * Stop a service; if it is null do nothing. Exceptions are caught and
- * logged at warn level. (but not Throwables). This operation is intended to
- * be used in cleanup operations
- *
- * @param log the log to warn at
- * @param service a service; may be null
- * @return any exception that was caught; null if none was.
- * @see ServiceOperations#stopQuietly(Service)
- */
- public static Exception stopQuietly(Log log, Service service) {
Review Comment:
Unfortunately we can't keep this method on trunk branch, because this method
directly takes commons logging Log reference :(
And since this class is IA.Public, this change can only be taken up in trunk
branch only and not on branch-3.3 for now.
We can move forward with log4j2 only on 4.0.0 release (something similar to
hbase). FYI @Apache9
> Cleanup the commons-logging references in the code base
> ---
>
> Key: HADOOP-18206
> URL: https://issues.apache.org/jira/browse/HADOOP-18206
> Project: Hadoop Common
> Issue Type: Sub-task
>Reporter: Duo Zhang
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Should always use slf4j
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
virajjasani commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1090918735
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java:
##
@@ -223,35 +222,6 @@ public synchronized static void
printThreadInfo(PrintStream stream,
private static long previousLogTime = 0;
- /**
- * Log the current thread stacks at INFO level.
- * @param log the logger that logs the stack trace
- * @param title a descriptive title for the call stacks
- * @param minInterval the minimum time from the last
- */
- public static void logThreadInfo(Log log,
Review Comment:
Same as above, this can whole work behind log4j2 can only be in 4.0.0 if we
follow compatibility guidelines.
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/ServiceOperations.java:
##
@@ -67,26 +66,6 @@ public static Exception stopQuietly(Service service) {
return stopQuietly(LOG, service);
}
- /**
- * Stop a service; if it is null do nothing. Exceptions are caught and
- * logged at warn level. (but not Throwables). This operation is intended to
- * be used in cleanup operations
- *
- * @param log the log to warn at
- * @param service a service; may be null
- * @return any exception that was caught; null if none was.
- * @see ServiceOperations#stopQuietly(Service)
- */
- public static Exception stopQuietly(Log log, Service service) {
Review Comment:
Unfortunately we can't keep this method on trunk branch, because this method
directly takes commons logging Log reference :(
And since this class is IA.Public, this change can only be taken up in trunk
branch only and not on branch-3.3 for now.
We can move forward with log4j2 only on 4.0.0 release (something similar to
hbase). FYI @Apache9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18235) vulnerability: we may leak sensitive information in LocalKeyStoreProvider
[
https://issues.apache.org/jira/browse/HADOOP-18235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682195#comment-17682195
]
ASF GitHub Bot commented on HADOOP-18235:
-
steveloughran commented on code in PR #4998:
URL: https://github.com/apache/hadoop/pull/4998#discussion_r1090918223
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalKeyStoreProvider.java:
##
@@ -142,20 +142,26 @@ protected void initFileSystem(URI uri)
@Override
public void flush() throws IOException {
-super.flush();
-if (LOG.isDebugEnabled()) {
- LOG.debug("Resetting permissions to '" + permissions + "'");
-}
-if (!Shell.WINDOWS) {
- Files.setPosixFilePermissions(Paths.get(file.getCanonicalPath()),
- permissions);
-} else {
- // FsPermission expects a 10-character string because of the leading
- // directory indicator, i.e. "drwx--". The JDK toString method
returns
- // a 9-character string, so prepend a leading character.
- FsPermission fsPermission = FsPermission.valueOf(
- "-" + PosixFilePermissions.toString(permissions));
- FileUtil.setPermission(file, fsPermission);
+super.getWriteLock().lock();
Review Comment:
no need for the super. prefix here
> vulnerability: we may leak sensitive information in LocalKeyStoreProvider
> --
>
> Key: HADOOP-18235
> URL: https://issues.apache.org/jira/browse/HADOOP-18235
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: lujie
>Assignee: Clay B.
>Priority: Critical
> Labels: pull-request-available
>
> Currently, we implement flush like:
> {code:java}
> // public void flush() throws IOException {
> super.flush();
> if (LOG.isDebugEnabled()) {
> LOG.debug("Resetting permissions to '" + permissions + "'");
> }
> if (!Shell.WINDOWS) {
> Files.setPosixFilePermissions(Paths.get(file.getCanonicalPath()),
> permissions);
> } else {
> // FsPermission expects a 10-character string because of the leading
> // directory indicator, i.e. "drwx--". The JDK toString method
> returns
> // a 9-character string, so prepend a leading character.
> FsPermission fsPermission = FsPermission.valueOf(
> "-" + PosixFilePermissions.toString(permissions));
> FileUtil.setPermission(file, fsPermission);
> }
> } {code}
> we wirite the Credential first, then set permission.
> The correct order is setPermission first, then write Credential .
> Otherswise, we may leak Credential . For example, the origin perms of file is
> 755(default on linux), when the Credential is flushed, Credential can be
> leaked when
>
> 1)between flush and setPermission, others have a chance to access the file.
> 2) CredentialShell(or the machine node ) crash between flush and
> setPermission, the file permission is 755 for ever before we run the
> CredentialShell again.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #4998: HADOOP-18235. vulnerability: we may leak sensitive information in LocalKeyStoreProvider
steveloughran commented on code in PR #4998:
URL: https://github.com/apache/hadoop/pull/4998#discussion_r1090918223
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/alias/LocalKeyStoreProvider.java:
##
@@ -142,20 +142,26 @@ protected void initFileSystem(URI uri)
@Override
public void flush() throws IOException {
-super.flush();
-if (LOG.isDebugEnabled()) {
- LOG.debug("Resetting permissions to '" + permissions + "'");
-}
-if (!Shell.WINDOWS) {
- Files.setPosixFilePermissions(Paths.get(file.getCanonicalPath()),
- permissions);
-} else {
- // FsPermission expects a 10-character string because of the leading
- // directory indicator, i.e. "drwx--". The JDK toString method
returns
- // a 9-character string, so prepend a leading character.
- FsPermission fsPermission = FsPermission.valueOf(
- "-" + PosixFilePermissions.toString(permissions));
- FileUtil.setPermission(file, fsPermission);
+super.getWriteLock().lock();
Review Comment:
no need for the super. prefix here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18584) [NFS GW] Fix regression after netty4 migration
[ https://issues.apache.org/jira/browse/HADOOP-18584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682194#comment-17682194 ] ASF GitHub Bot commented on HADOOP-18584: - jojochuang merged PR #5252: URL: https://github.com/apache/hadoop/pull/5252 > [NFS GW] Fix regression after netty4 migration > -- > > Key: HADOOP-18584 > URL: https://issues.apache.org/jira/browse/HADOOP-18584 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 3.4.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Labels: pull-request-available > > Two bugs were found recently. Both related to hadoop portmap utility. > (1) The ChannelHandler that can be added to a ChannelPipeline in Netty4 must > be Shareable. However, IdleStateHandler, used by Portmap is not. We need to > fix it otherwise hadoop portmap may fail to start. > (2) RpcUdpResponseStage does not get the request from client (hadoop mountd) > because of response type mismatch, and therefore it does not respond back to > client request. > This is a regression from HADOOP-11245 but I consider it not a blocker, > because most recent Linux distro supports rpcbind and there's little reason > to use Hadoop's portmap. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang merged pull request #5252: HADOOP-18584. [NFS GW] Fix regression after netty4 migration.
jojochuang merged PR #5252: URL: https://github.com/apache/hadoop/pull/5252 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[ https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682191#comment-17682191 ] ASF GitHub Bot commented on HADOOP-18206: - virajjasani commented on PR #5315: URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1409002069 Moreover, the main goal of this sub-task is to remove only commons-logging references hence I think it would be better to replace Datanode/Namenode async logger with SLF4J only (with log4j.properties including async logger). > If those log4j refs can be hidden entirely within MetricsLoggerTask then some reflection games could be done there perhaps. We would have to pass in the name of the logger and let that class do any reflection-based look up of the actual log. We have sub-task to introduce hadoop-logging module altogether. The plan is to move all log4j appender and other logic to that module and have reflection do the work. All hadoop modules to only extend hadoop-logging module. But this sub-task is only related to removal of commons-logging. And places in hdfs would not be functional even as of today (without this patch) if client doesn't include Log4J in the classpath (because dynamically adding appender is part of hdfs own logic). > Cleanup the commons-logging references in the code base > --- > > Key: HADOOP-18206 > URL: https://issues.apache.org/jira/browse/HADOOP-18206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Should always use slf4j -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
virajjasani commented on PR #5315: URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1409002069 Moreover, the main goal of this sub-task is to remove only commons-logging references hence I think it would be better to replace Datanode/Namenode async logger with SLF4J only (with log4j.properties including async logger). > If those log4j refs can be hidden entirely within MetricsLoggerTask then some reflection games could be done there perhaps. We would have to pass in the name of the logger and let that class do any reflection-based look up of the actual log. We have sub-task to introduce hadoop-logging module altogether. The plan is to move all log4j appender and other logic to that module and have reflection do the work. All hadoop modules to only extend hadoop-logging module. But this sub-task is only related to removal of commons-logging. And places in hdfs would not be functional even as of today (without this patch) if client doesn't include Log4J in the classpath (because dynamically adding appender is part of hdfs own logic). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[ https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682187#comment-17682187 ] ASF GitHub Bot commented on HADOOP-18206: - virajjasani commented on PR #5315: URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1408995424 > I've reviewed where log4j gets used in the production code. I do worry that today there's enough reflection that if deployed with logback all is good, but now log4j is required. Even without this patch, if we don't have log4j in classpath, hdfs cannot come up right? For instance, `MetricsLoggerTask` is adding appenders dynamically and for that, we do need appender classes to be present in the classpath, meaning log4j is mandatory in the classpath. However, I am going to cover all dynamic async appenders and replace with log4j properties, then we can directly use SLF4J and remove all Log4J references in the codebase. Does this sound good? > Cleanup the commons-logging references in the code base > --- > > Key: HADOOP-18206 > URL: https://issues.apache.org/jira/browse/HADOOP-18206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Should always use slf4j -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
virajjasani commented on PR #5315: URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1408995424 > I've reviewed where log4j gets used in the production code. I do worry that today there's enough reflection that if deployed with logback all is good, but now log4j is required. Even without this patch, if we don't have log4j in classpath, hdfs cannot come up right? For instance, `MetricsLoggerTask` is adding appenders dynamically and for that, we do need appender classes to be present in the classpath, meaning log4j is mandatory in the classpath. However, I am going to cover all dynamic async appenders and replace with log4j properties, then we can directly use SLF4J and remove all Log4J references in the codebase. Does this sound good? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4237: MAPREDUCE-7375 JobSubmissionFiles don't set right permission after mkdirs
steveloughran commented on PR #4237: URL: https://github.com/apache/hadoop/pull/4237#issuecomment-1408981404 sorry, @cnauroth , misread your comment and put it in -i don't think it is worth reverting now -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5330: HDFS-16898. Make write lock fine-grain in method processCommandFromActor
hadoop-yetus commented on PR #5330:
URL: https://github.com/apache/hadoop/pull/5330#issuecomment-1408966217
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 2s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 2s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 1s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 1m 21s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 6s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 26s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 40s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 12s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 1m 12s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 50s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 18s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 50s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 291m 26s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5330/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 49s | | The patch does not
generate ASF License warnings. |
| | | 409m 7s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestLeaseRecovery2 |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5330/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5330 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux d867773c0aa6 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 81cc655288a46ad4dd08392fdf8813281f9c6375 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5330/2/testReport/ |
| Max. process+thread count | 3286 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5330/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0
[GitHub] [hadoop] steveloughran commented on pull request #4237: MAPREDUCE-7375 JobSubmissionFiles don't set right permission after mkdirs
steveloughran commented on PR #4237: URL: https://github.com/apache/hadoop/pull/4237#issuecomment-1408966067 okay, let's cherrypick. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5321: YARN-11424. [Federation] Router Supports DeregisterSubCluster.
slfan1989 commented on PR #5321: URL: https://github.com/apache/hadoop/pull/5321#issuecomment-1408929903 @goiri Can you help review this pr? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18596) Distcp -update between different cloud stores to use modification time while checking for file skip.
[
https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682172#comment-17682172
]
ASF GitHub Bot commented on HADOOP-18596:
-
steveloughran commented on code in PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#discussion_r1090843134
##
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java:
##
@@ -142,6 +142,13 @@ private DistCpConstants() {
"distcp.blocks.per.chunk";
public static final String CONF_LABEL_USE_ITERATOR = "distcp.use.iterator";
+
+ /** Distcp -update to use modification time of source and target file to
+ * check while skipping.
+ */
+ public static final String CONF_LABEL_UPDATE_MOD_TIME =
Review Comment:
going to need docs.
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -857,4 +862,83 @@ public void testDistCpWithUpdateExistFile() throws
Exception {
verifyFileContents(localFS, dest, block);
}
+ @Test
+ public void testDistCpUpdateCheckFileSkip() throws Exception {
Review Comment:
I'm thinking of a way to test that 0 byte files don't get copied
the testUpdateDeepDirectoryStructureNoChange() test shows how the counters
are used for validation. The new test should validate the files are skipped as
well as checking the contents.
That should be usable to verify that 0 byte files are always skipped,
something we can't do with content validation.
one thing to be aware of is that this test suite isn't implemented by hdfs,
because of the way it creates a new fs every call is too slow. there should be
some specific hdfs to local test we should cover too.
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -857,4 +862,83 @@ public void testDistCpWithUpdateExistFile() throws
Exception {
verifyFileContents(localFS, dest, block);
}
+ @Test
+ public void testDistCpUpdateCheckFileSkip() throws Exception {
+describe("Distcp update to check file skips.");
+
+Path source = new Path(remoteDir, "file");
+Path dest = new Path(localDir, "file");
+dest = localFS.makeQualified(dest);
+
+// Creating a source file with certain dataset.
+byte[] sourceBlock = dataset(10, 'a', 'z');
+
+// Write the dataset and as well create the target path.
+try (FSDataOutputStream out = remoteFS.create(source)) {
Review Comment:
if you use `ContractTestUtils.writeDataset()` here the write is followed by
the check that the file is of the correct length; L882 just looks for existence
> Distcp -update between different cloud stores to use modification time while
> checking for file skip.
>
>
> Key: HADOOP-18596
> URL: https://issues.apache.org/jira/browse/HADOOP-18596
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools/distcp
>Reporter: Mehakmeet Singh
>Assignee: Mehakmeet Singh
>Priority: Major
> Labels: pull-request-available
>
> Distcp -update currently relies on File size, block size, and Checksum
> comparisons to figure out which files should be skipped or copied.
> Since different cloud stores have different checksum algorithms we should
> check for modification time as well to the checks.
> This would ensure that while performing -update if the files are perceived to
> be out of sync we should copy them. The machines between which the file
> transfers occur should be in time sync to avoid any extra copies.
> Improving testing and documentation for modification time checks between
> different object stores to ensure no incorrect skipping of files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5308: HADOOP-18596. Distcp -update to use modification time while checking for file skip.
steveloughran commented on code in PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#discussion_r1090843134
##
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCpConstants.java:
##
@@ -142,6 +142,13 @@ private DistCpConstants() {
"distcp.blocks.per.chunk";
public static final String CONF_LABEL_USE_ITERATOR = "distcp.use.iterator";
+
+ /** Distcp -update to use modification time of source and target file to
+ * check while skipping.
+ */
+ public static final String CONF_LABEL_UPDATE_MOD_TIME =
Review Comment:
going to need docs.
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -857,4 +862,83 @@ public void testDistCpWithUpdateExistFile() throws
Exception {
verifyFileContents(localFS, dest, block);
}
+ @Test
+ public void testDistCpUpdateCheckFileSkip() throws Exception {
Review Comment:
I'm thinking of a way to test that 0 byte files don't get copied
the testUpdateDeepDirectoryStructureNoChange() test shows how the counters
are used for validation. The new test should validate the files are skipped as
well as checking the contents.
That should be usable to verify that 0 byte files are always skipped,
something we can't do with content validation.
one thing to be aware of is that this test suite isn't implemented by hdfs,
because of the way it creates a new fs every call is too slow. there should be
some specific hdfs to local test we should cover too.
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/contract/AbstractContractDistCpTest.java:
##
@@ -857,4 +862,83 @@ public void testDistCpWithUpdateExistFile() throws
Exception {
verifyFileContents(localFS, dest, block);
}
+ @Test
+ public void testDistCpUpdateCheckFileSkip() throws Exception {
+describe("Distcp update to check file skips.");
+
+Path source = new Path(remoteDir, "file");
+Path dest = new Path(localDir, "file");
+dest = localFS.makeQualified(dest);
+
+// Creating a source file with certain dataset.
+byte[] sourceBlock = dataset(10, 'a', 'z');
+
+// Write the dataset and as well create the target path.
+try (FSDataOutputStream out = remoteFS.create(source)) {
Review Comment:
if you use `ContractTestUtils.writeDataset()` here the write is followed by
the check that the file is of the correct length; L882 just looks for existence
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #5326: YARN-11425. [Federation] Router Supports SubClusterCleaner.
slfan1989 commented on PR #5326: URL: https://github.com/apache/hadoop/pull/5326#issuecomment-1408928125 @goiri Can you help to review this PR again? Thank you very much! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18596) Distcp -update between different cloud stores to use modification time while checking for file skip.
[ https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18596: Target Version/s: 3.3.9 (was: site) > Distcp -update between different cloud stores to use modification time while > checking for file skip. > > > Key: HADOOP-18596 > URL: https://issues.apache.org/jira/browse/HADOOP-18596 > Project: Hadoop Common > Issue Type: Improvement > Components: tools/distcp >Reporter: Mehakmeet Singh >Assignee: Mehakmeet Singh >Priority: Major > Labels: pull-request-available > > Distcp -update currently relies on File size, block size, and Checksum > comparisons to figure out which files should be skipped or copied. > Since different cloud stores have different checksum algorithms we should > check for modification time as well to the checks. > This would ensure that while performing -update if the files are perceived to > be out of sync we should copy them. The machines between which the file > transfers occur should be in time sync to avoid any extra copies. > Improving testing and documentation for modification time checks between > different object stores to ensure no incorrect skipping of files. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18601) Fix build failure with docs profile
[
https://issues.apache.org/jira/browse/HADOOP-18601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682155#comment-17682155
]
ASF GitHub Bot commented on HADOOP-18601:
-
hadoop-yetus commented on PR #5331:
URL: https://github.com/apache/hadoop/pull/5331#issuecomment-1408864757
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 46s | | trunk passed |
| +1 :green_heart: | compile | 23m 4s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 21m 31s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | mvnsite | 2m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 58s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 31s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 121m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 52s | | the patch passed |
| +1 :green_heart: | compile | 22m 16s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 22m 16s | | the patch passed |
| +1 :green_heart: | compile | 20m 23s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 20m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 2m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 54s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 32s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 32m 30s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 321m 19s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5331/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 0m 47s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-documentstore.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5331/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-documentstore.txt)
| hadoop-yarn-server-timelineservice-documentstore in the patch failed. |
| +0 :ok: | asflicense | 0m 47s | | ASF License check generated no
output? |
| | | 525m 34s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.TestDFSStorageStateRecovery |
| | hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.TestDeadNodeDetection |
| | hadoop.hdfs.TestErasureCodingMultipleRacks |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.TestMissingBlocksAlert |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5331/1/artifact/out/Dockerfile
|
| GITHUB PR |
[jira] [Commented] (HADOOP-11452) Make FileSystem.rename(path, path, options) public, specified, tested
[
https://issues.apache.org/jira/browse/HADOOP-11452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682156#comment-17682156
]
ASF GitHub Bot commented on HADOOP-11452:
-
hadoop-yetus commented on PR #2735:
URL: https://github.com/apache/hadoop/pull/2735#issuecomment-1408865087
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 9 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 31s | | trunk passed |
| +1 :green_heart: | compile | 25m 14s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 22m 58s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 25s | | trunk passed |
| -1 :x: | javadoc | 1m 10s |
[/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-common in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 4m 52s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 12m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| -1 :x: | mvninstall | 0m 51s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | mvninstall | 0m 18s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 44s |
[/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | javac | 0m 44s |
[/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | compile | 0m 40s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | javac | 0m 40s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/blanks-eol.txt)
| The patch has 7 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer
[GitHub] [hadoop] hadoop-yetus commented on pull request #2735: HADOOP-11452 make rename/3 public
hadoop-yetus commented on PR #2735:
URL: https://github.com/apache/hadoop/pull/2735#issuecomment-1408865087
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 0s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 9 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 31s | | trunk passed |
| +1 :green_heart: | compile | 25m 14s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 22m 58s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 4m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 25s | | trunk passed |
| -1 :x: | javadoc | 1m 10s |
[/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-common in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 4m 52s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 12m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 48s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| -1 :x: | mvninstall | 0m 51s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | mvninstall | 0m 18s |
[/patch-mvninstall-hadoop-tools_hadoop-aws.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/patch-mvninstall-hadoop-tools_hadoop-aws.txt)
| hadoop-aws in the patch failed. |
| -1 :x: | compile | 0m 44s |
[/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | javac | 0m 44s |
[/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/patch-compile-root-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| root in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | compile | 0m 40s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | javac | 0m 40s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| root in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/blanks-eol.txt)
| The patch has 7 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 3m 35s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2735/3/artifact/out/results-checkstyle-root.txt)
| root: The patch
[GitHub] [hadoop] hadoop-yetus commented on pull request #5331: HADOOP-18601. Fix build failure with docs profile.
hadoop-yetus commented on PR #5331:
URL: https://github.com/apache/hadoop/pull/5331#issuecomment-1408864757
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 25s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 46s | | trunk passed |
| +1 :green_heart: | compile | 23m 4s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 21m 31s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | mvnsite | 2m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 58s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 31s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 121m 12s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 52s | | the patch passed |
| +1 :green_heart: | compile | 22m 16s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 22m 16s | | the patch passed |
| +1 :green_heart: | compile | 20m 23s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 20m 23s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 2m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 54s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 2m 32s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | shadedclient | 32m 30s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 321m 19s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5331/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| -1 :x: | unit | 0m 47s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-documentstore.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5331/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice-documentstore.txt)
| hadoop-yarn-server-timelineservice-documentstore in the patch failed. |
| +0 :ok: | asflicense | 0m 47s | | ASF License check generated no
output? |
| | | 525m 34s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestSafeModeWithStripedFile |
| | hadoop.hdfs.TestDFSStorageStateRecovery |
| | hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.TestDeadNodeDetection |
| | hadoop.hdfs.TestErasureCodingMultipleRacks |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.TestViewDistributedFileSystemWithMountLinks |
| | hadoop.hdfs.TestFileChecksum |
| | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.TestMissingBlocksAlert |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5331/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5331 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint |
| uname | Linux f64d9b47924a 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #5309: YARN-11419. Remove redundant exception capture in NMClientAsyncImpl and improve readability in ContainerShellWebSocket, etc
slfan1989 commented on code in PR #5309:
URL: https://github.com/apache/hadoop/pull/5309#discussion_r1090779876
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/SharedCacheClientImpl.java:
##
@@ -158,14 +158,8 @@ public void release(ApplicationId applicationId, String
resourceKey)
public String getFileChecksum(Path sourceFile)
throws IOException {
FileSystem fs = sourceFile.getFileSystem(this.conf);
-FSDataInputStream in = null;
-try {
- in = fs.open(sourceFile);
+try (FSDataInputStream in = fs.open(sourceFile)) {
Review Comment:
The try-with-resources statement is a try statement that declares one or
more resources. A resource is an object that must be closed after the program
is finished with it. The try-with-resources statement ensures that each
resource is closed at the end of the statement. Any object that implements
java.lang.AutoCloseable, which includes all objects which implement
java.io.Closeable, can be used as a resource.
`FSDataInputStream` implement `java.io.Closeable`, Can we add a junit Test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[
https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682096#comment-17682096
]
ASF GitHub Bot commented on HADOOP-18206:
-
steveloughran commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1090640899
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/ServiceOperations.java:
##
@@ -67,26 +66,6 @@ public static Exception stopQuietly(Service service) {
return stopQuietly(LOG, service);
}
- /**
- * Stop a service; if it is null do nothing. Exceptions are caught and
- * logged at warn level. (but not Throwables). This operation is intended to
- * be used in cleanup operations
- *
- * @param log the log to warn at
- * @param service a service; may be null
- * @return any exception that was caught; null if none was.
- * @see ServiceOperations#stopQuietly(Service)
- */
- public static Exception stopQuietly(Log log, Service service) {
Review Comment:
again, we should tag as deprecated in branch 3
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java:
##
@@ -223,35 +222,6 @@ public synchronized static void
printThreadInfo(PrintStream stream,
private static long previousLogTime = 0;
- /**
- * Log the current thread stacks at INFO level.
- * @param log the logger that logs the stack trace
- * @param title a descriptive title for the call stacks
- * @param minInterval the minimum time from the last
- */
- public static void logThreadInfo(Log log,
Review Comment:
deprecate in 3.3
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java:
##
@@ -256,7 +255,7 @@ public static void skipFully(InputStream in, long len)
throws IOException {
* instead
*/
@Deprecated
- public static void cleanup(Log log, java.io.Closeable... closeables) {
+ public static void cleanup(Logger log, java.io.Closeable... closeables) {
Review Comment:
* for 3.3.5+ do a pr which tags as deprecated "will be cut" for early
warning.
* then do 1 or 2 here.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:
##
@@ -360,7 +358,8 @@ public class DataNode extends ReconfigurableBase
FS_GETSPACEUSED_JITTER_KEY,
FS_GETSPACEUSED_CLASSNAME));
- public static final Log METRICS_LOG =
LogFactory.getLog("DataNodeMetricsLog");
+ public static final org.apache.log4j.Logger METRICS_LOG =
+ org.apache.log4j.Logger.getLogger("DataNodeMetricsLog");
Review Comment:
is there a way the log4j refs can be kept inside MetricsLoggerTask...for
example, `makeMetricsLoggerAsync` could take a log name as a string and do the
lookup itself?
this would leave open a future option to move to reflection for the target
log platform
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -949,13 +947,13 @@ protected void startMetricsLogger(Configuration conf) {
return;
}
-MetricsLoggerTask.makeMetricsLoggerAsync(MetricsLog);
+MetricsLoggerTask.makeMetricsLoggerAsync(METRICS_LOG);
// Schedule the periodic logging.
metricsLoggerTimer = new ScheduledThreadPoolExecutor(1);
metricsLoggerTimer.setExecuteExistingDelayedTasksAfterShutdownPolicy(
false);
-metricsLoggerTimer.scheduleWithFixedDelay(new MetricsLoggerTask(MetricsLog,
+metricsLoggerTimer.scheduleWithFixedDelay(new
MetricsLoggerTask(METRICS_LOG,
Review Comment:
and we would take a log name here
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -949,13 +947,13 @@ protected void startMetricsLogger(Configuration conf) {
return;
}
-MetricsLoggerTask.makeMetricsLoggerAsync(MetricsLog);
+MetricsLoggerTask.makeMetricsLoggerAsync(METRICS_LOG);
Review Comment:
again, a version of this method which took a log name would handle this
internally.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -8853,16 +8849,12 @@ public void logAuditEvent(boolean succeeded, String
userName,
}
public void logAuditMessage(String message) {
- auditLog.info(message);
+ AUDIT_LOG.info(message);
}
}
private static void enableAsyncAuditLog(Configuration conf) {
-if (!(auditLog instanceof Log4JLogger)) {
Review Comment:
this is a key diff in the patch. today if the backend isn't log4j things
should work, just not do async logging (remember, logback doesn't need
this...), so this is possibly a regression
##
[GitHub] [hadoop] steveloughran commented on a diff in pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
steveloughran commented on code in PR #5315:
URL: https://github.com/apache/hadoop/pull/5315#discussion_r1090640899
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/service/ServiceOperations.java:
##
@@ -67,26 +66,6 @@ public static Exception stopQuietly(Service service) {
return stopQuietly(LOG, service);
}
- /**
- * Stop a service; if it is null do nothing. Exceptions are caught and
- * logged at warn level. (but not Throwables). This operation is intended to
- * be used in cleanup operations
- *
- * @param log the log to warn at
- * @param service a service; may be null
- * @return any exception that was caught; null if none was.
- * @see ServiceOperations#stopQuietly(Service)
- */
- public static Exception stopQuietly(Log log, Service service) {
Review Comment:
again, we should tag as deprecated in branch 3
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/ReflectionUtils.java:
##
@@ -223,35 +222,6 @@ public synchronized static void
printThreadInfo(PrintStream stream,
private static long previousLogTime = 0;
- /**
- * Log the current thread stacks at INFO level.
- * @param log the logger that logs the stack trace
- * @param title a descriptive title for the call stacks
- * @param minInterval the minimum time from the last
- */
- public static void logThreadInfo(Log log,
Review Comment:
deprecate in 3.3
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/IOUtils.java:
##
@@ -256,7 +255,7 @@ public static void skipFully(InputStream in, long len)
throws IOException {
* instead
*/
@Deprecated
- public static void cleanup(Log log, java.io.Closeable... closeables) {
+ public static void cleanup(Logger log, java.io.Closeable... closeables) {
Review Comment:
* for 3.3.5+ do a pr which tags as deprecated "will be cut" for early
warning.
* then do 1 or 2 here.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:
##
@@ -360,7 +358,8 @@ public class DataNode extends ReconfigurableBase
FS_GETSPACEUSED_JITTER_KEY,
FS_GETSPACEUSED_CLASSNAME));
- public static final Log METRICS_LOG =
LogFactory.getLog("DataNodeMetricsLog");
+ public static final org.apache.log4j.Logger METRICS_LOG =
+ org.apache.log4j.Logger.getLogger("DataNodeMetricsLog");
Review Comment:
is there a way the log4j refs can be kept inside MetricsLoggerTask...for
example, `makeMetricsLoggerAsync` could take a log name as a string and do the
lookup itself?
this would leave open a future option to move to reflection for the target
log platform
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -949,13 +947,13 @@ protected void startMetricsLogger(Configuration conf) {
return;
}
-MetricsLoggerTask.makeMetricsLoggerAsync(MetricsLog);
+MetricsLoggerTask.makeMetricsLoggerAsync(METRICS_LOG);
// Schedule the periodic logging.
metricsLoggerTimer = new ScheduledThreadPoolExecutor(1);
metricsLoggerTimer.setExecuteExistingDelayedTasksAfterShutdownPolicy(
false);
-metricsLoggerTimer.scheduleWithFixedDelay(new MetricsLoggerTask(MetricsLog,
+metricsLoggerTimer.scheduleWithFixedDelay(new
MetricsLoggerTask(METRICS_LOG,
Review Comment:
and we would take a log name here
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java:
##
@@ -949,13 +947,13 @@ protected void startMetricsLogger(Configuration conf) {
return;
}
-MetricsLoggerTask.makeMetricsLoggerAsync(MetricsLog);
+MetricsLoggerTask.makeMetricsLoggerAsync(METRICS_LOG);
Review Comment:
again, a version of this method which took a log name would handle this
internally.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -8853,16 +8849,12 @@ public void logAuditEvent(boolean succeeded, String
userName,
}
public void logAuditMessage(String message) {
- auditLog.info(message);
+ AUDIT_LOG.info(message);
}
}
private static void enableAsyncAuditLog(Configuration conf) {
-if (!(auditLog instanceof Log4JLogger)) {
Review Comment:
this is a key diff in the patch. today if the backend isn't log4j things
should work, just not do async logging (remember, logback doesn't need
this...), so this is possibly a regression
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/log/LogLevel.java:
##
@@ -51,6 +47,7 @@
import org.apache.hadoop.util.ServletUtil;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;
+import org.apache.log4j.Logger;
Review Comment:
with this as an explicit import,
[GitHub] [hadoop] hadoop-yetus commented on pull request #5312: YARN-11375. [Federation] Support refreshAdminAcls、refreshServiceAcls API's for Federation.
hadoop-yetus commented on PR #5312:
URL: https://github.com/apache/hadoop/pull/5312#issuecomment-1408638903
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +0 :ok: | buf | 0m 0s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 24s | | trunk passed |
| +1 :green_heart: | compile | 9m 35s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 8m 29s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 48s | | trunk passed |
| +1 :green_heart: | mvnsite | 4m 4s | | trunk passed |
| -1 :x: | javadoc | 0m 59s |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5312/6/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 3m 30s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 7m 14s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 12s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 25m 34s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 0s | | the patch passed |
| +1 :green_heart: | compile | 9m 1s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | cc | 9m 1s | | the patch passed |
| +1 :green_heart: | javac | 9m 1s | | the patch passed |
| +1 :green_heart: | compile | 8m 24s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | cc | 8m 24s | | the patch passed |
| +1 :green_heart: | javac | 8m 24s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5312/6/artifact/out/blanks-eol.txt)
| The patch has 3 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 1m 37s | |
hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 7 unchanged - 9
fixed = 7 total (was 16) |
| +1 :green_heart: | mvnsite | 3m 44s | | the patch passed |
| -1 :x: | javadoc | 0m 56s |
[/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5312/6/artifact/out/patch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 3m 10s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 7m 34s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 1m 13s | | hadoop-yarn-api in the patch
passed. |
| +1 :green_heart: | unit | 5m 42s | | hadoop-yarn-common in the patch
passed. |
| +1 :green_heart: | unit | 99m 0s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 0m
[jira] [Commented] (HADOOP-18206) Cleanup the commons-logging references in the code base
[ https://issues.apache.org/jira/browse/HADOOP-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682084#comment-17682084 ] ASF GitHub Bot commented on HADOOP-18206: - steveloughran commented on PR #5315: URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1408635376 so those production classes I am worrying about are already explicitly requiring log4j on the CP? OK, that's not as bad as I feared...I was worrying that the code was going from commons-logging wrappers of logj to raw log4j, which is a problem > Cleanup the commons-logging references in the code base > --- > > Key: HADOOP-18206 > URL: https://issues.apache.org/jira/browse/HADOOP-18206 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > > Should always use slf4j -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5315: HADOOP-18206 Cleanup the commons-logging references and restrict its usage in future
steveloughran commented on PR #5315: URL: https://github.com/apache/hadoop/pull/5315#issuecomment-1408635376 so those production classes I am worrying about are already explicitly requiring log4j on the CP? OK, that's not as bad as I feared...I was worrying that the code was going from commons-logging wrappers of logj to raw log4j, which is a problem -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5302: YARN-11221. [Federation] Add replaceLabelsOnNodes, replaceLabelsOnNode REST APIs for Router.
hadoop-yetus commented on PR #5302:
URL: https://github.com/apache/hadoop/pull/5302#issuecomment-1408537446
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 2s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 30m 52s | | trunk passed |
| +1 :green_heart: | compile | 3m 52s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 3m 18s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 15s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 37s | | trunk passed |
| -1 :x: | javadoc | 0m 55s |
[/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5302/11/artifact/out/branch-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server-resourcemanager in trunk failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| +1 :green_heart: | javadoc | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 58s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 16s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 35s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 21s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5302/11/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt)
| hadoop-yarn-server-router in the patch failed. |
| -1 :x: | compile | 3m 31s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5302/11/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | javac | 3m 31s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5302/11/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkUbuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04.txt)
| hadoop-yarn-server in the patch failed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04. |
| -1 :x: | compile | 3m 2s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5302/11/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| hadoop-yarn-server in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | javac | 3m 2s |
[/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5302/11/artifact/out/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server-jdkPrivateBuild-1.8.0_352-8u352-ga-1~20.04-b08.txt)
| hadoop-yarn-server in the patch failed with JDK Private
Build-1.8.0_352-8u352-ga-1~20.04-b08. |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5302/11/artifact/out/blanks-eol.txt)
| The patch has 6 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer
[jira] [Created] (HADOOP-18607) Fix TestAddOverReplicatedStripedBlocks#testProcessOverReplicatedAndCorruptStripedBlock failed
Haiyang Hu created HADOOP-18607: --- Summary: Fix TestAddOverReplicatedStripedBlocks#testProcessOverReplicatedAndCorruptStripedBlock failed Key: HADOOP-18607 URL: https://issues.apache.org/jira/browse/HADOOP-18607 Project: Hadoop Common Issue Type: Bug Reporter: Haiyang Hu -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18606) Add reason in in x-ms-client-request-id on a retry API call.
[ https://issues.apache.org/jira/browse/HADOOP-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HADOOP-18606: Labels: pull-request-available (was: ) > Add reason in in x-ms-client-request-id on a retry API call. > > > Key: HADOOP-18606 > URL: https://issues.apache.org/jira/browse/HADOOP-18606 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > In the header, x-ms-client-request-id contains informaiton on what retry this > particular API call is: for ex: > :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1. > We want to add the reason for the retry in the header_value:Now the same > header would include retry reason in case its not the 0th iteration of the > API operation. It would be like > :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1_RT. > This corresponds that its retry number 1. The 0th iteration was failed due > to read timeout. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18606) Add reason in in x-ms-client-request-id on a retry API call.
[ https://issues.apache.org/jira/browse/HADOOP-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682023#comment-17682023 ] ASF GitHub Bot commented on HADOOP-18606: - pranavsaxena-microsoft commented on PR #5299: URL: https://github.com/apache/hadoop/pull/5299#issuecomment-1408461428 @steveloughran @mukund-thakur @mehakmeet , requesting you to kindly review the PR please. Thanks. > Add reason in in x-ms-client-request-id on a retry API call. > > > Key: HADOOP-18606 > URL: https://issues.apache.org/jira/browse/HADOOP-18606 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Minor > Fix For: 3.4.0 > > > In the header, x-ms-client-request-id contains informaiton on what retry this > particular API call is: for ex: > :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1. > We want to add the reason for the retry in the header_value:Now the same > header would include retry reason in case its not the 0th iteration of the > API operation. It would be like > :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1_RT. > This corresponds that its retry number 1. The 0th iteration was failed due > to read timeout. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] pranavsaxena-microsoft commented on pull request #5299: HADOOP-18606. Add reason in in x-ms-client-request-id on a retry API call.
pranavsaxena-microsoft commented on PR #5299: URL: https://github.com/apache/hadoop/pull/5299#issuecomment-1408461428 @steveloughran @mukund-thakur @mehakmeet , requesting you to kindly review the PR please. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18606) Add reason in in x-ms-client-request-id on a retry API call.
[ https://issues.apache.org/jira/browse/HADOOP-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pranav Saxena updated HADOOP-18606: --- Fix Version/s: 3.4.0 > Add reason in in x-ms-client-request-id on a retry API call. > > > Key: HADOOP-18606 > URL: https://issues.apache.org/jira/browse/HADOOP-18606 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Minor > Fix For: 3.4.0 > > > In the header, x-ms-client-request-id contains informaiton on what retry this > particular API call is: for ex: > :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1. > We want to add the reason for the retry in the header_value:Now the same > header would include retry reason in case its not the 0th iteration of the > API operation. It would be like > :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1_RT. > This corresponds that its retry number 1. The 0th iteration was failed due > to read timeout. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18606) Add reason in in x-ms-client-request-id on a retry API call.
[ https://issues.apache.org/jira/browse/HADOOP-18606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pranav Saxena updated HADOOP-18606: --- Component/s: fs/azure > Add reason in in x-ms-client-request-id on a retry API call. > > > Key: HADOOP-18606 > URL: https://issues.apache.org/jira/browse/HADOOP-18606 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure >Reporter: Pranav Saxena >Assignee: Pranav Saxena >Priority: Minor > Fix For: 3.4.0 > > > In the header, x-ms-client-request-id contains informaiton on what retry this > particular API call is: for ex: > :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1. > We want to add the reason for the retry in the header_value:Now the same > header would include retry reason in case its not the 0th iteration of the > API operation. It would be like > :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1_RT. > This corresponds that its retry number 1. The 0th iteration was failed due > to read timeout. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18596) Distcp -update between different cloud stores to use modification time while checking for file skip.
[
https://issues.apache.org/jira/browse/HADOOP-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17682021#comment-17682021
]
ASF GitHub Bot commented on HADOOP-18596:
-
hadoop-yetus commented on PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#issuecomment-1408457069
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 19s | | trunk passed |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 4s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 46m 3s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 154m 24s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5308/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5308 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 33f023938bd4 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d23f13be729f0fe0770ce856b4f676f6782d83b9 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5308/3/testReport/ |
| Max. process+thread count | 540 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5308/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Distcp -update between different cloud stores to use modification time while
> checking for
[jira] [Created] (HADOOP-18606) Add reason in in x-ms-client-request-id on a retry API call.
Pranav Saxena created HADOOP-18606: -- Summary: Add reason in in x-ms-client-request-id on a retry API call. Key: HADOOP-18606 URL: https://issues.apache.org/jira/browse/HADOOP-18606 Project: Hadoop Common Issue Type: Sub-task Reporter: Pranav Saxena Assignee: Pranav Saxena In the header, x-ms-client-request-id contains informaiton on what retry this particular API call is: for ex: :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1. We want to add the reason for the retry in the header_value:Now the same header would include retry reason in case its not the 0th iteration of the API operation. It would be like :eb06d8f6-5693-461b-b63c-5858fa7655e6:29cb0d19-2b68-4409-bc35-cb7160b90dd8:::CF:1_RT. This corresponds that its retry number 1. The 0th iteration was failed due to read timeout. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5308: HADOOP-18596. Distcp -update to use modification time while checking for file skip.
hadoop-yetus commented on PR #5308:
URL: https://github.com/apache/hadoop/pull/5308#issuecomment-1408457069
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 57s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 19s | | trunk passed |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 27s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 27s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 31s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 25s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 4s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 30s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 46m 3s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 32s | | The patch does not
generate ASF License warnings. |
| | | 154m 24s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5308/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5308 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 33f023938bd4 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d23f13be729f0fe0770ce856b4f676f6782d83b9 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5308/3/testReport/ |
| Max. process+thread count | 540 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-distcp U: hadoop-tools/hadoop-distcp |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5308/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5272: YARN-11217. [Federation] Add dumpSchedulerLogs REST APIs for Router.
hadoop-yetus commented on PR #5272:
URL: https://github.com/apache/hadoop/pull/5272#issuecomment-1408378798
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 52s | | trunk passed |
| +1 :green_heart: | compile | 0m 31s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 28s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 28s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 32s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 12s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 26m 31s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 25s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 25s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5272/14/artifact/out/blanks-eol.txt)
| The patch has 9 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 11s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 27s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 33s | | The patch does not
generate ASF License warnings. |
| | | 108m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5272/14/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5272 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b1bf70a6ada4 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3c04fe60228dcecf1337b72923bbf714fed376c6 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5272/14/testReport/ |
| Max. process+thread count | 553 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5272/14/console |
| versions | git=2.25.1
[GitHub] [hadoop] hadoop-yetus commented on pull request #5328: YARN-11222. [Federation] Add addToClusterNodeLabels, removeFromClusterNodeLabels REST APIs for Router.
hadoop-yetus commented on PR #5328:
URL: https://github.com/apache/hadoop/pull/5328#issuecomment-1408374029
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 27s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 21s | | trunk passed |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | compile | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 36s | | trunk passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 24s | | trunk passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 26m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 29s | | the patch passed |
| +1 :green_heart: | compile | 0m 27s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javac | 0m 27s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 21s | | the patch passed with JDK
Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04 |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 0m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 26m 20s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 0m 29s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 111m 59s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5328/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5328 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux eca95ba6967a 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24
18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c16c9ba64ef16737fc34fd2f5f77aff43f5f9493 |
| Default Java | Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.17+8-post-Ubuntu-1ubuntu220.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_352-8u352-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5328/3/testReport/ |
| Max. process+thread count | 536 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5328/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[jira] [Updated] (HADOOP-18605) S3A AssumedRole credentials provider should use Instance Role credentials in chain for assuming role
[ https://issues.apache.org/jira/browse/HADOOP-18605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Carl Jones updated HADOOP-18605: --- Summary: S3A AssumedRole credentials provider should use Instance Role credentials in chain for assuming role (was: S3A AssumedRoleCredProvider should use Instance Role credentials in chain for assuming role) > S3A AssumedRole credentials provider should use Instance Role credentials in > chain for assuming role > > > Key: HADOOP-18605 > URL: https://issues.apache.org/jira/browse/HADOOP-18605 > Project: Hadoop Common > Issue Type: Sub-task > Components: auth, fs/s3 >Reporter: Daniel Carl Jones >Priority: Minor > > The AssumedRoleCredentialsProvider today creates its own credential provider > set to use when getting session credentials from STS. This includes only > environment variables and simple credentials provider. > [https://github.com/apache/hadoop/blob/952d707240cb7dd088820d8b39e705b77fa3b6c4/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AssumedRoleCredentialProvider.java#L114-L116] > > It should at least include EC2 Instance Role credentials, possibly others, by > default. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18605) S3A AssumedRoleCredProvider should use Instance Role credentials in chain for assuming role
Daniel Carl Jones created HADOOP-18605: -- Summary: S3A AssumedRoleCredProvider should use Instance Role credentials in chain for assuming role Key: HADOOP-18605 URL: https://issues.apache.org/jira/browse/HADOOP-18605 Project: Hadoop Common Issue Type: Sub-task Components: auth, fs/s3 Reporter: Daniel Carl Jones The AssumedRoleCredentialsProvider today creates its own credential provider set to use when getting session credentials from STS. This includes only environment variables and simple credentials provider. [https://github.com/apache/hadoop/blob/952d707240cb7dd088820d8b39e705b77fa3b6c4/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/auth/AssumedRoleCredentialProvider.java#L114-L116] It should at least include EC2 Instance Role credentials, possibly others, by default. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] p-szucs opened a new pull request, #5332: Yarn 11041. Replace all occurences of queuePath with the new QueuePath class - followup
p-szucs opened a new pull request, #5332: URL: https://github.com/apache/hadoop/pull/5332 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
