[GitHub] [hadoop] hadoop-yetus commented on pull request #4375: HDFS-16605. Improve Code With Lambda in hadoop-hdfs-rbf moudle.
hadoop-yetus commented on PR #4375:
URL: https://github.com/apache/hadoop/pull/4375#issuecomment-1191073055
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 10 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 4s | | trunk passed |
| +1 :green_heart: | compile | 1m 0s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 43s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 42s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 25m 5s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 23s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 39s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 40m 42s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not
generate ASF License warnings. |
| | | 150m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4375/13/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4375 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 2abba7ce183f 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 0eb4c4a62514b72e989e87534a46c823d201e14a |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4375/13/testReport/ |
| Max. process+thread count | 2296 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4375/13/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git
[GitHub] [hadoop] hadoop-yetus commented on pull request #4594: YARN-6572. Refactoring Router services to use common util classes for pipeline creations.
hadoop-yetus commented on PR #4594:
URL: https://github.com/apache/hadoop/pull/4594#issuecomment-1191049698
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 24s | | trunk passed |
| +1 :green_heart: | compile | 4m 46s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 4m 8s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 29s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 27s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 13s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 36s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 4m 38s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 38s | | the patch passed |
| +1 :green_heart: | compile | 3m 45s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 13s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4594/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1
new + 3 unchanged - 2 fixed = 4 total (was 5) |
| +1 :green_heart: | mvnsite | 1m 9s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 31s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 48s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 3m 9s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 136m 51s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4594/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4594 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 05609be9b151 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8
19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 34f83e76d08304ca4213e859bc5172c26399b83a |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4594/4/testReport/ |
| Max. process+thread count | 1408 (vs. ulimit of 5500) |
| modules | C:
[GitHub] [hadoop] hadoop-yetus commented on pull request #4594: YARN-6572. Refactoring Router services to use common util classes for pipeline creations.
hadoop-yetus commented on PR #4594:
URL: https://github.com/apache/hadoop/pull/4594#issuecomment-1191044211
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 26s | | trunk passed |
| +1 :green_heart: | compile | 4m 22s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 3m 32s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 26s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 36s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 25s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 0m 59s | | the patch passed |
| +1 :green_heart: | compile | 4m 15s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 4m 15s | | the patch passed |
| +1 :green_heart: | compile | 3m 30s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 3m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 1m 14s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4594/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server.txt)
| hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server: The patch generated 1
new + 3 unchanged - 2 fixed = 4 total (was 5) |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 56s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 45s | | hadoop-yarn-server-common in
the patch passed. |
| +1 :green_heart: | unit | 3m 12s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 42s | | The patch does not
generate ASF License warnings. |
| | | 136m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4594/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4594 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux de910e769797 4.15.0-166-generic #174-Ubuntu SMP Wed Dec 8
19:07:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 58555baa13435644be441039e512ac5999543e5d |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4594/3/testReport/ |
| Max. process+thread count | 1351 (vs. ulimit of 5500) |
| modules | C:
[GitHub] [hadoop] hadoop-yetus commented on pull request #4597: HDFS-16671. RBF: RouterRpcFairnessPolicyController supports configurable permit acquire timeout
hadoop-yetus commented on PR #4597:
URL: https://github.com/apache/hadoop/pull/4597#issuecomment-1191041960
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 56s | | trunk passed |
| +1 :green_heart: | compile | 1m 2s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 52s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 8s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 15s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 46s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 26s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 21m 49s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not
generate ASF License warnings. |
| | | 120m 42s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4597/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux a5492a0cb95f 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / bfb922926e46f644cad83389f6ca9c404085315d |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4597/3/testReport/ |
| Max. process+thread count | 2761 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4597/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific
[GitHub] [hadoop] skysiders commented on pull request #4026: MAPREDUCE-7372 MapReduce set permission too late in copyJar method
skysiders commented on PR #4026: URL: https://github.com/apache/hadoop/pull/4026#issuecomment-1191035649 Hi @cnauroth ,thanks for your review. I add some comment to explain this. Could you please take a look this?Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] cxzl25 commented on a diff in pull request #4139: HDFS-16524. Add configuration to control blocks deletion asynchronous…
cxzl25 commented on code in PR #4139:
URL: https://github.com/apache/hadoop/pull/4139#discussion_r926235580
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:
##
@@ -3410,6 +3410,33 @@ FSPermissionChecker getPermissionChecker()
return dir.getPermissionChecker();
}
+ /**
+ * If blockDeletionAsync enables, blocks will be deleted asynchronously.
+ * If not, incrementally remove the blocks from blockManager
+ * Writelock is dropped and reacquired every BLOCK_DELETION_INCREMENT to
+ * ensure that other waiters on the lock can get in. See HDFS-2938
+ *
+ * @param toDeleteList
+ * a list of blocks that need to be removed from blocksMap
+ */
+ void removeBlocks(List toDeleteList) {
+if (this.blockDeletionAsync) {
+ blockManager.addBLocksToMarkedDeleteQueue(toDeleteList);
+} else {
+ Iterator iter = toDeleteList.iterator();
+ while (iter.hasNext()) {
+writeLock();
+try {
+ for (int i = 0; i < blockDeletionIncrement && iter.hasNext(); i++) {
Review Comment:
Removed blockDeletionIncrement in HDFS-16554
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:
##
@@ -481,6 +481,10 @@ public class DFSConfigKeys extends CommonConfigurationKeys
{
"dfs.namenode.block.deletion.increment";
public static final int DFS_NAMENODE_BLOCK_DELETION_INCREMENT_DEFAULT = 1000;
+ /** Block deletion asynchronous. */
+ public static final String DFS_NAMENODE_BLOCK_DELETION_ASYNC_KEY =
"dfs.namenode.block.deletion.async";
+ public static final boolean DFS_NAMENODE_BLOCK_DELETION_ASYNC_DEFAULT =
false;
Review Comment:
Should it default to true?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ferhui commented on a diff in pull request #4529: HDFS-16648. Add isDebugEnabled check for debug logs in some classes
ferhui commented on code in PR #4529:
URL: https://github.com/apache/hadoop/pull/4529#discussion_r926223561
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Server.java:
##
@@ -3152,15 +3152,17 @@ public void run() {
IOUtils.cleanupWithLogger(LOG, traceScope);
if (call != null) {
updateMetrics(call, startTimeNanos, connDropped);
-ProcessingDetails.LOG.debug(
Review Comment:
@aajisaka mentioned on other PRs.
> If the log level is not debug, toString() is not actually called.
> http://www.slf4j.org/faq.html#logging_performance
So it is unnecessary that add isDebugEnabled here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4594: YARN-6572. Refactoring Router services to use common util classes for pipeline creations.
slfan1989 commented on code in PR #4594:
URL: https://github.com/apache/hadoop/pull/4594#discussion_r926209505
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/rmadmin/RouterRMAdminService.java:
##
@@ -121,8 +119,7 @@ protected void serviceStart() throws Exception {
conf.getInt(YarnConfiguration.ROUTER_PIPELINE_CACHE_MAX_SIZE,
YarnConfiguration.DEFAULT_ROUTER_PIPELINE_CACHE_MAX_SIZE);
this.userPipelineMap = Collections.synchronizedMap(
-new LRUCacheHashMap(
-maxCacheSize, true));
+ new LRUCacheHashMap<>(maxCacheSize, true));
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4594: YARN-6572. Refactoring Router services to use common util classes for pipeline creations.
slfan1989 commented on code in PR #4594:
URL: https://github.com/apache/hadoop/pull/4594#discussion_r926208922
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -507,42 +480,25 @@ protected Map
getPipelines() {
@VisibleForTesting
protected ClientRequestInterceptor createRequestInterceptorChain() {
Configuration conf = getConfig();
+ClientRequestInterceptor pipeline = null;
+ClientMethod remoteMethod = null;
+try {
+ remoteMethod = new ClientMethod("setNextInterceptor",
+ new Class[]{ClientRequestInterceptor.class}, new Object[]{null});
-List interceptorClassNames = getInterceptorClassNames(conf);
+ pipeline = RouterServerUtil.createRequestInterceptorChain(conf,
+ YarnConfiguration.ROUTER_CLIENTRM_INTERCEPTOR_CLASS_PIPELINE,
+ YarnConfiguration.DEFAULT_ROUTER_CLIENTRM_INTERCEPTOR_CLASS,
+ remoteMethod, ClientRequestInterceptor.class);
-ClientRequestInterceptor pipeline = null;
-ClientRequestInterceptor current = null;
-for (String interceptorClassName : interceptorClassNames) {
- try {
-Class interceptorClass = conf.getClassByName(interceptorClassName);
-if (ClientRequestInterceptor.class.isAssignableFrom(interceptorClass))
{
- ClientRequestInterceptor interceptorInstance =
- (ClientRequestInterceptor) ReflectionUtils
- .newInstance(interceptorClass, conf);
- if (pipeline == null) {
-pipeline = interceptorInstance;
-current = interceptorInstance;
-continue;
- } else {
-current.setNextInterceptor(interceptorInstance);
-current = interceptorInstance;
- }
-} else {
- throw new YarnRuntimeException(
- "Class: " + interceptorClassName + " not instance of "
- + ClientRequestInterceptor.class.getCanonicalName());
-}
- } catch (ClassNotFoundException e) {
+ if (pipeline == null) {
throw new YarnRuntimeException(
-"Could not instantiate ApplicationClientRequestInterceptor: "
-+ interceptorClassName,
-e);
+"RequestInterceptor pipeline is not configured in the system.");
}
-}
-
-if (pipeline == null) {
- throw new YarnRuntimeException(
- "RequestInterceptor pipeline is not configured in the system");
+} catch (IOException | InvocationTargetException | NoSuchMethodException |
RuntimeException
Review Comment:
I will refactor this part of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4594: YARN-6572. Refactoring Router services to use common util classes for pipeline creations.
slfan1989 commented on code in PR #4594:
URL: https://github.com/apache/hadoop/pull/4594#discussion_r926208755
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/rmadmin/RouterRMAdminService.java:
##
@@ -215,43 +190,24 @@ protected Map
getPipelines() {
@VisibleForTesting
protected RMAdminRequestInterceptor createRequestInterceptorChain() {
Configuration conf = getConfig();
+RMAdminRequestInterceptor pipeline = null;
+ClientMethod remoteMethod = null;
+try {
+ remoteMethod = new ClientMethod("setNextInterceptor",
+ new Class[]{RMAdminRequestInterceptor.class}, new
Object[]{null});
-List interceptorClassNames = getInterceptorClassNames(conf);
+ pipeline = RouterServerUtil.createRequestInterceptorChain(conf,
+ YarnConfiguration.ROUTER_RMADMIN_INTERCEPTOR_CLASS_PIPELINE,
+ YarnConfiguration.DEFAULT_ROUTER_RMADMIN_INTERCEPTOR_CLASS,
+ remoteMethod, RMAdminRequestInterceptor.class);
-RMAdminRequestInterceptor pipeline = null;
-RMAdminRequestInterceptor current = null;
-for (String interceptorClassName : interceptorClassNames) {
- try {
-Class interceptorClass = conf.getClassByName(interceptorClassName);
-if (RMAdminRequestInterceptor.class
-.isAssignableFrom(interceptorClass)) {
- RMAdminRequestInterceptor interceptorInstance =
- (RMAdminRequestInterceptor) ReflectionUtils
- .newInstance(interceptorClass, conf);
- if (pipeline == null) {
-pipeline = interceptorInstance;
-current = interceptorInstance;
-continue;
- } else {
-current.setNextInterceptor(interceptorInstance);
-current = interceptorInstance;
- }
-} else {
- throw new YarnRuntimeException(
- "Class: " + interceptorClassName + " not instance of "
- + RMAdminRequestInterceptor.class.getCanonicalName());
-}
- } catch (ClassNotFoundException e) {
+ if (pipeline == null) {
throw new YarnRuntimeException(
-"Could not instantiate RMAdminRequestInterceptor: "
-+ interceptorClassName,
-e);
+"RequestInterceptor pipeline is not configured in the system.");
}
-}
-
-if (pipeline == null) {
- throw new YarnRuntimeException(
- "RequestInterceptor pipeline is not configured in the system");
+} catch (IOException | InvocationTargetException | NoSuchMethodException |
RuntimeException
+ | IllegalAccessException ex) {
+ throw new YarnRuntimeException("Create RequestInterceptor Chain error.",
ex);
}
return pipeline;
Review Comment:
Thank you for your suggestion. I handled different Exceptions in
RouterServerUtil#createRequestInterceptorChain, and used YarnRuntimeException
to make it easier for the caller to use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #4529: HDFS-16648. Add isDebugEnabled check for debug logs in some classes
ZanderXu commented on PR #4529: URL: https://github.com/apache/hadoop/pull/4529#issuecomment-1190973192 @slfan1989 ping. Can you push this PR forward? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #4596: HDFS-16670. Improve Code With Lambda in EditLogTailer class
ZanderXu commented on PR #4596: URL: https://github.com/apache/hadoop/pull/4596#issuecomment-1190972132 @slfan1989 Thanks for your suggestion, I will create a draft PR to found all places of Hadoop-Hdfs module. After I completed it, I'm asking for your help to review it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4510: YARN-11203. Fix typo in hadoop-yarn-server-router module.
slfan1989 commented on PR #4510: URL: https://github.com/apache/hadoop/pull/4510#issuecomment-1190970385 @tomscut Can you help merge this pr to trunk branch? thank you very much! I hope other prs can use the correct typo info. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17461) Add thread-level IOStatistics Context
[
https://issues.apache.org/jira/browse/HADOOP-17461?focusedWorklogId=793463=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793463
]
ASF GitHub Bot logged work on HADOOP-17461:
---
Author: ASF GitHub Bot
Created on: 20/Jul/22 23:02
Start Date: 20/Jul/22 23:02
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4352:
URL: https://github.com/apache/hadoop/pull/4352#issuecomment-1190851834
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 31s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 34s | | trunk passed |
| +1 :green_heart: | compile | 23m 11s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 51s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 50s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 33s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 2s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 52s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 24s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 45s | | the patch passed |
| +1 :green_heart: | compile | 22m 24s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 24s | | the patch passed |
| +1 :green_heart: | compile | 20m 50s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 50s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 44s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 41s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 51s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 20s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 39s | | The patch does not
generate ASF License warnings. |
| | | 244m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4352/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4352 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 747c1495282e 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7b3c3356cd52f5378c4d68b6ddfed548d074ddf6 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
[GitHub] [hadoop] hadoop-yetus commented on pull request #4352: HADOOP-17461. Thread-level IOStatistics in S3A
hadoop-yetus commented on PR #4352:
URL: https://github.com/apache/hadoop/pull/4352#issuecomment-1190851834
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 46s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 31s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 34s | | trunk passed |
| +1 :green_heart: | compile | 23m 11s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 51s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 21s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 50s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 33s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 2s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 52s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 24m 24s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 35s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 45s | | the patch passed |
| +1 :green_heart: | compile | 22m 24s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 22m 24s | | the patch passed |
| +1 :green_heart: | compile | 20m 50s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 20m 50s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 19s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 44s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 41s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 12s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 51s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 20s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 39s | | The patch does not
generate ASF License warnings. |
| | | 244m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4352/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4352 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 747c1495282e 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 7b3c3356cd52f5378c4d68b6ddfed548d074ddf6 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4352/12/testReport/ |
| Max. process+thread count | 3158 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws
U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4352/12/console |
|
[GitHub] [hadoop] slfan1989 commented on pull request #4596: HDFS-16670. Improve Code With Lambda in EditLogTailer class
slfan1989 commented on PR #4596: URL: https://github.com/apache/hadoop/pull/4596#issuecomment-1190845896 @ZanderXu I suggest to create a draft pr first, find all the places to be optimized in hadoop-hdfs (with Lambda), let's see how many changes are there? If it is really a lot, split multiple prs. If the number is not too many, it is best to complete it in one pr, so that it will be easier for others to backport your code. thanks for your contribution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4594: YARN-6572. Refactoring Router services to use common util classes for pipeline creations.
goiri commented on code in PR #4594:
URL: https://github.com/apache/hadoop/pull/4594#discussion_r926110190
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/rmadmin/RouterRMAdminService.java:
##
@@ -121,8 +119,7 @@ protected void serviceStart() throws Exception {
conf.getInt(YarnConfiguration.ROUTER_PIPELINE_CACHE_MAX_SIZE,
YarnConfiguration.DEFAULT_ROUTER_PIPELINE_CACHE_MAX_SIZE);
this.userPipelineMap = Collections.synchronizedMap(
-new LRUCacheHashMap(
-maxCacheSize, true));
+ new LRUCacheHashMap<>(maxCacheSize, true));
Review Comment:
Single line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4594: YARN-6572. Refactoring Router services to use common util classes for pipeline creations.
slfan1989 commented on code in PR #4594:
URL: https://github.com/apache/hadoop/pull/4594#discussion_r926106188
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -566,14 +522,14 @@ private RequestInterceptorChainWrapper
initializePipeline(String user) {
// We should init the pipeline instance after it is created and then
// add to the map, to ensure thread safe.
LOG.info("Initializing request processing pipeline for application "
-+ "for the user: {}", user);
++ "for the user: {}.", user);
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4594: YARN-6572. Refactoring Router services to use common util classes for pipeline creations.
slfan1989 commented on code in PR #4594:
URL: https://github.com/apache/hadoop/pull/4594#discussion_r926105958
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/rmadmin/RouterRMAdminService.java:
##
@@ -215,43 +190,24 @@ protected Map
getPipelines() {
@VisibleForTesting
protected RMAdminRequestInterceptor createRequestInterceptorChain() {
Configuration conf = getConfig();
+RMAdminRequestInterceptor pipeline = null;
+ClientMethod remoteMethod = null;
+try {
+ remoteMethod = new ClientMethod("setNextInterceptor",
Review Comment:
i will fix it.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/rmadmin/RouterRMAdminService.java:
##
@@ -215,43 +190,24 @@ protected Map
getPipelines() {
@VisibleForTesting
protected RMAdminRequestInterceptor createRequestInterceptorChain() {
Configuration conf = getConfig();
+RMAdminRequestInterceptor pipeline = null;
+ClientMethod remoteMethod = null;
+try {
+ remoteMethod = new ClientMethod("setNextInterceptor",
Review Comment:
I will fix it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on a diff in pull request #4594: YARN-6572. Refactoring Router services to use common util classes for pipeline creations.
slfan1989 commented on code in PR #4594:
URL: https://github.com/apache/hadoop/pull/4594#discussion_r926105787
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/RouterWebServices.java:
##
@@ -206,50 +181,33 @@ protected Map
getPipelines() {
*/
@VisibleForTesting
protected RESTRequestInterceptor createRequestInterceptorChain() {
+RESTRequestInterceptor pipeline = null;
Review Comment:
Thanks for your help reviewing the code, I will continue to refactor this
part of the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4597: HDFS-16671. RBF: RouterRpcFairnessPolicyController supports configurable permit acquire timeout
goiri commented on code in PR #4597:
URL: https://github.com/apache/hadoop/pull/4597#discussion_r926056965
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/fairness/StaticRouterRpcFairnessPolicyController.java:
##
@@ -109,16 +109,15 @@ public void init(Configuration conf)
}
private static void logAssignment(String nsId, int count) {
-LOG.info("Assigned {} handlers to nsId {} ",
-count, nsId);
+LOG.info("Assigned {} handlers to nsId {} ", count, nsId);
}
- private void validateHandlersCount(Configuration conf, int handlerCount,
- Set allConfiguredNS) {
+ private void validateHandlersCount(Configuration conf,
+ int handlerCount, Set allConfiguredNS) {
int totalDedicatedHandlers = 0;
for (String nsId : allConfiguredNS) {
int dedicatedHandlers =
- conf.getInt(DFS_ROUTER_FAIR_HANDLER_COUNT_KEY_PREFIX + nsId, 0);
+ conf.getInt(DFS_ROUTER_FAIR_HANDLER_COUNT_KEY_PREFIX + nsId, 0);
Review Comment:
Make it one line as we are touching this.
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/fairness/AbstractRouterRpcFairnessPolicyController.java:
##
@@ -42,15 +45,22 @@
/** Hash table to hold semaphore for each configured name service. */
private Map permits;
+ private long acquireTimeout = DFS_ROUTER_FAIRNESS_ACQUIRE_TIMEOUT_DEFAULT;
+
public void init(Configuration conf) {
this.permits = new HashMap<>();
+long timeout = conf.getLong(DFS_ROUTER_FAIRNESS_ACQUIRE_TIMEOUT_KEY,
Review Comment:
What's the unit? We should do getTimeDuration()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] goiri commented on a diff in pull request #4594: YARN-6572. Refactoring Router services to use common util classes for pipeline creations.
goiri commented on code in PR #4594:
URL: https://github.com/apache/hadoop/pull/4594#discussion_r926043147
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/rmadmin/RouterRMAdminService.java:
##
@@ -215,43 +190,24 @@ protected Map
getPipelines() {
@VisibleForTesting
protected RMAdminRequestInterceptor createRequestInterceptorChain() {
Configuration conf = getConfig();
+RMAdminRequestInterceptor pipeline = null;
+ClientMethod remoteMethod = null;
+try {
+ remoteMethod = new ClientMethod("setNextInterceptor",
Review Comment:
Declare it directly here:
```
ClientMethod remoteMethod = new ClientMethod("setNextInterceptor",
```
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -566,14 +522,14 @@ private RequestInterceptorChainWrapper
initializePipeline(String user) {
// We should init the pipeline instance after it is created and then
// add to the map, to ensure thread safe.
LOG.info("Initializing request processing pipeline for application "
-+ "for the user: {}", user);
++ "for the user: {}.", user);
Review Comment:
We could put the string in a single line.
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/rmadmin/RouterRMAdminService.java:
##
@@ -215,43 +190,24 @@ protected Map
getPipelines() {
@VisibleForTesting
protected RMAdminRequestInterceptor createRequestInterceptorChain() {
Configuration conf = getConfig();
+RMAdminRequestInterceptor pipeline = null;
+ClientMethod remoteMethod = null;
+try {
+ remoteMethod = new ClientMethod("setNextInterceptor",
+ new Class[]{RMAdminRequestInterceptor.class}, new
Object[]{null});
-List interceptorClassNames = getInterceptorClassNames(conf);
+ pipeline = RouterServerUtil.createRequestInterceptorChain(conf,
+ YarnConfiguration.ROUTER_RMADMIN_INTERCEPTOR_CLASS_PIPELINE,
+ YarnConfiguration.DEFAULT_ROUTER_RMADMIN_INTERCEPTOR_CLASS,
+ remoteMethod, RMAdminRequestInterceptor.class);
-RMAdminRequestInterceptor pipeline = null;
-RMAdminRequestInterceptor current = null;
-for (String interceptorClassName : interceptorClassNames) {
- try {
-Class interceptorClass = conf.getClassByName(interceptorClassName);
-if (RMAdminRequestInterceptor.class
-.isAssignableFrom(interceptorClass)) {
- RMAdminRequestInterceptor interceptorInstance =
- (RMAdminRequestInterceptor) ReflectionUtils
- .newInstance(interceptorClass, conf);
- if (pipeline == null) {
-pipeline = interceptorInstance;
-current = interceptorInstance;
-continue;
- } else {
-current.setNextInterceptor(interceptorInstance);
-current = interceptorInstance;
- }
-} else {
- throw new YarnRuntimeException(
- "Class: " + interceptorClassName + " not instance of "
- + RMAdminRequestInterceptor.class.getCanonicalName());
-}
- } catch (ClassNotFoundException e) {
+ if (pipeline == null) {
throw new YarnRuntimeException(
-"Could not instantiate RMAdminRequestInterceptor: "
-+ interceptorClassName,
-e);
+"RequestInterceptor pipeline is not configured in the system.");
}
-}
-
-if (pipeline == null) {
- throw new YarnRuntimeException(
- "RequestInterceptor pipeline is not configured in the system");
+} catch (IOException | InvocationTargetException | NoSuchMethodException |
RuntimeException
+ | IllegalAccessException ex) {
+ throw new YarnRuntimeException("Create RequestInterceptor Chain error.",
ex);
}
return pipeline;
Review Comment:
Given that if error we always return exception, can we define the `pipeline`
inside of the try and do the exception returns properly?
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/webapp/RouterWebServices.java:
##
@@ -206,50 +181,33 @@ protected Map
getPipelines() {
*/
@VisibleForTesting
protected RESTRequestInterceptor createRequestInterceptorChain() {
+RESTRequestInterceptor pipeline = null;
Review Comment:
This code is very similar to the other one.
As we are refactoring, can we clean this up?
##
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router/src/main/java/org/apache/hadoop/yarn/server/router/clientrm/RouterClientRMService.java:
##
@@ -507,42 +480,25 @@
[GitHub] [hadoop] goiri commented on a diff in pull request #4531: HDFS-13274. RBF: Extend RouterRpcClient to use multiple sockets
goiri commented on code in PR #4531:
URL: https://github.com/apache/hadoop/pull/4531#discussion_r926048644
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/ConnectionPool.java:
##
@@ -210,24 +217,22 @@ public AtomicInteger getClientIndex() {
* @return Connection context.
*/
protected ConnectionContext getConnection() {
-
this.lastActiveTime = Time.now();
-
-// Get a connection from the pool following round-robin
-ConnectionContext conn = null;
List tmpConnections = this.connections;
-int size = tmpConnections.size();
-// Inc and mask off sign bit, lookup index should be non-negative int
-int threadIndex = this.clientIndex.getAndIncrement() & 0x7FFF;
-for (int i=0; i
[jira] [Resolved] (HADOOP-17948) JAR in conflict with timestamp check causes AM errors
[
https://issues.apache.org/jira/browse/HADOOP-17948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HADOOP-17948.
Resolution: Duplicate
I'm closing this as a duplicate of YARN-3606. I don't expect there will be
changes made in this area, because the timestamp check has worked well for a
fast and lightweight check of unexpected resource changes. YARN-3606 contains
more discussion.
> JAR in conflict with timestamp check causes AM errors
> -
>
> Key: HADOOP-17948
> URL: https://issues.apache.org/jira/browse/HADOOP-17948
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.9.2
>Reporter: Michael Taylor
>Priority: Blocker
>
> After an init action pulls down a new JAR and the check of a JAR's timestamp
> is performed [1]we can sometimes cause an incorrect error if the timestamp
> does not match. In order to address this you can perform workarounds like:
> record old timestamp at the beginning before the connector is changed
> local -r old_file_time=$(date -r ${dataproc_common_lib_dir}/gcs-connector.jar
> "+%m%d%H%M.00")
> # at end of script.
> touch -t "${old_file_time}"
> touch -h -t "${old_file_time}"
> We should instead of checking the date be comparing version compatibility
> tests.
>
>
> 1.
> https://github.com/apache/hadoop/blob/release-2.7.3-RC2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/FSDownload.java#L255-L258
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #4598: HDFS-16672. Fix lease interval comparison in BlockReportLeaseManager
hadoop-yetus commented on PR #4598:
URL: https://github.com/apache/hadoop/pull/4598#issuecomment-1190691175
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 36s | | trunk passed |
| +1 :green_heart: | compile | 1m 43s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 1m 40s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 24s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 48s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 26s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 1m 29s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 1m 29s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 1s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 56s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 55s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 28s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 18s | | the patch passed |
| +1 :green_heart: | shadedclient | 23m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 248m 21s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 15s | | The patch does not
generate ASF License warnings. |
| | | 359m 40s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4598/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4598 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4e1fa9a68002 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4781ea4e8673838fc7990955de10c6d8573a8c4f |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4598/1/testReport/ |
| Max. process+thread count | 3295 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4598/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL
[GitHub] [hadoop] hadoop-yetus commented on pull request #4601: HDFS-16467. Ensure Protobuf generated headers are included first
hadoop-yetus commented on PR #4601:
URL: https://github.com/apache/hadoop/pull/4601#issuecomment-1190664601
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 37m 47s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 7 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 52s | | trunk passed |
| +1 :green_heart: | compile | 3m 55s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 63m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 24s | | the patch passed |
| -1 :x: | compile | 2m 15s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4601/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-native-client in the patch failed. |
| -1 :x: | cc | 2m 15s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4601/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-native-client in the patch failed. |
| -1 :x: | golang | 2m 15s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4601/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-native-client in the patch failed. |
| -1 :x: | javac | 2m 15s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4601/1/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-native-client in the patch failed. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 18m 52s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 2m 25s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4601/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client.txt)
| hadoop-hdfs-native-client in the patch failed. |
| +1 :green_heart: | asflicense | 0m 53s | | The patch does not
generate ASF License warnings. |
| | | 129m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4601/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4601 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 32abce4b370b 4.15.0-58-generic #64-Ubuntu SMP Tue Aug 6
11:12:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 517433f93e26e380b64145af0f24afb91c469a14 |
| Default Java | Red Hat, Inc.-1.8.0_332-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4601/1/testReport/ |
| Max. process+thread count | 554 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4601/1/console |
| versions | git=2.9.5 maven=3.6.3 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail:
[jira] [Work logged] (HADOOP-17461) Add thread-level IOStatistics Context
[ https://issues.apache.org/jira/browse/HADOOP-17461?focusedWorklogId=793393=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793393 ] ASF GitHub Bot logged work on HADOOP-17461: --- Author: ASF GitHub Bot Created on: 20/Jul/22 18:58 Start Date: 20/Jul/22 18:58 Worklog Time Spent: 10m Work Description: mehakmeet commented on PR #4352: URL: https://github.com/apache/hadoop/pull/4352#issuecomment-1190642189 added your latest commit + Testing for full suite done for both enabled and disabled `fs.s3a.committer.experimental.collect.iostatistics` too. Issue Time Tracking --- Worklog Id: (was: 793393) Time Spent: 6h 50m (was: 6h 40m) > Add thread-level IOStatistics Context > - > > Key: HADOOP-17461 > URL: https://issues.apache.org/jira/browse/HADOOP-17461 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs, fs/azure, fs/s3 >Affects Versions: 3.3.1 >Reporter: Steve Loughran >Assignee: Mehakmeet Singh >Priority: Major > Labels: pull-request-available > Time Spent: 6h 50m > Remaining Estimate: 0h > > For effective reporting of the iostatistics of individual worker threads, we > need a thread-level context which IO components update. > * this contact needs to be passed in two background thread forming work on > behalf of a task. > * IO Components (streams, iterators, filesystems) need to update this context > statistics as they perform work > * Without double counting anything. > I imagine a ThreadLocal IOStatisticContext which will be updated in the > FileSystem API Calls. This context MUST be passed into the background threads > used by a task, so that IO is correctly aggregated. > I don't want streams, listIterators to do the updating as there is more > risk of double counting. However, we need to see their statistics if we want > to know things like "bytes discarded in backwards seeks". And I don't want to > be updating a shared context object on every read() call. > If all we want is store IO (HEAD, GET, DELETE, list performance etc) then the > FS is sufficient. > If we do want the stream-specific detail, then I propose > * caching the context in the constructor > * updating it only in close() or unbuffer() (as we do from S3AInputStream to > S3AInstrumenation) > * excluding those we know the FS already collects. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on pull request #4352: HADOOP-17461. Thread-level IOStatistics in S3A
mehakmeet commented on PR #4352: URL: https://github.com/apache/hadoop/pull/4352#issuecomment-1190642189 added your latest commit + Testing for full suite done for both enabled and disabled `fs.s3a.committer.experimental.collect.iostatistics` too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-17461) Add thread-level IOStatistics Context
[
https://issues.apache.org/jira/browse/HADOOP-17461?focusedWorklogId=793390=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793390
]
ASF GitHub Bot logged work on HADOOP-17461:
---
Author: ASF GitHub Bot
Created on: 20/Jul/22 18:52
Start Date: 20/Jul/22 18:52
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4566:
URL: https://github.com/apache/hadoop/pull/4566#issuecomment-1190636833
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 15s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 46s | | trunk passed |
| +1 :green_heart: | compile | 25m 9s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 52s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 13s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 24s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 6s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 37s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 25m 4s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 41s | | the patch passed |
| +1 :green_heart: | compile | 24m 20s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 20s | | the patch passed |
| +1 :green_heart: | compile | 21m 49s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 22s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/3/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 77 unchanged - 0 fixed = 79 total (was
77) |
| +1 :green_heart: | mvnsite | 3m 11s | | the patch passed |
| -1 :x: | javadoc | 1m 24s |
[/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/3/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
|
hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | javadoc | 2m 4s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 42s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 10s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 18s | | The patch does not
generate ASF License warnings. |
| | | 250m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/3/artifact/out/Dockerfile
|
| GITHUB PR |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4566: HADOOP-17461. IOStatisticsContext + committer integration
hadoop-yetus commented on PR #4566:
URL: https://github.com/apache/hadoop/pull/4566#issuecomment-1190636833
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 15s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 12s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 46s | | trunk passed |
| +1 :green_heart: | compile | 25m 9s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 52s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 29s | | trunk passed |
| +1 :green_heart: | mvnsite | 3m 13s | | trunk passed |
| +1 :green_heart: | javadoc | 2m 24s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 6s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 37s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 25m 4s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 27s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 41s | | the patch passed |
| +1 :green_heart: | compile | 24m 20s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 20s | | the patch passed |
| +1 :green_heart: | compile | 21m 49s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 22s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/3/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 77 unchanged - 0 fixed = 79 total (was
77) |
| +1 :green_heart: | mvnsite | 3m 11s | | the patch passed |
| -1 :x: | javadoc | 1m 24s |
[/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/3/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
|
hadoop-common-project_hadoop-common-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
with JDK Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | javadoc | 2m 4s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 50s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 44s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 42s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 10s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 18s | | The patch does not
generate ASF License warnings. |
| | | 250m 50s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4566/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4566 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f66999faca41 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ef9643c8b6ecb772481d3694e32930cef405fa55 |
[jira] [Work logged] (HADOOP-18330) S3AFileSystem removes Path when calling createS3Client
[
https://issues.apache.org/jira/browse/HADOOP-18330?focusedWorklogId=793385=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793385
]
ASF GitHub Bot logged work on HADOOP-18330:
---
Author: ASF GitHub Bot
Created on: 20/Jul/22 18:35
Start Date: 20/Jul/22 18:35
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4572:
URL: https://github.com/apache/hadoop/pull/4572#issuecomment-1190622231
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 5s | | trunk passed |
| +1 :green_heart: | compile | 1m 3s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 27s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 44s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not
generate ASF License warnings. |
| | | 102m 46s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4572/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4572 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 32b996b7804d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1de79b6766c59f26890a2968c83a7e0ad85cf9e4 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4572/3/testReport/ |
| Max. process+thread count | 719 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4572/3/console |
| versions | git=2.25.1
[GitHub] [hadoop] hadoop-yetus commented on pull request #4572: HADOOP-18330-S3AFileSystem removes Path when calling createS3Client
hadoop-yetus commented on PR #4572:
URL: https://github.com/apache/hadoop/pull/4572#issuecomment-1190622231
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 42m 5s | | trunk passed |
| +1 :green_heart: | compile | 1m 3s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 50s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 45s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 41s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 31s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 27s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 23s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 13s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 26s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 2m 44s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not
generate ASF License warnings. |
| | | 102m 46s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4572/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4572 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 32b996b7804d 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1de79b6766c59f26890a2968c83a7e0ad85cf9e4 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4572/3/testReport/ |
| Max. process+thread count | 719 (vs. ulimit of 5500) |
| modules | C: hadoop-tools/hadoop-aws U: hadoop-tools/hadoop-aws |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4572/3/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please
[GitHub] [hadoop] hadoop-yetus commented on pull request #4599: YARN-11211. QueueMetrics leaks Configuration objects when validation …
hadoop-yetus commented on PR #4599:
URL: https://github.com/apache/hadoop/pull/4599#issuecomment-1190594301
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 41s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 39m 41s | | trunk passed |
| +1 :green_heart: | compile | 1m 20s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 1m 6s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 2s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 15s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 2s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | compile | 1m 2s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 2s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 53s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 5s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 52s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 47s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 14s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 100m 41s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 207m 34s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4599/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4599 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 5360a832303a 4.15.0-156-generic #163-Ubuntu SMP Thu Aug 19
23:31:58 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 26dbcc0780a87f4c6b0d913d56b50a9638ae0a7b |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4599/1/testReport/ |
| Max. process+thread count | 967 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4599/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
[GitHub] [hadoop] hadoop-yetus commented on pull request #4597: HDFS-16671. RBF: RouterRpcFairnessPolicyController supports configurable permit acquire timeout
hadoop-yetus commented on PR #4597:
URL: https://github.com/apache/hadoop/pull/4597#issuecomment-1190583416
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 43s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 30s | | trunk passed |
| +1 :green_heart: | compile | 0m 58s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 54s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 51s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 3s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 25s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 41s | | the patch passed |
| +1 :green_heart: | compile | 0m 46s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 46s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 24s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 40s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 59s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 29s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 35s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 21m 56s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4597/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 54s | | The patch does not
generate ASF License warnings. |
| | | 120m 31s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.fairness.TestRouterRpcFairnessPolicyController |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4597/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux b5ac6535d8d3 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 300d8ee4cdc6e947826d469442a41c5cfe0534b7 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4597/2/testReport/ |
| Max. process+thread count | 2297 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4597/2/console |
| versions |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4596: HDFS-16670. Improve Code With Lambda in EditLogTailer class
hadoop-yetus commented on PR #4596:
URL: https://github.com/apache/hadoop/pull/4596#issuecomment-1190581579
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 50s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 38m 42s | | trunk passed |
| +1 :green_heart: | compile | 1m 43s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 1m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 45s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 24s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 47s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 57s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 24s | | the patch passed |
| +1 :green_heart: | compile | 1m 26s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 1m 26s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 0s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 28s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 33s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 3m 21s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 239m 21s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4596/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 57s | | The patch does not
generate ASF License warnings. |
| | | 349m 23s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4596/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4596 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e69c6b62e0b4 4.15.0-112-generic #113-Ubuntu SMP Thu Jul 9
23:41:39 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / a256d53000663c8fdab5a144f9275180211bd271 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4596/1/testReport/ |
| Max. process+thread count | 2979 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4596/1/console |
| versions |
[GitHub] [hadoop] hadoop-yetus commented on pull request #4595: YARN-11158. Support getDelegationToken, renewDelegationToken, cancelDelegationToken API's for Federation
hadoop-yetus commented on PR #4595:
URL: https://github.com/apache/hadoop/pull/4595#issuecomment-1190574210
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 33s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +0 :ok: | buf | 0m 1s | | buf was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 13s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 48s | | trunk passed |
| +1 :green_heart: | compile | 25m 36s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 24m 9s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 5m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 6m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 58s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 15s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 8m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 32s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 26s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 3m 1s | | the patch passed |
| +1 :green_heart: | compile | 24m 32s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | cc | 24m 32s | | the patch passed |
| -1 :x: | javac | 24m 32s |
[/results-compile-javac-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4595/2/artifact/out/results-compile-javac-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 12 new + 2880 unchanged - 0
fixed = 2892 total (was 2880) |
| +1 :green_heart: | compile | 21m 54s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | cc | 21m 54s | | the patch passed |
| -1 :x: | javac | 21m 54s |
[/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4595/2/artifact/out/results-compile-javac-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 12 new + 2678
unchanged - 0 fixed = 2690 total (was 2678) |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 32s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4595/2/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 169 unchanged - 2 fixed = 171 total (was
171) |
| +1 :green_heart: | mvnsite | 5m 35s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 44s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 14s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | spotbugs | 1m 32s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4595/2/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.html)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router
generated 4 new + 0 unchanged - 0 fixed = 4 total (was 0) |
| +1 :green_heart: | shadedclient | 24m 48s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: |
[jira] [Work logged] (HADOOP-12007) GzipCodec native CodecPool leaks memory
[ https://issues.apache.org/jira/browse/HADOOP-12007?focusedWorklogId=793359=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793359 ] ASF GitHub Bot logged work on HADOOP-12007: --- Author: ASF GitHub Bot Created on: 20/Jul/22 17:44 Start Date: 20/Jul/22 17:44 Worklog Time Spent: 10m Work Description: sunchao commented on PR #4585: URL: https://github.com/apache/hadoop/pull/4585#issuecomment-1190572837 > @sunchao Should I create a new Jira ticket for this? @kevins-29 , yes please create a new JIRA for this issue, and change the PR title afterwards. Thanks. Issue Time Tracking --- Worklog Id: (was: 793359) Time Spent: 2h 50m (was: 2h 40m) > GzipCodec native CodecPool leaks memory > --- > > Key: HADOOP-12007 > URL: https://issues.apache.org/jira/browse/HADOOP-12007 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Yejun Yang >Priority: Major > Labels: pull-request-available > Time Spent: 2h 50m > Remaining Estimate: 0h > > org/apache/hadoop/io/compress/GzipCodec.java call > CompressionCodec.Util.createOutputStreamWithCodecPool to use CodecPool. But > compressor objects are actually never returned to pool which cause memory > leak. > HADOOP-10591 uses CompressionOutputStream.close() to return Compressor object > to pool. But CompressionCodec.Util.createOutputStreamWithCodecPool actually > returns a CompressorStream which overrides close(). > This cause CodecPool.returnCompressor never being called. In my log file I > can see lots of "Got brand-new compressor [.gz]" but no "Got recycled > compressor". -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18353) HEAD OBJECT returns only 400 BAD REQUEST when token is expired
[
https://issues.apache.org/jira/browse/HADOOP-18353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569137#comment-17569137
]
Mukund Thakur commented on HADOOP-18353:
CC [~dannycjones] [~ste...@apache.org] [~mehakmeetSingh]
> HEAD OBJECT returns only 400 BAD REQUEST when token is expired
> --
>
> Key: HADOOP-18353
> URL: https://issues.apache.org/jira/browse/HADOOP-18353
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
>Affects Versions: 3.3.3
>Reporter: Mukund Thakur
>Priority: Major
>
> I tried reproducing this today by changing this test
> [https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ATemporaryCredentials.java#L116]
> . Getting a session token for 15 mins and trying every one mins and finally
> it fails after 15 mins.
>
> Looks like the AWS SDK is not having the Expired Token error message as I
> could see the same in access logs but I see BadRequest on the SDK logs.
>
> *S3A Connector logs with SDK debug enabled.*
> 2022-07-13 15:44:15,318 [JUnit-testSTS] DEBUG s3a.AWSCredentialProviderList
> (AWSCredentialProviderList.java:getCredentials(184)) - Using credentials from
> TemporaryAWSCredentialsProvider
> 2022-07-13 15:44:15,319 [JUnit-testSTS] DEBUG amazonaws.request
> (AmazonHttpClient.java:executeOneRequest(1285)) - Sending Request: HEAD
> [https://mthakur-us-west-1.s3.us-west-1.amazonaws.com|https://mthakur-us-west-1.s3.us-west-1.amazonaws.com/]
> /test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e Headers:
> (amz-sdk-invocation-id: 41e6e504-1c2b-2701-09bb-ae692dff2515, Content-Type:
> application/octet-stream, Referer:
> [https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-0098/?op=op_create=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e=mthakur=46c6d232-80aa-4405-9e39-5df880932fdc=ca2778f8-085e-4d1f-aef3-73794869f275-0098=11=ca2778f8-085e-4d1f-aef3-73794869f275=11=1657745055318],
> User-Agent: Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7
> Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10
> vendor/Oracle_Corporation cfg/retry-mode/legacy, )
> 2022-07-13 15:44:15,623 [JUnit-testSTS] DEBUG amazonaws.request
> (AmazonHttpClient.java:handleErrorResponse(1846)) - \{*}Received error
> response: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request
> (Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request
> ID{*}: WMGQ0KC4MHEMZTQC; S3 Extended Request ID:
> IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=;
> Proxy: null), S3 Extended Request ID:
> IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=
> 2022-07-13 15:44:15,624 [JUnit-testSTS] DEBUG s3a.S3AFileSystem
> (S3AFileSystem.java:close(3814)) - Filesystem s3a://mthakur-us-west-1 is
> closed
> *AWS access logs*
> 183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
> mthakur-us-west-1 [13/Jul/2022:20:44:15 +] 67.79.115.98 -
> WMGQ0KC4MHEMZTQC REST.HEAD.OBJECT
> test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e "HEAD
> /test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e HTTP/1.1" *400
> ExpiredToken* 556 - 5 -
> "[https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-0098/?op=op_create=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e=mthakur=46c6d232-80aa-4405-9e39-5df880932fdc=ca2778f8-085e-4d1f-aef3-73794869f275-0098=11=ca2778f8-085e-4d1f-aef3-73794869f275=11=1657745055318];
> "Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7
> Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10
> vendor/Oracle_Corporation cfg/retry-mode/legacy" -
> IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=
> SigV4 ECDHE-RSA-AES128-SHA AuthHeader
> [mthakur-us-west-1.s3.us-west-1.amazonaws.com|http://mthakur-us-west-1.s3.us-west-1.amazonaws.com/]
> TLSv1.2 -
>
> I tested by running repeatedly ITestCustomSigner in S3A, and also just
> ListObjectsV2 on loop… I did just notice your test is failing with HEAD, and
> *I can reproduce* by running this after credential expiry.
>
> aws s3api head-object --bucket djonesoa-us-west-2 --region us-west-2 --key
> test-object –debug
>
> To summarise:
> * If I run ListObjectsV2, I get “400 ExpiredToken”{+}{+}{+}{+}
> * If I run HeadObject, I get “400 Bad Request”{+}{+}{+}{+}
> * If I run GetObject, I get “400 ExpiredToken”
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] sunchao commented on pull request #4585: HADOOP-12007. GzipCodec native CodecPool leaks memory
sunchao commented on PR #4585: URL: https://github.com/apache/hadoop/pull/4585#issuecomment-1190572837 > @sunchao Should I create a new Jira ticket for this? @kevins-29 , yes please create a new JIRA for this issue, and change the PR title afterwards. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-18353) HEAD OBJECT returns only 400 BAD REQUEST when token is expired
Mukund Thakur created HADOOP-18353:
--
Summary: HEAD OBJECT returns only 400 BAD REQUEST when token is
expired
Key: HADOOP-18353
URL: https://issues.apache.org/jira/browse/HADOOP-18353
Project: Hadoop Common
Issue Type: Sub-task
Components: fs/s3
Affects Versions: 3.3.3
Reporter: Mukund Thakur
I tried reproducing this today by changing this test
[https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3ATemporaryCredentials.java#L116]
. Getting a session token for 15 mins and trying every one mins and finally it
fails after 15 mins.
Looks like the AWS SDK is not having the Expired Token error message as I could
see the same in access logs but I see BadRequest on the SDK logs.
*S3A Connector logs with SDK debug enabled.*
2022-07-13 15:44:15,318 [JUnit-testSTS] DEBUG s3a.AWSCredentialProviderList
(AWSCredentialProviderList.java:getCredentials(184)) - Using credentials from
TemporaryAWSCredentialsProvider
2022-07-13 15:44:15,319 [JUnit-testSTS] DEBUG amazonaws.request
(AmazonHttpClient.java:executeOneRequest(1285)) - Sending Request: HEAD
[https://mthakur-us-west-1.s3.us-west-1.amazonaws.com|https://mthakur-us-west-1.s3.us-west-1.amazonaws.com/]
/test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e Headers:
(amz-sdk-invocation-id: 41e6e504-1c2b-2701-09bb-ae692dff2515, Content-Type:
application/octet-stream, Referer:
[https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-0098/?op=op_create=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e=mthakur=46c6d232-80aa-4405-9e39-5df880932fdc=ca2778f8-085e-4d1f-aef3-73794869f275-0098=11=ca2778f8-085e-4d1f-aef3-73794869f275=11=1657745055318],
User-Agent: Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7
Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10
vendor/Oracle_Corporation cfg/retry-mode/legacy, )
2022-07-13 15:44:15,623 [JUnit-testSTS] DEBUG amazonaws.request
(AmazonHttpClient.java:handleErrorResponse(1846)) - \{*}Received error
response: com.amazonaws.services.s3.model.AmazonS3Exception: Bad Request
(Service: Amazon S3; Status Code: 400; Error Code: 400 Bad Request; Request
ID{*}: WMGQ0KC4MHEMZTQC; S3 Extended Request ID:
IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=;
Proxy: null), S3 Extended Request ID:
IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=
2022-07-13 15:44:15,624 [JUnit-testSTS] DEBUG s3a.S3AFileSystem
(S3AFileSystem.java:close(3814)) - Filesystem s3a://mthakur-us-west-1 is closed
*AWS access logs*
183c9826b45486e485693808f38e2c4071004bf5dfd4c3ab210f0a21a4235ef8
mthakur-us-west-1 [13/Jul/2022:20:44:15 +] 67.79.115.98 - WMGQ0KC4MHEMZTQC
REST.HEAD.OBJECT test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e "HEAD
/test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e HTTP/1.1" *400 ExpiredToken*
556 - 5 -
"[https://audit.example.org/hadoop/1/op_create/ca2778f8-085e-4d1f-aef3-73794869f275-0098/?op=op_create=test/testSTS/040112e1-d954-46d9-9def-aedd297bd42e=mthakur=46c6d232-80aa-4405-9e39-5df880932fdc=ca2778f8-085e-4d1f-aef3-73794869f275-0098=11=ca2778f8-085e-4d1f-aef3-73794869f275=11=1657745055318];
"Hadoop 3.4.0-SNAPSHOT, aws-sdk-java/1.12.132 Mac_OS_X/10.15.7
Java_HotSpot(TM)_64-Bit_Server_VM/25.161-b12 java/1.8.0_161 kotlin/1.4.10
vendor/Oracle_Corporation cfg/retry-mode/legacy" -
IztdwNq71aWBYavfaj8rV5b/Y0GzV4tqJBEVDSdZH+RRR3B1vUVIMV0qWez9ulBrjDM1GQxeT1Q=
SigV4 ECDHE-RSA-AES128-SHA AuthHeader
[mthakur-us-west-1.s3.us-west-1.amazonaws.com|http://mthakur-us-west-1.s3.us-west-1.amazonaws.com/]
TLSv1.2 -
I tested by running repeatedly ITestCustomSigner in S3A, and also just
ListObjectsV2 on loop… I did just notice your test is failing with HEAD, and *I
can reproduce* by running this after credential expiry.
aws s3api head-object --bucket djonesoa-us-west-2 --region us-west-2 --key
test-object –debug
To summarise:
* If I run ListObjectsV2, I get “400 ExpiredToken”{+}{+}{+}{+}
* If I run HeadObject, I get “400 Bad Request”{+}{+}{+}{+}
* If I run GetObject, I get “400 ExpiredToken”
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra opened a new pull request, #4601: HDFS-16467. Ensure Protobuf generated headers are included first
GauthamBanasandra opened a new pull request, #4601: URL: https://github.com/apache/hadoop/pull/4601 * This PR ensures that the Protobuf generated headers are always included first, even when these headers are included transitively. * This problem is specific to Windows only. ### Description of PR We need to ensure that the Protobuf generated headers ([such as ClientNamenodeProtocol.pb.h](https://github.com/apache/hadoop/blob/cce5a6f6094cefd2e23b73d202cc173cf4fc2cc5/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfspp/lib/connection/datanodeconnection.h#L23)) are included at the top. In other words, `*.ph.h` should be the first header files to be included in any of the .c/.cc/.h files. Otherwise, it causes symbol redefinition errors during compilation. Also, we need to ensure that Protobuf generated header files are the first ones to be included even in the case of `transitive inclusion` of header files. This issue seems to be specific to Windows only. Not sure why the other platforms aren't running into this. ### How was this patch tested? 1. Tested this locally by building on my Windows system. 2. Hadoop Jenkins CI validation. ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18351) S3A prefetching: Error logging during reads
[ https://issues.apache.org/jira/browse/HADOOP-18351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569118#comment-17569118 ] Steve Loughran commented on HADOOP-18351: - bq. but: we do want the most recent failure to be raised on the next read() on the stream when there is no data in the cache. yes. doesn't mean that the stream should be too noisy. some errror messages are good, but stacks can be a distraction, especially on transient failures > S3A prefetching: Error logging during reads > --- > > Key: HADOOP-18351 > URL: https://issues.apache.org/jira/browse/HADOOP-18351 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ahmar Suhail >Priority: Minor > > Look at how errors during read are logged, current implementation could flood > logs with stack traces on failures. > proposed > * errors in prefetch only logged at info with error text but not stack > * full stack logged at debug > but: we do want the most recent failure to be raised on the next read() on > the stream when there is no data in the cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18351) S3A prefetching: Error logging during reads
[ https://issues.apache.org/jira/browse/HADOOP-18351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated HADOOP-18351: Summary: S3A prefetching: Error logging during reads (was: Error logging during reads) > S3A prefetching: Error logging during reads > --- > > Key: HADOOP-18351 > URL: https://issues.apache.org/jira/browse/HADOOP-18351 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Ahmar Suhail >Priority: Minor > > Look at how errors during read are logged, current implementation could flood > logs with stack traces on failures. > proposed > * errors in prefetch only logged at info with error text but not stack > * full stack logged at debug > but: we do want the most recent failure to be raised on the next read() on > the stream when there is no data in the cache. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18333) hadoop-client-runtime impact by CVE-2022-2047 due to shaded jetty
[ https://issues.apache.org/jira/browse/HADOOP-18333?focusedWorklogId=793338=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793338 ] ASF GitHub Bot logged work on HADOOP-18333: --- Author: ASF GitHub Bot Created on: 20/Jul/22 16:27 Start Date: 20/Jul/22 16:27 Worklog Time Spent: 10m Work Description: jojochuang opened a new pull request, #4600: URL: https://github.com/apache/hadoop/pull/4600 Cherry picked #4553 to branch-3.3 Issue Time Tracking --- Worklog Id: (was: 793338) Time Spent: 1.5h (was: 1h 20m) > hadoop-client-runtime impact by CVE-2022-2047 due to shaded jetty > - > > Key: HADOOP-18333 > URL: https://issues.apache.org/jira/browse/HADOOP-18333 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.3 >Reporter: phoebe chen >Assignee: groot >Priority: Major > Labels: pull-request-available > Time Spent: 1.5h > Remaining Estimate: 0h > > CVE-2022-2047 is recently found for Eclipse Jetty, and impacts 9.4.0 thru > 9.4.46. > In latest 3.3.3 of hadoop-client-runtime, it shaded 9.4.43.v20210629 version > jetty which is impacted. > In Trunk, Jetty is in version 9.4.44.v20210927, which is still impacted. > Need to upgrade Jetty Version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang opened a new pull request, #4600: HADOOP-18333. Upgrade jetty version to 9.4.48.v20220622
jojochuang opened a new pull request, #4600: URL: https://github.com/apache/hadoop/pull/4600 Cherry picked #4553 to branch-3.3 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] GauthamBanasandra merged pull request #4576: HDFS-16667. Use malloc for buffer allocation in uriparser2
GauthamBanasandra merged PR #4576: URL: https://github.com/apache/hadoop/pull/4576 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #4560: HDFS-16659. JournalNode should throw CacheMissException when SinceTxId is bigger than HighestWrittenTxId
ZanderXu commented on PR #4560: URL: https://github.com/apache/hadoop/pull/4560#issuecomment-1190487791 @jojochuang @goiri Can you help me review this patch? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #4518: HDFS-16645. [JN] Bugfix: java.lang.IllegalStateException: Invalid log manifest
ZanderXu commented on PR #4518: URL: https://github.com/apache/hadoop/pull/4518#issuecomment-1190487355 @jojochuang @goiri Can you help me review this patch? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #4360: HDFS-16596. Improve the processing capability of FsDatasetAsyncDiskSe…
ZanderXu commented on PR #4360: URL: https://github.com/apache/hadoop/pull/4360#issuecomment-1190486645 @jojochuang @goiri Can you help me reivew this patch? Thanks. @saintstack ping, can you help me reivew this patch again? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dependabot[bot] closed pull request #4535: Bump jetty-server from 9.4.44.v20210927 to 10.0.10 in /hadoop-project
dependabot[bot] closed pull request #4535: Bump jetty-server from 9.4.44.v20210927 to 10.0.10 in /hadoop-project URL: https://github.com/apache/hadoop/pull/4535 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] dependabot[bot] commented on pull request #4535: Bump jetty-server from 9.4.44.v20210927 to 10.0.10 in /hadoop-project
dependabot[bot] commented on PR #4535: URL: https://github.com/apache/hadoop/pull/4535#issuecomment-1190484430 Looks like org.eclipse.jetty:jetty-server is up-to-date now, so this is no longer needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18333) hadoop-client-runtime impact by CVE-2022-2047 due to shaded jetty
[ https://issues.apache.org/jira/browse/HADOOP-18333?focusedWorklogId=793331=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793331 ] ASF GitHub Bot logged work on HADOOP-18333: --- Author: ASF GitHub Bot Created on: 20/Jul/22 16:15 Start Date: 20/Jul/22 16:15 Worklog Time Spent: 10m Work Description: jojochuang merged PR #4553: URL: https://github.com/apache/hadoop/pull/4553 Issue Time Tracking --- Worklog Id: (was: 793331) Time Spent: 1h 20m (was: 1h 10m) > hadoop-client-runtime impact by CVE-2022-2047 due to shaded jetty > - > > Key: HADOOP-18333 > URL: https://issues.apache.org/jira/browse/HADOOP-18333 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 3.3.3 >Reporter: phoebe chen >Assignee: groot >Priority: Major > Labels: pull-request-available > Time Spent: 1h 20m > Remaining Estimate: 0h > > CVE-2022-2047 is recently found for Eclipse Jetty, and impacts 9.4.0 thru > 9.4.46. > In latest 3.3.3 of hadoop-client-runtime, it shaded 9.4.43.v20210629 version > jetty which is impacted. > In Trunk, Jetty is in version 9.4.44.v20210927, which is still impacted. > Need to upgrade Jetty Version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] jojochuang merged pull request #4553: HADOOP-18333.Upgrade jetty version to 9.4.48.v20220622
jojochuang merged PR #4553: URL: https://github.com/apache/hadoop/pull/4553 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18330) S3AFileSystem removes Path when calling createS3Client
[
https://issues.apache.org/jira/browse/HADOOP-18330?focusedWorklogId=793330=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793330
]
ASF GitHub Bot logged work on HADOOP-18330:
---
Author: ASF GitHub Bot
Created on: 20/Jul/22 16:14
Start Date: 20/Jul/22 16:14
Worklog Time Spent: 10m
Work Description: steveloughran commented on code in PR #4572:
URL: https://github.com/apache/hadoop/pull/4572#discussion_r925806836
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ClientFactory.java:
##
@@ -264,5 +270,26 @@ public S3ClientCreationParameters withHeader(
public Map getHeaders() {
return headers;
}
+
+/**
+ * Get the full s3 path.
+ * added in HADOOP-18330
+ * @return path URI
+ */
+public URI getPath() {
Review Comment:
can you name the getter/setter to pathUri for consistency?
Issue Time Tracking
---
Worklog Id: (was: 793330)
Time Spent: 3.5h (was: 3h 20m)
> S3AFileSystem removes Path when calling createS3Client
> --
>
> Key: HADOOP-18330
> URL: https://issues.apache.org/jira/browse/HADOOP-18330
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/s3
>Affects Versions: 3.3.0, 3.3.1, 3.3.2, 3.3.3
>Reporter: Ashutosh Pant
>Assignee: Ashutosh Pant
>Priority: Minor
> Labels: pull-request-available
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> when using hadoop and spark to read/write data from an s3 bucket like ->
> s3a://bucket/path and using a custom Credentials Provider, the path is
> removed from the s3a URI and the credentials provider fails because the full
> path is gone.
> In Spark 3.2,
> It was invoked as -> s3 = ReflectionUtils.newInstance(s3ClientFactoryClass,
> conf)
> .createS3Client(name, bucket, credentials);
> But In spark 3.3.3
> It is invoked as s3 = ReflectionUtils.newInstance(s3ClientFactoryClass,
> conf).createS3Client(getUri(), parameters);
> the getUri() removes the path from the s3a URI
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on a diff in pull request #4572: HADOOP-18330-S3AFileSystem removes Path when calling createS3Client
steveloughran commented on code in PR #4572:
URL: https://github.com/apache/hadoop/pull/4572#discussion_r925806836
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3ClientFactory.java:
##
@@ -264,5 +270,26 @@ public S3ClientCreationParameters withHeader(
public Map getHeaders() {
return headers;
}
+
+/**
+ * Get the full s3 path.
+ * added in HADOOP-18330
+ * @return path URI
+ */
+public URI getPath() {
Review Comment:
can you name the getter/setter to pathUri for consistency?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18190) s3a prefetching streams to collect iostats on prefetching operations
[
https://issues.apache.org/jira/browse/HADOOP-18190?focusedWorklogId=793329=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793329
]
ASF GitHub Bot logged work on HADOOP-18190:
---
Author: ASF GitHub Bot
Created on: 20/Jul/22 16:12
Start Date: 20/Jul/22 16:12
Worklog Time Spent: 10m
Work Description: steveloughran commented on code in PR #4458:
URL: https://github.com/apache/hadoop/pull/4458#discussion_r925717550
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/StreamStatisticNames.java:
##
@@ -393,6 +393,12 @@ public final class StreamStatisticNames {
public static final String STREAM_READ_PREFETCH_OPERATIONS
= "stream_read_prefetch_operations";
+ /**
+ * Total number of failed prefetching operations.
+ */
+ public static final String STREAM_READ_FAILED_PREFETCH_OPERATIONS
Review Comment:
not needed. every duration type you build for a store automatically gets
.failures stats entries (count, min, mean, max), use
StoreStatisticNames.SUFFIX_FAILURES if you want to see usages
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3Reader.java:
##
@@ -95,26 +102,24 @@ public void close() {
private int readOneBlockWithRetries(ByteBuffer buffer, long offset, int size)
throws IOException {
-this.s3File.getStatistics().readOperationStarted(offset, size);
+this.streamStatistics.readOperationStarted(offset, size);
Invoker invoker = this.s3File.getReadInvoker();
-int invokerResponse = invoker.retry(
-"read", this.s3File.getPath(), true,
-() -> {
+int invokerResponse = invoker.retry("read", this.s3File.getPath(), true,
+trackDurationOfOperation(streamStatistics,
STREAM_READ_REMOTE_BLOCK_READ, () -> {
Review Comment:
FYI, this will do the right thing about updating the stream statistics with
the pass/fail stats differentiated
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/TestBufferPool.java:
##
@@ -37,28 +38,33 @@ public class TestBufferPool extends AbstractHadoopTestBase {
@Test
public void testArgChecks() throws Exception {
// Should not throw.
-BufferPool pool = new BufferPool(POOL_SIZE, BUFFER_SIZE);
+BufferPool pool = new BufferPool(POOL_SIZE, BUFFER_SIZE,
+EmptyS3AStatisticsContext.EMPTY_INPUT_STREAM_STATISTICS);
// Verify it throws correctly.
ExceptionAsserts.assertThrows(
IllegalArgumentException.class,
"'size' must be a positive integer",
-() -> new BufferPool(0, 10));
+() -> new BufferPool(0, 10,
EmptyS3AStatisticsContext.EMPTY_INPUT_STREAM_STATISTICS));
Review Comment:
use newInputStreamStatistics() which is part of S3AStatisticsContext.
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:
##
@@ -0,0 +1,59 @@
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.fs.common;
+
+import java.time.Duration;
+
+public final class EmptyPrefetchingStatistics implements PrefetchingStatistics
{
Review Comment:
add a static final instance of the class; make constructor private. no need
to create > 1
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/PrefetchingStatistics.java:
##
@@ -0,0 +1,41 @@
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either
[GitHub] [hadoop] steveloughran commented on a diff in pull request #4458: HADOOP-18190. Adds iostats for prefetching
steveloughran commented on code in PR #4458:
URL: https://github.com/apache/hadoop/pull/4458#discussion_r925717550
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/statistics/StreamStatisticNames.java:
##
@@ -393,6 +393,12 @@ public final class StreamStatisticNames {
public static final String STREAM_READ_PREFETCH_OPERATIONS
= "stream_read_prefetch_operations";
+ /**
+ * Total number of failed prefetching operations.
+ */
+ public static final String STREAM_READ_FAILED_PREFETCH_OPERATIONS
Review Comment:
not needed. every duration type you build for a store automatically gets
.failures stats entries (count, min, mean, max), use
StoreStatisticNames.SUFFIX_FAILURES if you want to see usages
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3Reader.java:
##
@@ -95,26 +102,24 @@ public void close() {
private int readOneBlockWithRetries(ByteBuffer buffer, long offset, int size)
throws IOException {
-this.s3File.getStatistics().readOperationStarted(offset, size);
+this.streamStatistics.readOperationStarted(offset, size);
Invoker invoker = this.s3File.getReadInvoker();
-int invokerResponse = invoker.retry(
-"read", this.s3File.getPath(), true,
-() -> {
+int invokerResponse = invoker.retry("read", this.s3File.getPath(), true,
+trackDurationOfOperation(streamStatistics,
STREAM_READ_REMOTE_BLOCK_READ, () -> {
Review Comment:
FYI, this will do the right thing about updating the stream statistics with
the pass/fail stats differentiated
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/TestBufferPool.java:
##
@@ -37,28 +38,33 @@ public class TestBufferPool extends AbstractHadoopTestBase {
@Test
public void testArgChecks() throws Exception {
// Should not throw.
-BufferPool pool = new BufferPool(POOL_SIZE, BUFFER_SIZE);
+BufferPool pool = new BufferPool(POOL_SIZE, BUFFER_SIZE,
+EmptyS3AStatisticsContext.EMPTY_INPUT_STREAM_STATISTICS);
// Verify it throws correctly.
ExceptionAsserts.assertThrows(
IllegalArgumentException.class,
"'size' must be a positive integer",
-() -> new BufferPool(0, 10));
+() -> new BufferPool(0, 10,
EmptyS3AStatisticsContext.EMPTY_INPUT_STREAM_STATISTICS));
Review Comment:
use newInputStreamStatistics() which is part of S3AStatisticsContext.
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/common/EmptyPrefetchingStatistics.java:
##
@@ -0,0 +1,59 @@
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.fs.common;
+
+import java.time.Duration;
+
+public final class EmptyPrefetchingStatistics implements PrefetchingStatistics
{
Review Comment:
add a static final instance of the class; make constructor private. no need
to create > 1
##
hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/common/PrefetchingStatistics.java:
##
@@ -0,0 +1,41 @@
+ /*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.hadoop.fs.common;
+
+import java.time.Duration;
+
+import org.apache.hadoop.fs.statistics.IOStatisticsSource;
+
+public interface PrefetchingStatistics extends IOStatisticsSource {
Review Comment:
needs javadocs i'm afraid
##
[GitHub] [hadoop] hadoop-yetus commented on pull request #4595: YARN-11158. Support getDelegationToken, renewDelegationToken, cancelDelegationToken API's for Federation
hadoop-yetus commented on PR #4595:
URL: https://github.com/apache/hadoop/pull/4595#issuecomment-1190475035
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 36s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 31s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 54s | | trunk passed |
| +1 :green_heart: | compile | 23m 18s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 45s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 5m 13s | | trunk passed |
| +1 :green_heart: | javadoc | 4m 35s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 2s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 7m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 2s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 28s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 2m 25s | | the patch passed |
| +1 :green_heart: | compile | 23m 10s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 23m 10s | | the patch passed |
| +1 :green_heart: | compile | 22m 43s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 4m 11s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4595/1/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 2 new + 169 unchanged - 2 fixed = 171 total (was
171) |
| +1 :green_heart: | mvnsite | 4m 55s | | the patch passed |
| +1 :green_heart: | javadoc | 4m 27s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 4m 1s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| -1 :x: | spotbugs | 1m 48s |
[/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4595/1/artifact/out/new-spotbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.html)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router
generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) |
| +1 :green_heart: | shadedclient | 22m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 46s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 1m 51s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4595/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt)
| hadoop-yarn-api in the patch passed. |
| -1 :x: | unit | 2m 51s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4595/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt)
| hadoop-yarn-server-router in the patch passed. |
| +1 :green_heart: | asflicense | 1m 36s | | The patch does not
generate ASF License warnings. |
| | | 261m 30s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-router
|
| | org.apache.hadoop.yarn.server.router.security.RouterDelegationKey
doesn't
[GitHub] [hadoop] hadoop-yetus commented on pull request #4375: HDFS-16605. Improve Code With Lambda in hadoop-hdfs-rbf moudle.
hadoop-yetus commented on PR #4375:
URL: https://github.com/apache/hadoop/pull/4375#issuecomment-1190444082
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 56s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 10 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 43m 5s | | trunk passed |
| +1 :green_heart: | compile | 0m 59s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 35s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 2m 11s | | trunk passed |
| +1 :green_heart: | shadedclient | 25m 24s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 25m 47s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 42s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 24s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4375/12/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-project/hadoop-hdfs-rbf: The patch generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | mvnsite | 0m 42s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 41s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 41m 24s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 153m 13s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4375/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4375 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux a9df7b414795 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 79d57f811d03c361567cd25995bc7a3b6f68d2c5 |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4375/12/testReport/ |
| Max. process+thread count | 2289 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
[jira] [Work logged] (HADOOP-18190) s3a prefetching streams to collect iostats on prefetching operations
[ https://issues.apache.org/jira/browse/HADOOP-18190?focusedWorklogId=793290=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793290 ] ASF GitHub Bot logged work on HADOOP-18190: --- Author: ASF GitHub Bot Created on: 20/Jul/22 14:57 Start Date: 20/Jul/22 14:57 Worklog Time Spent: 10m Work Description: steveloughran commented on PR #4458: URL: https://github.com/apache/hadoop/pull/4458#issuecomment-1190394031 look at `org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding#trackFunctionDuration` for an example of failure handling. the failed() call switches to the same stats key with the suffix .failures and updates those counters and durations Issue Time Tracking --- Worklog Id: (was: 793290) Time Spent: 2h 20m (was: 2h 10m) > s3a prefetching streams to collect iostats on prefetching operations > > > Key: HADOOP-18190 > URL: https://issues.apache.org/jira/browse/HADOOP-18190 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.4.0 >Reporter: Steve Loughran >Assignee: Ahmar Suhail >Priority: Major > Labels: pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > There is a lot more happening in reads, so there's a lot more data to collect > and publish in IO stats for us to view in a summary at the end of processes > as well as get from the stream while it is active. > Some useful ones would seem to be: > counters > * is in memory. using 0 or 1 here lets aggregation reports count total #of > memory cached files. > * prefetching operations executed > * errors during prefetching > gauges > * number of blocks in cache > * total size of blocks > * active prefetches > + active memory used > duration tracking count/min/max/ave > * time to fetch a block > * time queued before the actual fetch begins > * time a reader is blocked waiting for a block fetch to complete > and some info on cache use itself > * number of blocks discarded unread > * number of prefetched blocks later used > * number of backward seeks to a prefetched block > * number of forward seeks to a prefetched block > the key ones I care about are > # memory consumption > # can we determine if cache is working (reads with cache hit) and when it is > not (misses, wasted prefetches) > # time blocked on executors > The stats need to be accessible on a stream even when closed, and aggregated > into the FS. once we get per-thread stats contexts we can publish there too > and collect in worker threads for reporting in task commits -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #4458: HADOOP-18190. Adds iostats for prefetching
steveloughran commented on PR #4458: URL: https://github.com/apache/hadoop/pull/4458#issuecomment-1190394031 look at `org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding#trackFunctionDuration` for an example of failure handling. the failed() call switches to the same stats key with the suffix .failures and updates those counters and durations -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18190) s3a prefetching streams to collect iostats on prefetching operations
[
https://issues.apache.org/jira/browse/HADOOP-18190?focusedWorklogId=793289=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793289
]
ASF GitHub Bot logged work on HADOOP-18190:
---
Author: ASF GitHub Bot
Created on: 20/Jul/22 14:54
Start Date: 20/Jul/22 14:54
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4458:
URL: https://github.com/apache/hadoop/pull/4458#issuecomment-1190390179
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +0 :ok: | mvndep | 14m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 59s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | compile | 24m 57s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 41s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 27s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 3m 12s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | javadoc | 2m 23s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 5s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 40s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 24m 37s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 25m 5s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 43s | | the patch passed |
| +1 :green_heart: | compile | 24m 12s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 12s | | the patch passed |
| +1 :green_heart: | compile | 21m 42s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 15s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 5s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 13s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 7s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 16s | | The patch does not
generate ASF License warnings. |
| | | 264m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4458 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 195ef8bf1a6b 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
1d710f11205fc0c554de5b9e931828f4890312cd |
|
[GitHub] [hadoop] hadoop-yetus commented on pull request #4458: HADOOP-18190. Adds iostats for prefetching
hadoop-yetus commented on PR #4458:
URL: https://github.com/apache/hadoop/pull/4458#issuecomment-1190390179
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 17m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +0 :ok: | mvndep | 14m 33s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 27m 59s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | compile | 24m 57s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 41s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 27s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 3m 12s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | javadoc | 2m 23s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 5s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 40s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 24m 37s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 25m 5s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 25s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 43s | | the patch passed |
| +1 :green_heart: | compile | 24m 12s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 12s | | the patch passed |
| +1 :green_heart: | compile | 21m 42s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 42s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 12s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 15s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 5s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 25m 7s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 13s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 7s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 16s | | The patch does not
generate ASF License warnings. |
| | | 264m 26s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4458 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 195ef8bf1a6b 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
1d710f11205fc0c554de5b9e931828f4890312cd |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/7/testReport/ |
| Max. process+thread count | 1252 (vs. ulimit of 5500)
[jira] [Work logged] (HADOOP-18190) s3a prefetching streams to collect iostats on prefetching operations
[
https://issues.apache.org/jira/browse/HADOOP-18190?focusedWorklogId=793282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793282
]
ASF GitHub Bot logged work on HADOOP-18190:
---
Author: ASF GitHub Bot
Created on: 20/Jul/22 14:41
Start Date: 20/Jul/22 14:41
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4458:
URL: https://github.com/apache/hadoop/pull/4458#issuecomment-1190374310
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +0 :ok: | mvndep | 14m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 18s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | compile | 23m 3s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 27s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 22s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 3m 20s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | javadoc | 2m 33s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 2s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 42s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 23m 3s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 32s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 50s | | the patch passed |
| +1 :green_heart: | compile | 24m 41s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 41s | | the patch passed |
| +1 :green_heart: | compile | 22m 14s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 41s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 32s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 45s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 7s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 30s | | The patch does not
generate ASF License warnings. |
| | | 254m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4458 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 20d8139f0c38 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
1d710f11205fc0c554de5b9e931828f4890312cd |
|
[GitHub] [hadoop] hadoop-yetus commented on pull request #4458: HADOOP-18190. Adds iostats for prefetching
hadoop-yetus commented on PR #4458:
URL: https://github.com/apache/hadoop/pull/4458#issuecomment-1190374310
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 12m 48s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 6 new or modified test files. |
_ feature-HADOOP-18028-s3a-prefetch Compile Tests _ |
| +0 :ok: | mvndep | 14m 48s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 25m 18s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | compile | 23m 3s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 27s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 22s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | mvnsite | 3m 20s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | javadoc | 2m 33s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 2s | |
feature-HADOOP-18028-s3a-prefetch passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 4m 42s | |
feature-HADOOP-18028-s3a-prefetch passed |
| +1 :green_heart: | shadedclient | 23m 3s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 32s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 29s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 50s | | the patch passed |
| +1 :green_heart: | compile | 24m 41s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 41s | | the patch passed |
| +1 :green_heart: | compile | 22m 14s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 22m 14s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 4m 6s | | the patch passed |
| +1 :green_heart: | mvnsite | 3m 26s | | the patch passed |
| +1 :green_heart: | javadoc | 2m 41s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 2m 32s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 5m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 32s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 18m 45s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 3m 7s | | hadoop-aws in the patch passed.
|
| +1 :green_heart: | asflicense | 1m 30s | | The patch does not
generate ASF License warnings. |
| | | 254m 45s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4458 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell |
| uname | Linux 20d8139f0c38 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | feature-HADOOP-18028-s3a-prefetch /
1d710f11205fc0c554de5b9e931828f4890312cd |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4458/6/testReport/ |
| Max. process+thread count | 1349 (vs. ulimit of 5500)
[GitHub] [hadoop] hadoop-yetus commented on pull request #4597: HDFS-16671. RBF: RouterRpcFairnessPolicyController supports configurable permit acquire timeout
hadoop-yetus commented on PR #4597:
URL: https://github.com/apache/hadoop/pull/4597#issuecomment-1190372307
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 40m 9s | | trunk passed |
| +1 :green_heart: | compile | 0m 59s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 0m 52s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 0m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 1s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 1m 11s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 22m 0s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 44s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 0m 44s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 26s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 43s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 40s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 0m 57s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 1m 27s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 46s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 21m 49s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4597/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-rbf.txt)
| hadoop-hdfs-rbf in the patch passed. |
| +1 :green_heart: | asflicense | 0m 52s | | The patch does not
generate ASF License warnings. |
| | | 122m 43s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.federation.fairness.TestRouterRpcFairnessPolicyController |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4597/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4597 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 14bc3953b314 4.15.0-169-generic #177-Ubuntu SMP Thu Feb 3
10:50:38 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f965c90296fe2623d577c332105ef3454dc2667e |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4597/1/testReport/ |
| Max. process+thread count | 2793 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4597/1/console |
| versions |
[GitHub] [hadoop] 9uapaw opened a new pull request, #4599: YARN-11211. QueueMetrics leaks Configuration objects when validation …
9uapaw opened a new pull request, #4599: URL: https://github.com/apache/hadoop/pull/4599 …API is called multiple times ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18154) S3A Authentication to support WebIdentity
[ https://issues.apache.org/jira/browse/HADOOP-18154?focusedWorklogId=793276=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793276 ] ASF GitHub Bot logged work on HADOOP-18154: --- Author: ASF GitHub Bot Created on: 20/Jul/22 14:28 Start Date: 20/Jul/22 14:28 Worklog Time Spent: 10m Work Description: vikaskoppineedi commented on PR #4070: URL: https://github.com/apache/hadoop/pull/4070#issuecomment-1190358882 we need this feature, can somebody please help? Issue Time Tracking --- Worklog Id: (was: 793276) Time Spent: 3h 20m (was: 3h 10m) > S3A Authentication to support WebIdentity > - > > Key: HADOOP-18154 > URL: https://issues.apache.org/jira/browse/HADOOP-18154 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/s3 >Affects Versions: 2.10.1 >Reporter: Ju Clarysse >Assignee: Ju Clarysse >Priority: Major > Labels: pull-request-available > Time Spent: 3h 20m > Remaining Estimate: 0h > > We are using the latest version of > [delta-sharing|https://github.com/delta-io/delta-sharing] which takes > advantage of > [hadoop-aws|https://hadoop.apache.org/docs/current/hadoop-aws/tools/hadoop-aws/index.html] > (S3A) connector in [Hadoop release version > 2.10.1|https://github.com/apache/hadoop/tree/rel/release-2.10.1] to mount an > AWS S3 File System. In our particular setup, all services are operated in > Amazon Elastic Kubernetes Service (EKS) and need to comply to the AWS > security concept [IAM roles for service > accounts|https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html] > (IRSA). > As [Delta sharing S3 connection|https://github.com/delta-io/delta-sharing#s3] > doesn't offer any corresponding support, we patched hadoop-aws-2.10.1 to > address this need via a new credentials provider class > org.apache.hadoop.fs.s3a.OIDCTokenCredentialsProvider. We also upgraded > dependency aws-java-sdk-bundle to its latest version 1.12.167 as [AWS > WebIdentityTokenCredentialsProvider > class|https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/auth/WebIdentityTokenCredentialsProvider.html%E2%80%A6] > was not yet available in original version 1.11.271. > We believe that other delta-sharing users could benefit from this short-term > contribution. Then sooner or later, delta-sharing owners will have to upgrade > their project to a more recent version of hadoop-aws that is probably more > widely used. The effort to promote this change is probably low. > Additional note: AWS WebIdentityTokenCredentialsProvider class is directly > supported by Spark applications submitted with configuration properties > `spark.hadoop.fs.s3a.aws.credentials.provider`and > `spark.kubernetes.authenticate.submission.oauthToken` > ([doc|https://spark.apache.org/docs/latest/running-on-kubernetes.html#spark-properties]). > So bringing this support to Hadoop will primarily be interesting for > non-Spark users. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] vikaskoppineedi commented on pull request #4070: HADOOP-18154. S3A Authentication to support WebIdentity
vikaskoppineedi commented on PR #4070: URL: https://github.com/apache/hadoop/pull/4070#issuecomment-1190358882 we need this feature, can somebody please help? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-12007) GzipCodec native CodecPool leaks memory
[ https://issues.apache.org/jira/browse/HADOOP-12007?focusedWorklogId=793273=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793273 ] ASF GitHub Bot logged work on HADOOP-12007: --- Author: ASF GitHub Bot Created on: 20/Jul/22 14:22 Start Date: 20/Jul/22 14:22 Worklog Time Spent: 10m Work Description: kevins-29 commented on PR #4585: URL: https://github.com/apache/hadoop/pull/4585#issuecomment-1190351693 > Thanks @kevins-29 , the fix looks good to me. However, is this addressing the issue mentioned in [HADOOP-12007](https://issues.apache.org/jira/browse/HADOOP-12007)? my understanding is that the issue there is related to `CompressorStream` overrides `close` and doesn't return the compressor to the pool as result. @sunchao Should I create a new Jira ticket for this? Issue Time Tracking --- Worklog Id: (was: 793273) Time Spent: 2h 40m (was: 2.5h) > GzipCodec native CodecPool leaks memory > --- > > Key: HADOOP-12007 > URL: https://issues.apache.org/jira/browse/HADOOP-12007 > Project: Hadoop Common > Issue Type: Bug >Affects Versions: 2.7.0 >Reporter: Yejun Yang >Priority: Major > Labels: pull-request-available > Time Spent: 2h 40m > Remaining Estimate: 0h > > org/apache/hadoop/io/compress/GzipCodec.java call > CompressionCodec.Util.createOutputStreamWithCodecPool to use CodecPool. But > compressor objects are actually never returned to pool which cause memory > leak. > HADOOP-10591 uses CompressionOutputStream.close() to return Compressor object > to pool. But CompressionCodec.Util.createOutputStreamWithCodecPool actually > returns a CompressorStream which overrides close(). > This cause CodecPool.returnCompressor never being called. In my log file I > can see lots of "Got brand-new compressor [.gz]" but no "Got recycled > compressor". -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4596: HDFS-16670. Improve Code With Lambda in EditLogTailer class
slfan1989 commented on PR #4596: URL: https://github.com/apache/hadoop/pull/4596#issuecomment-1190351776 > I can create an issue to optimize lambda in hadoop-hdfs module. And create some sub tasks to fix one by one. We can review sub PRs and merge them into the big issue first, then merge it to the trunk. > > How about this? @slfan1989 @goiri I don't think 1 by 1 submits pr, which is really unreasonable. But hadoop-hdfs is indeed relatively large. Can it be submitted according to functional categories, such as HDFS-NN, HDFS-DN, HDFS-JN? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] kevins-29 commented on pull request #4585: HADOOP-12007. GzipCodec native CodecPool leaks memory
kevins-29 commented on PR #4585: URL: https://github.com/apache/hadoop/pull/4585#issuecomment-1190351693 > Thanks @kevins-29 , the fix looks good to me. However, is this addressing the issue mentioned in [HADOOP-12007](https://issues.apache.org/jira/browse/HADOOP-12007)? my understanding is that the issue there is related to `CompressorStream` overrides `close` and doesn't return the compressor to the pool as result. @sunchao Should I create a new Jira ticket for this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18349) Support EKS for IAM service account
[ https://issues.apache.org/jira/browse/HADOOP-18349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569055#comment-17569055 ] vikas koppineedi commented on HADOOP-18349: --- [~dannycjones] yes this is a duplicate. > Support EKS for IAM service account > --- > > Key: HADOOP-18349 > URL: https://issues.apache.org/jira/browse/HADOOP-18349 > Project: Hadoop Common > Issue Type: New Feature > Components: auth, fs/s3 >Reporter: vikas koppineedi >Priority: Major > > Unable to use for authenticating AWS. > *AWS_WEB_IDENTITY_TOKEN_FILE: > /var/run/secrets/[eks.amazonaws.com/serviceaccount/token|http://eks.amazonaws.com/serviceaccount/token]* -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #4596: HDFS-16670. Improve Code With Lambda in EditLogTailer class
ZanderXu commented on PR #4596: URL: https://github.com/apache/hadoop/pull/4596#issuecomment-1190317977 I can create an issue to optimize lambda in hadoop-hdfs module. And create some sub tasks to fix one by one. We can review sub PRs and merge them into the big issue first, then merge it to the trunk. How about this? @slfan1989 @goiri -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] cxzl25 opened a new pull request, #4598: HDFS-16672. Fix lease interval comparison in BlockReportLeaseManager
cxzl25 opened a new pull request, #4598:
URL: https://github.com/apache/hadoop/pull/4598
### Description of PR
monotonicNowMs is generated by System.nanoTime(), direct comparison is not
recommended.
org.apache.hadoop.hdfs.server.blockmanagement.BlockReportLeaseManager#pruneIfExpired
```java
if (monotonicNowMs < node.leaseTimeMs + leaseExpiryMs) {
return false;
}
```
### How was this patch tested?
exist UT
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu commented on pull request #4596: HDFS-16670. Improve Code With Lambda in EditLogTailer class
ZanderXu commented on PR #4596: URL: https://github.com/apache/hadoop/pull/4596#issuecomment-1190310464 Thanks @slfan1989 for your suggestion. I can create a new PR to optimize lambda in hadoop-hdfs moudle, but it will be a huge PR. It will be difficult for reviewing and merging. In addition to lambda optimization, there are some modifications about typo. @goiri do you have some good ideas? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-12007) GzipCodec native CodecPool leaks memory
[
https://issues.apache.org/jira/browse/HADOOP-12007?focusedWorklogId=793251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793251
]
ASF GitHub Bot logged work on HADOOP-12007:
---
Author: ASF GitHub Bot
Created on: 20/Jul/22 13:38
Start Date: 20/Jul/22 13:38
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4585:
URL: https://github.com/apache/hadoop/pull/4585#issuecomment-1190299965
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 16s | | trunk passed |
| +1 :green_heart: | compile | 25m 26s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 52s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 19m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 20s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 7m 27s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 39m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 53m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 53s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 1m 1s |
[/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| -1 :x: | mvninstall | 1m 16s |
[/patch-mvninstall-hadoop-common-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-mvninstall-hadoop-common-project.txt)
| hadoop-common-project in the patch failed. |
| -1 :x: | mvninstall | 1m 32s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| -1 :x: | compile | 1m 42s |
[/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| root in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 1m 42s |
[/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| root in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 1m 29s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 1m 29s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 36s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 6 new
[GitHub] [hadoop] hadoop-yetus commented on pull request #4585: HADOOP-12007. GzipCodec native CodecPool leaks memory
hadoop-yetus commented on PR #4585:
URL: https://github.com/apache/hadoop/pull/4585#issuecomment-1190299965
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 37s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 26m 16s | | trunk passed |
| +1 :green_heart: | compile | 25m 26s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 20m 52s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 19m 11s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 20s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 7m 27s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 39m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 53m 5s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 53s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 1m 1s |
[/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| -1 :x: | mvninstall | 1m 16s |
[/patch-mvninstall-hadoop-common-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-mvninstall-hadoop-common-project.txt)
| hadoop-common-project in the patch failed. |
| -1 :x: | mvninstall | 1m 32s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| -1 :x: | compile | 1m 42s |
[/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| root in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 1m 42s |
[/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| root in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 1m 29s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 1m 29s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 3m 36s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 6 new + 12 unchanged - 0 fixed = 18 total (was
12) |
| -1 :x: | mvnsite | 9m 2s |
[/patch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/5/artifact/out/patch-mvnsite-root.txt)
| root in the patch failed. |
| +1 :green_heart: | javadoc | 8m 17s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 7m 34s |
[jira] [Work logged] (HADOOP-12007) GzipCodec native CodecPool leaks memory
[
https://issues.apache.org/jira/browse/HADOOP-12007?focusedWorklogId=793229=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793229
]
ASF GitHub Bot logged work on HADOOP-12007:
---
Author: ASF GitHub Bot
Created on: 20/Jul/22 13:09
Start Date: 20/Jul/22 13:09
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4585:
URL: https://github.com/apache/hadoop/pull/4585#issuecomment-1190267288
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 0s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 51s | | trunk passed |
| +1 :green_heart: | compile | 25m 30s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 22m 5s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 20m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 33s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 7m 26s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 41m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 58m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 1m 0s |
[/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| -1 :x: | mvninstall | 1m 10s |
[/patch-mvninstall-hadoop-common-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-mvninstall-hadoop-common-project.txt)
| hadoop-common-project in the patch failed. |
| -1 :x: | mvninstall | 1m 16s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| -1 :x: | compile | 1m 29s |
[/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| root in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 1m 29s |
[/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| root in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 1m 21s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 1m 21s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 56s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 6 new
[GitHub] [hadoop] hadoop-yetus commented on pull request #4585: HADOOP-12007. GzipCodec native CodecPool leaks memory
hadoop-yetus commented on PR #4585:
URL: https://github.com/apache/hadoop/pull/4585#issuecomment-1190267288
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 3s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 0s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 51s | | trunk passed |
| +1 :green_heart: | compile | 25m 30s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 22m 5s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | checkstyle | 4m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 20m 18s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 33s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 7m 26s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | spotbugs | 41m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 58m 9s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 33s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 1m 0s |
[/patch-mvninstall-hadoop-common-project_hadoop-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-mvninstall-hadoop-common-project_hadoop-common.txt)
| hadoop-common in the patch failed. |
| -1 :x: | mvninstall | 1m 10s |
[/patch-mvninstall-hadoop-common-project.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-mvninstall-hadoop-common-project.txt)
| hadoop-common-project in the patch failed. |
| -1 :x: | mvninstall | 1m 16s |
[/patch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-mvninstall-root.txt)
| root in the patch failed. |
| -1 :x: | compile | 1m 29s |
[/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| root in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | javac | 1m 29s |
[/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-compile-root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1.txt)
| root in the patch failed with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1. |
| -1 :x: | compile | 1m 21s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| -1 :x: | javac | 1m 21s |
[/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-compile-root-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt)
| root in the patch failed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 56s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 6 new + 12 unchanged - 0 fixed = 18 total (was
12) |
| -1 :x: | mvnsite | 1m 17s |
[/patch-mvnsite-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4585/4/artifact/out/patch-mvnsite-root.txt)
| root in the patch failed. |
| +1 :green_heart: | javadoc | 0m 59s | |
root-jdkPrivateBuild-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 with JDK Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 generated 0
[jira] [Updated] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmar Suhail updated HADOOP-18073:
--
Attachment: Upgrading S3A to SDKV2.pdf
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
> Attachments: Upgrading S3A to SDKV2.pdf
>
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmar Suhail updated HADOOP-18073:
--
Attachment: (was: Upgrading S3A to SDKV2-2.pdf)
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmar Suhail updated HADOOP-18073:
--
Attachment: (was: Upgrading S3A to SDKV2-3.pdf)
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmar Suhail updated HADOOP-18073:
--
Attachment: (was: Upgrading S3A to SDKV2-1.pdf)
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmar Suhail updated HADOOP-18073:
--
Attachment: (was: Upgrading S3A to SDKV2.pdf)
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmar Suhail updated HADOOP-18073:
--
Attachment: Upgrading S3A to SDKV2-3.pdf
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
> Attachments: Upgrading S3A to SDKV2-1.pdf, Upgrading S3A to
> SDKV2-2.pdf, Upgrading S3A to SDKV2-3.pdf, Upgrading S3A to SDKV2.pdf
>
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569010#comment-17569010
]
Ahmar Suhail commented on HADOOP-18073:
---
I've been look at this a bit more and am attaching a document which explains
the issues with upgrading and explores possible solutions. [^Upgrading S3A to
SDKV2.pdf]
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
> Attachments: Upgrading S3A to SDKV2-1.pdf, Upgrading S3A to
> SDKV2-2.pdf, Upgrading S3A to SDKV2-3.pdf, Upgrading S3A to SDKV2.pdf
>
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] (HADOOP-18073) Upgrade AWS SDK to v2
[ https://issues.apache.org/jira/browse/HADOOP-18073 ]
Ahmar Suhail deleted comment on HADOOP-18073:
---
was (Author: JIRAUSER283484):
I've been look at this a bit more and am attaching a document which explains
the issues with upgrading and explores possible solutions. [^Upgrading S3A to
SDKV2.pdf]
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
> Attachments: Upgrading S3A to SDKV2-1.pdf, Upgrading S3A to
> SDKV2-2.pdf, Upgrading S3A to SDKV2.pdf
>
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569008#comment-17569008
]
Ahmar Suhail commented on HADOOP-18073:
---
I've been look at this a bit more and am attaching a document which explains
the issues with upgrading and explores possible solutions. [^Upgrading S3A to
SDKV2.pdf]
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
> Attachments: Upgrading S3A to SDKV2-1.pdf, Upgrading S3A to
> SDKV2-2.pdf, Upgrading S3A to SDKV2.pdf
>
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmar Suhail updated HADOOP-18073:
--
Attachment: Upgrading S3A to SDKV2-2.pdf
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
> Attachments: Upgrading S3A to SDKV2-1.pdf, Upgrading S3A to
> SDKV2-2.pdf, Upgrading S3A to SDKV2.pdf
>
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] slfan1989 commented on pull request #4596: HDFS-16670. Improve Code With Lambda in EditLogTailer class
slfan1989 commented on PR #4596: URL: https://github.com/apache/hadoop/pull/4596#issuecomment-1190257794 Lambda is a very important feature and should indeed be used in the project, but should submitting PRs be submitted at the moudle level？ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Work logged] (HADOOP-18341) upgrade to commons-configuration2 2.8.0
[
https://issues.apache.org/jira/browse/HADOOP-18341?focusedWorklogId=793225=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-793225
]
ASF GitHub Bot logged work on HADOOP-18341:
---
Author: ASF GitHub Bot
Created on: 20/Jul/22 12:59
Start Date: 20/Jul/22 12:59
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on PR #4578:
URL: https://github.com/apache/hadoop/pull/4578#issuecomment-1190256616
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 23s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 59s | | trunk passed |
| +1 :green_heart: | compile | 25m 2s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 51s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 20m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 30s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 7m 36s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 38m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 24m 47s | | the patch passed |
| +1 :green_heart: | compile | 24m 38s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 38s | | the patch passed |
| +1 :green_heart: | compile | 21m 48s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 19m 40s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 25s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 7m 28s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 41m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 1032m 4s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4578/2/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 15s | | The patch does not
generate ASF License warnings. |
| | | 1322m 24s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.mover.TestMover |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4578/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4578 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs |
| uname | Linux 505482efa82e 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b9800b61c36ee1e40e7c326f026e85cddaee87ae |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
[GitHub] [hadoop] hadoop-yetus commented on pull request #4578: HADOOP-18341: upgrade commons-configuration2 to 2.8.0
hadoop-yetus commented on PR #4578:
URL: https://github.com/apache/hadoop/pull/4578#issuecomment-1190256616
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 23s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 1s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 15m 36s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 28m 59s | | trunk passed |
| +1 :green_heart: | compile | 25m 2s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | compile | 21m 51s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | mvnsite | 20m 20s | | trunk passed |
| +1 :green_heart: | javadoc | 8m 30s | | trunk passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 7m 36s | | trunk passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 38m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 31s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 24m 47s | | the patch passed |
| +1 :green_heart: | compile | 24m 38s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javac | 24m 38s | | the patch passed |
| +1 :green_heart: | compile | 21m 48s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | javac | 21m 48s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 19m 40s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 8m 25s | | the patch passed with JDK
Private Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1 |
| +1 :green_heart: | javadoc | 7m 28s | | the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| +1 :green_heart: | shadedclient | 41m 57s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 1032m 4s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4578/2/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| +1 :green_heart: | asflicense | 2m 15s | | The patch does not
generate ASF License warnings. |
| | | 1322m 24s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.mover.TestMover |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.41 ServerAPI=1.41 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4578/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/4578 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient codespell detsecrets xmllint shellcheck shelldocs |
| uname | Linux 505482efa82e 4.15.0-175-generic #184-Ubuntu SMP Thu Mar 24
17:48:36 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / b9800b61c36ee1e40e7c326f026e85cddaee87ae |
| Default Java | Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Multi-JDK versions | /usr/lib/jvm/java-11-openjdk-amd64:Private
Build-11.0.15+10-Ubuntu-0ubuntu0.20.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4578/2/testReport/ |
| Max. process+thread count | 3040 (vs. ulimit of 5500) |
| modules | C: hadoop-project . U: . |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-4578/2/console |
| versions | git=2.25.1
[jira] (HADOOP-18073) Upgrade AWS SDK to v2
[ https://issues.apache.org/jira/browse/HADOOP-18073 ]
Ahmar Suhail deleted comment on HADOOP-18073:
---
was (Author: JIRAUSER283484):
I've been look at this a bit more and am attaching a document which explains
the issues with upgrading and explores possible solutions. [^Upgrading S3A to
SDKV2.pdf]
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
> Attachments: Upgrading S3A to SDKV2-1.pdf, Upgrading S3A to SDKV2.pdf
>
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17569006#comment-17569006
]
Ahmar Suhail commented on HADOOP-18073:
---
I've been look at this a bit more and am attaching a document which explains
the issues with upgrading and explores possible solutions. [^Upgrading S3A to
SDKV2.pdf]
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
> Attachments: Upgrading S3A to SDKV2-1.pdf, Upgrading S3A to SDKV2.pdf
>
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmar Suhail updated HADOOP-18073:
--
Attachment: Upgrading S3A to SDKV2-1.pdf
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
> Attachments: Upgrading S3A to SDKV2-1.pdf, Upgrading S3A to SDKV2.pdf
>
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ZanderXu opened a new pull request, #4597: HDFS-16671. RBF: RouterRpcFairnessPolicyController supports configurable permit acquire timeout
ZanderXu opened a new pull request, #4597: URL: https://github.com/apache/hadoop/pull/4597 RouterRpcFairnessPolicyController supports configurable permit acquire timeout. Hardcode 1s is very long, and it has causes an incident in our prod environment when one nameserivce is busy. And the optimal timeout maybe should be less than p50(avgTime). And all handlers in RBF is waiting to acquire the permit of the busy ns. ``` "IPC Server handler 12 on default port " #2370 daemon prio=5 os_prio=0 tid=? nid=? waiting on condition [?] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for (a java.util.concurrent.Semaphore$NonfairSync) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037) at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328) at java.util.concurrent.Semaphore.tryAcquire(Semaphore.java:409) at org.apache.hadoop.hdfs.server.federation.fairness.AbstractRouterRpcFairnessPolicyController.acquirePermit(AbstractRouterRpcFairnessPolicyController.java:56) at org.apache.hadoop.hdfs.server.federation.fairness.DynamicRouterRpcFairnessPolicyController.acquirePermit(DynamicRouterRpcFairnessPolicyController.java:123) at org.apache.hadoop.hdfs.server.federation.router.RouterRpcClient.acquirePermit(RouterRpcClient.java:1500) ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmar Suhail updated HADOOP-18073:
--
Attachment: Upgrading S3A to SDKV2.pdf
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
> Attachments: Upgrading S3A to SDKV2.pdf
>
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18352) Support AWS SSO for providing credentials to S3A
[ https://issues.apache.org/jira/browse/HADOOP-18352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Carl Jones updated HADOOP-18352: --- Description: HADOOP-18073 was opened regarding upgrading AWS SDK to V2 which supports a credential provider for AWS SSO. Opening this ticket to track that feature explicitly separately from the SDK upgrade. Related SDK issue for AWS SSO support in AWS SDK for Java V1: [https://github.com/aws/aws-sdk-java/issues/2434] In particular, from [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html], when to set 'fs.s3a.aws.credentials.provider', it must be "com.amazonaws.auth.AWSCredentialsProvider". We would like to support "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which supports AWS SSO, so users only need to authenticate once. was: HADOOP-18073 was opened regarding upgrading AWS SDK to V2 which supports a credential provider for AWS SSO. Opening this ticket to track that feature explicitly separately from the SDK upgrade. Related SDK issue for AWS SSO support in AWS SDK for Java V1: https://github.com/aws/aws-sdk-java/issues/2434 > Support AWS SSO for providing credentials to S3A > > > Key: HADOOP-18352 > URL: https://issues.apache.org/jira/browse/HADOOP-18352 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/s3 >Affects Versions: 3.3.3 >Reporter: Daniel Carl Jones >Priority: Minor > > HADOOP-18073 was opened regarding upgrading AWS SDK to V2 which supports a > credential provider for AWS SSO. Opening this ticket to track that feature > explicitly separately from the SDK upgrade. > Related SDK issue for AWS SSO support in AWS SDK for Java V1: > [https://github.com/aws/aws-sdk-java/issues/2434] > In particular, from > [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html], > when to set 'fs.s3a.aws.credentials.provider', it must be > "com.amazonaws.auth.AWSCredentialsProvider". We would like to support > "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which > supports AWS SSO, so users only need to authenticate once. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18073) Upgrade AWS SDK to v2
[
https://issues.apache.org/jira/browse/HADOOP-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17568994#comment-17568994
]
Daniel Carl Jones commented on HADOOP-18073:
This JIRA was originally opened for AWS SSO support. [~xiaoweisun], I've opened
HADOOP-18352 to continue tracking support for AWS SSO.
> Upgrade AWS SDK to v2
> -
>
> Key: HADOOP-18073
> URL: https://issues.apache.org/jira/browse/HADOOP-18073
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: auth, fs/s3
>Affects Versions: 3.3.1
>Reporter: xiaowei sun
>Priority: Major
>
> This task tracks upgrading Hadoop's AWS connector S3A from AWS SDK for Java
> V1 to AWS SDK for Java V2.
> Original use case:
> {quote}We would like to access s3 with AWS SSO, which is supported in
> software.amazon.awssdk:sdk-core:2.*.
> In particular, from
> [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html],
> when to set 'fs.s3a.aws.credentials.provider', it must be
> "com.amazonaws.auth.AWSCredentialsProvider". We would like to support
> "software.amazon.awssdk.auth.credentials.ProfileCredentialsProvider" which
> supports AWS SSO, so users only need to authenticate once.
> {quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
