[jira] [Commented] (HADOOP-15984) Update jersey from 1.19 to 2.x
[
https://issues.apache.org/jira/browse/HADOOP-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738918#comment-17738918
]
ASF GitHub Bot commented on HADOOP-15984:
-
hadoop-yetus commented on PR #5768:
URL: https://github.com/apache/hadoop/pull/5768#issuecomment-1614210841
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 19m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 31s | | trunk passed |
| +1 :green_heart: | compile | 19m 36s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 18m 1s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 4m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 22m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 20m 12s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 18m 31s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 45s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 39s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 34m 24s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 53s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 17s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 33s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/10/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
| hadoop-yarn-common in the patch failed. |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/10/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt)
| hadoop-yarn-server-applicationhistoryservice in the patch failed. |
| -1 :x: | mvninstall | 0m 19s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/10/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt)
| hadoop-yarn-server-timelineservice in the patch failed. |
| -1 :x: | mvninstall | 0m 36s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/10/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 27s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/10/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 22s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/10/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-client in the patch failed. |
| -1 :x: | mvninstall | 0m 34s |
[/patch-mvninstal
[GitHub] [hadoop] hadoop-yetus commented on pull request #5768: HADOOP-15984. Jersey 2.x upgrade
hadoop-yetus commented on PR #5768:
URL: https://github.com/apache/hadoop/pull/5768#issuecomment-1614210841
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 45s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 19m 5s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 31s | | trunk passed |
| +1 :green_heart: | compile | 19m 36s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 18m 1s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 4m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 22m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 20m 12s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 18m 31s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 45s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 39s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 34m 24s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 53s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 17s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 33s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/10/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
| hadoop-yarn-common in the patch failed. |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/10/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt)
| hadoop-yarn-server-applicationhistoryservice in the patch failed. |
| -1 :x: | mvninstall | 0m 19s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/10/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt)
| hadoop-yarn-server-timelineservice in the patch failed. |
| -1 :x: | mvninstall | 0m 36s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/10/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 27s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/10/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 22s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/10/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-client in the patch failed. |
| -1 :x: | mvninstall | 0m 34s |
[/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/10/artifact/out/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
| hadoop-mapreduce-client in the patch failed. |
[GitHub] [hadoop] whbing commented on a diff in pull request #5759: HDFS-17052. Improve BlockPlacementPolicyRackFaultTolerant to avoid choose nodes failed when no enough Rack.
whbing commented on code in PR #5759:
URL: https://github.com/apache/hadoop/pull/5759#discussion_r1247501492
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java:
##
@@ -192,11 +192,23 @@ private void chooseEvenlyFromRemainingRacks(Node writer,
} finally {
excludedNodes.addAll(newExcludeNodes);
}
+ if (numResultsOflastChoose == results.size()) {
+Map nodesPerRack = new HashMap<>();
+for (DatanodeStorageInfo dsInfo : results) {
+ String rackName =
dsInfo.getDatanodeDescriptor().getNetworkLocation();
+ nodesPerRack.merge(rackName, 1, Integer::sum);
+}
+for (int numNodes : nodesPerRack.values()) {
+ if (numNodes > bestEffortMaxNodesPerRack) {
+bestEffortMaxNodesPerRack = numNodes;
+ }
Review Comment:
Line 201~203 get max value, maybe we can use the following code to make it
more clear and concise.
```java
bestEffortMaxNodesPerRack =
Math.max(bestEffortMaxNodesPerRack,
Collections.max(nodesPerRack.values()));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15984) Update jersey from 1.19 to 2.x
[
https://issues.apache.org/jira/browse/HADOOP-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738915#comment-17738915
]
ASF GitHub Bot commented on HADOOP-15984:
-
hadoop-yetus commented on PR #5768:
URL: https://github.com/apache/hadoop/pull/5768#issuecomment-1614207591
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 18m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 36s | | trunk passed |
| +1 :green_heart: | compile | 19m 13s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 17m 38s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 4m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 22m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 20m 4s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 18m 48s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 47s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 43s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 34m 8s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 36s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 19s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 31s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/9/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
| hadoop-yarn-common in the patch failed. |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/9/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt)
| hadoop-yarn-server-applicationhistoryservice in the patch failed. |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/9/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt)
| hadoop-yarn-server-timelineservice in the patch failed. |
| -1 :x: | mvninstall | 0m 36s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/9/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 30s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/9/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 21s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/9/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-client in the patch failed. |
| -1 :x: | mvninstall | 0m 33s |
[/patch-mvninstall-hado
[GitHub] [hadoop] hadoop-yetus commented on pull request #5768: HADOOP-15984. Jersey 2.x upgrade
hadoop-yetus commented on PR #5768:
URL: https://github.com/apache/hadoop/pull/5768#issuecomment-1614207591
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 51s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 18m 45s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 33m 36s | | trunk passed |
| +1 :green_heart: | compile | 19m 13s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 17m 38s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 4m 33s | | trunk passed |
| +1 :green_heart: | mvnsite | 22m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 20m 4s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 18m 48s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 47s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 43s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 34m 8s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 36s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 19s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 31s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/9/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
| hadoop-yarn-common in the patch failed. |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/9/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt)
| hadoop-yarn-server-applicationhistoryservice in the patch failed. |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/9/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt)
| hadoop-yarn-server-timelineservice in the patch failed. |
| -1 :x: | mvninstall | 0m 36s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/9/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 30s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/9/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 21s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/9/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-client in the patch failed. |
| -1 :x: | mvninstall | 0m 33s |
[/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/9/artifact/out/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
| hadoop-mapreduce-client in the patch failed. |
| -1 :
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738911#comment-17738911
]
ASF GitHub Bot commented on HADOOP-18291:
-
virajjasani commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247490918
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
+ // Don't read block 0 completely
+ in.read(buffer, 0, blockSize - S_1K * 10);
+
+ // Seek to block 1 and don't read completely
+ in.seek(blockSize);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 2 and don't read completely
+ in.seek(blockSize * 2L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 3 and don't read completely
+ in.seek(blockSize * 3L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 4 and don't read completely
+ in.seek(blockSize * 4L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 5 and don't read completely
+ in.seek(blockSize * 5L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // backward seek, can't use block 0 as it is evicted
+ in.seek(S_1K * 5);
+ in.read();
+
Review Comment:
sounds great, let me create follow-up sub-task for introducing the metric,
and update the test with the sub-task.
this will likely keep the commit history clean and easy to manage :)
thanks you once again!
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Ahmar Suhail
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Currently there is no limit on the size of disk cache. This means we could
> have a large number of files on files, especially for access patterns that
> are very random and do not always read the block fully.
>
> eg:
> in.seek(5);
> in.read();
> in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read
> in.read();
> in.seek(2 * blockSize + 10) // block 1 gets saved to disk
> .. and so on
>
> The in memory cache is bounded, and by default has a limit of 72MB (9
> blocks). When a block is fully read, and a seek is issued it's released
> [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109].
> We can also delete the on disk file for the block here if it exists.
>
> Also maybe add an upper limit on disk space, and delete the file which stores
> data of the block furthest from the current block (similar to the in memory
> cache) when this limit is reached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
virajjasani commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247490918
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
+ // Don't read block 0 completely
+ in.read(buffer, 0, blockSize - S_1K * 10);
+
+ // Seek to block 1 and don't read completely
+ in.seek(blockSize);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 2 and don't read completely
+ in.seek(blockSize * 2L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 3 and don't read completely
+ in.seek(blockSize * 3L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 4 and don't read completely
+ in.seek(blockSize * 4L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 5 and don't read completely
+ in.seek(blockSize * 5L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // backward seek, can't use block 0 as it is evicted
+ in.seek(S_1K * 5);
+ in.read();
+
Review Comment:
sounds great, let me create follow-up sub-task for introducing the metric,
and update the test with the sub-task.
this will likely keep the commit history clean and easy to manage :)
thanks you once again!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-18764) fs.azure.buffer.dir to be under Yarn container path on yarn applications
[ https://issues.apache.org/jira/browse/HADOOP-18764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mehakmeet Singh resolved HADOOP-18764. -- Fix Version/s: 3.3.9 Resolution: Fixed > fs.azure.buffer.dir to be under Yarn container path on yarn applications > > > Key: HADOOP-18764 > URL: https://issues.apache.org/jira/browse/HADOOP-18764 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Mehakmeet Singh >Priority: Major > Labels: pull-request-available > Fix For: 3.3.9 > > > replicate HADOOP-17386 for azure so things clean up better long lived yarn > clusters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18764) fs.azure.buffer.dir to be under Yarn container path on yarn applications
[ https://issues.apache.org/jira/browse/HADOOP-18764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738908#comment-17738908 ] ASF GitHub Bot commented on HADOOP-18764: - mehakmeet merged PR #5788: URL: https://github.com/apache/hadoop/pull/5788 > fs.azure.buffer.dir to be under Yarn container path on yarn applications > > > Key: HADOOP-18764 > URL: https://issues.apache.org/jira/browse/HADOOP-18764 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Mehakmeet Singh >Priority: Major > Labels: pull-request-available > > replicate HADOOP-17386 for azure so things clean up better long lived yarn > clusters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet merged pull request #5788: HADOOP-18764. fs.azure.buffer.dir to be under Yarn container path on yarn applications
mehakmeet merged PR #5788: URL: https://github.com/apache/hadoop/pull/5788 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18764) fs.azure.buffer.dir to be under Yarn container path on yarn applications
[ https://issues.apache.org/jira/browse/HADOOP-18764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738907#comment-17738907 ] ASF GitHub Bot commented on HADOOP-18764: - mehakmeet commented on PR #5788: URL: https://github.com/apache/hadoop/pull/5788#issuecomment-1614191112 mvn install failure due to connection reset. Ignoring, have built it locally successfully. > fs.azure.buffer.dir to be under Yarn container path on yarn applications > > > Key: HADOOP-18764 > URL: https://issues.apache.org/jira/browse/HADOOP-18764 > Project: Hadoop Common > Issue Type: Improvement > Components: fs/azure >Affects Versions: 3.3.5 >Reporter: Steve Loughran >Assignee: Mehakmeet Singh >Priority: Major > Labels: pull-request-available > > replicate HADOOP-17386 for azure so things clean up better long lived yarn > clusters -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738906#comment-17738906
]
ASF GitHub Bot commented on HADOOP-18291:
-
mehakmeet commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247486332
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
+ // Don't read block 0 completely
+ in.read(buffer, 0, blockSize - S_1K * 10);
+
+ // Seek to block 1 and don't read completely
+ in.seek(blockSize);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 2 and don't read completely
+ in.seek(blockSize * 2L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 3 and don't read completely
+ in.seek(blockSize * 3L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 4 and don't read completely
+ in.seek(blockSize * 4L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 5 and don't read completely
+ in.seek(blockSize * 5L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // backward seek, can't use block 0 as it is evicted
+ in.seek(S_1K * 5);
+ in.read();
+
Review Comment:
Yea, why not, it'll be good for debugging purposes if there's any difference
between them we would know that there's some issue with the proper deletion of
the files from cache. Although an overkill but never hurts 😄
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Ahmar Suhail
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Currently there is no limit on the size of disk cache. This means we could
> have a large number of files on files, especially for access patterns that
> are very random and do not always read the block fully.
>
> eg:
> in.seek(5);
> in.read();
> in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read
> in.read();
> in.seek(2 * blockSize + 10) // block 1 gets saved to disk
> .. and so on
>
> The in memory cache is bounded, and by default has a limit of 72MB (9
> blocks). When a block is fully read, and a seek is issued it's released
> [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109].
> We can also delete the on disk file for the block here if it exists.
>
> Also maybe add an upper limit on disk space, and delete the file which stores
> data of the block furthest from the current block (similar to the in memory
> cache) when this limit is reached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on pull request #5788: HADOOP-18764. fs.azure.buffer.dir to be under Yarn container path on yarn applications
mehakmeet commented on PR #5788: URL: https://github.com/apache/hadoop/pull/5788#issuecomment-1614191112 mvn install failure due to connection reset. Ignoring, have built it locally successfully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a diff in pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
mehakmeet commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247486332
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
+ // Don't read block 0 completely
+ in.read(buffer, 0, blockSize - S_1K * 10);
+
+ // Seek to block 1 and don't read completely
+ in.seek(blockSize);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 2 and don't read completely
+ in.seek(blockSize * 2L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 3 and don't read completely
+ in.seek(blockSize * 3L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 4 and don't read completely
+ in.seek(blockSize * 4L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 5 and don't read completely
+ in.seek(blockSize * 5L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // backward seek, can't use block 0 as it is evicted
+ in.seek(S_1K * 5);
+ in.read();
+
Review Comment:
Yea, why not, it'll be good for debugging purposes if there's any difference
between them we would know that there's some issue with the proper deletion of
the files from cache. Although an overkill but never hurts 😄
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] wzhallright opened a new pull request, #5796: HDFS-17065. Fix typos in hadoop-hdfs-project
wzhallright opened a new pull request, #5796: URL: https://github.com/apache/hadoop/pull/5796 ### Description of PR A few typos were found in daily work ### How was this patch tested? ### For code changes: issue: https://issues.apache.org/jira/browse/HDFS-17065 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738898#comment-17738898
]
ASF GitHub Bot commented on HADOOP-18291:
-
virajjasani commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247477275
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
+ // Don't read block 0 completely
+ in.read(buffer, 0, blockSize - S_1K * 10);
+
+ // Seek to block 1 and don't read completely
+ in.seek(blockSize);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 2 and don't read completely
+ in.seek(blockSize * 2L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 3 and don't read completely
+ in.seek(blockSize * 3L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 4 and don't read completely
+ in.seek(blockSize * 4L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 5 and don't read completely
+ in.seek(blockSize * 5L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // backward seek, can't use block 0 as it is evicted
+ in.seek(S_1K * 5);
+ in.read();
+
Review Comment:
got it, shall we introduce new metric that would help differentiate
`blockRemovedFromFileCache()` vs `blockEvictedFromFileCache()`?
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Ahmar Suhail
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Currently there is no limit on the size of disk cache. This means we could
> have a large number of files on files, especially for access patterns that
> are very random and do not always read the block fully.
>
> eg:
> in.seek(5);
> in.read();
> in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read
> in.read();
> in.seek(2 * blockSize + 10) // block 1 gets saved to disk
> .. and so on
>
> The in memory cache is bounded, and by default has a limit of 72MB (9
> blocks). When a block is fully read, and a seek is issued it's released
> [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109].
> We can also delete the on disk file for the block here if it exists.
>
> Also maybe add an upper limit on disk space, and delete the file which stores
> data of the block furthest from the current block (similar to the in memory
> cache) when this limit is reached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
virajjasani commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247477275
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
+ // Don't read block 0 completely
+ in.read(buffer, 0, blockSize - S_1K * 10);
+
+ // Seek to block 1 and don't read completely
+ in.seek(blockSize);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 2 and don't read completely
+ in.seek(blockSize * 2L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 3 and don't read completely
+ in.seek(blockSize * 3L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 4 and don't read completely
+ in.seek(blockSize * 4L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 5 and don't read completely
+ in.seek(blockSize * 5L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // backward seek, can't use block 0 as it is evicted
+ in.seek(S_1K * 5);
+ in.read();
+
Review Comment:
got it, shall we introduce new metric that would help differentiate
`blockRemovedFromFileCache()` vs `blockEvictedFromFileCache()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738897#comment-17738897
]
ASF GitHub Bot commented on HADOOP-18291:
-
virajjasani commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247476231
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -247,9 +305,46 @@ private Entry getEntry(int blockNumber) {
throw new IllegalStateException(String.format("block %d not found in
cache", blockNumber));
}
numGets++;
+addToHeadOfLinkedList(entry);
return entry;
}
+ /**
+ * Add the given entry to the head of the linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToHeadOfLinkedList(Entry entry) {
+blocksLock.writeLock().lock();
+try {
+ if (head == null) {
+head = entry;
+tail = entry;
+ }
+ if (entry != head) {
+Entry prev = entry.getPrevious();
+Entry nxt = entry.getNext();
+if (prev != null) {
+ prev.setNext(nxt);
+}
+if (nxt != null) {
+ nxt.setPrevious(prev);
+}
+entry.setPrevious(null);
+entry.setNext(head);
+head.setPrevious(entry);
+head = entry;
+ }
+ if (tail != null) {
+while (tail.getNext() != null) {
+ tail = tail.getNext();
+}
+ }
Review Comment:
nice idea, it might be more beneficial for UT to test this.
i am also planning to refactor Entry class on it's own new class rather than
as an inner class of SingleFilePerBlockCache as part of next follow-up
sub-task. once we do that, then it might be even more easier to write some UT
to directly access head, tail pointers.
sorry, i was thinking this as sub-task so maybe adding UT can also be done
with sub-task. does that sound good?
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Ahmar Suhail
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Currently there is no limit on the size of disk cache. This means we could
> have a large number of files on files, especially for access patterns that
> are very random and do not always read the block fully.
>
> eg:
> in.seek(5);
> in.read();
> in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read
> in.read();
> in.seek(2 * blockSize + 10) // block 1 gets saved to disk
> .. and so on
>
> The in memory cache is bounded, and by default has a limit of 72MB (9
> blocks). When a block is fully read, and a seek is issued it's released
> [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109].
> We can also delete the on disk file for the block here if it exists.
>
> Also maybe add an upper limit on disk space, and delete the file which stores
> data of the block furthest from the current block (similar to the in memory
> cache) when this limit is reached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
virajjasani commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247476231
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -247,9 +305,46 @@ private Entry getEntry(int blockNumber) {
throw new IllegalStateException(String.format("block %d not found in
cache", blockNumber));
}
numGets++;
+addToHeadOfLinkedList(entry);
return entry;
}
+ /**
+ * Add the given entry to the head of the linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToHeadOfLinkedList(Entry entry) {
+blocksLock.writeLock().lock();
+try {
+ if (head == null) {
+head = entry;
+tail = entry;
+ }
+ if (entry != head) {
+Entry prev = entry.getPrevious();
+Entry nxt = entry.getNext();
+if (prev != null) {
+ prev.setNext(nxt);
+}
+if (nxt != null) {
+ nxt.setPrevious(prev);
+}
+entry.setPrevious(null);
+entry.setNext(head);
+head.setPrevious(entry);
+head = entry;
+ }
+ if (tail != null) {
+while (tail.getNext() != null) {
+ tail = tail.getNext();
+}
+ }
Review Comment:
nice idea, it might be more beneficial for UT to test this.
i am also planning to refactor Entry class on it's own new class rather than
as an inner class of SingleFilePerBlockCache as part of next follow-up
sub-task. once we do that, then it might be even more easier to write some UT
to directly access head, tail pointers.
sorry, i was thinking this as sub-task so maybe adding UT can also be done
with sub-task. does that sound good?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738895#comment-17738895
]
ASF GitHub Bot commented on HADOOP-18291:
-
mehakmeet commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247474198
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
+ // Don't read block 0 completely
+ in.read(buffer, 0, blockSize - S_1K * 10);
+
+ // Seek to block 1 and don't read completely
+ in.seek(blockSize);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 2 and don't read completely
+ in.seek(blockSize * 2L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 3 and don't read completely
+ in.seek(blockSize * 3L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 4 and don't read completely
+ in.seek(blockSize * 4L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 5 and don't read completely
+ in.seek(blockSize * 5L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // backward seek, can't use block 0 as it is evicted
+ in.seek(S_1K * 5);
+ in.read();
+
Review Comment:
Yes, that should be fine as well, I saw this method, so thought we were
already recording that metric `blockRemovedFromFileCache()`
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Ahmar Suhail
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Currently there is no limit on the size of disk cache. This means we could
> have a large number of files on files, especially for access patterns that
> are very random and do not always read the block fully.
>
> eg:
> in.seek(5);
> in.read();
> in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read
> in.read();
> in.seek(2 * blockSize + 10) // block 1 gets saved to disk
> .. and so on
>
> The in memory cache is bounded, and by default has a limit of 72MB (9
> blocks). When a block is fully read, and a seek is issued it's released
> [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109].
> We can also delete the on disk file for the block here if it exists.
>
> Also maybe add an upper limit on disk space, and delete the file which stores
> data of the block furthest from the current block (similar to the in memory
> cache) when this limit is reached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a diff in pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
mehakmeet commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247474198
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
+ // Don't read block 0 completely
+ in.read(buffer, 0, blockSize - S_1K * 10);
+
+ // Seek to block 1 and don't read completely
+ in.seek(blockSize);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 2 and don't read completely
+ in.seek(blockSize * 2L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 3 and don't read completely
+ in.seek(blockSize * 3L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 4 and don't read completely
+ in.seek(blockSize * 4L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 5 and don't read completely
+ in.seek(blockSize * 5L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // backward seek, can't use block 0 as it is evicted
+ in.seek(S_1K * 5);
+ in.read();
+
Review Comment:
Yes, that should be fine as well, I saw this method, so thought we were
already recording that metric `blockRemovedFromFileCache()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738893#comment-17738893
]
ASF GitHub Bot commented on HADOOP-18291:
-
virajjasani commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247472501
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
+ // Don't read block 0 completely
+ in.read(buffer, 0, blockSize - S_1K * 10);
+
+ // Seek to block 1 and don't read completely
+ in.seek(blockSize);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 2 and don't read completely
+ in.seek(blockSize * 2L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 3 and don't read completely
+ in.seek(blockSize * 3L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 4 and don't read completely
+ in.seek(blockSize * 4L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 5 and don't read completely
+ in.seek(blockSize * 5L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // backward seek, can't use block 0 as it is evicted
+ in.seek(S_1K * 5);
+ in.read();
+
Review Comment:
for the num of assertions, i was thinking of adding metric as next sub-task
so that this patch doesn't become too complicated to review. is that fine with
you?
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Ahmar Suhail
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Currently there is no limit on the size of disk cache. This means we could
> have a large number of files on files, especially for access patterns that
> are very random and do not always read the block fully.
>
> eg:
> in.seek(5);
> in.read();
> in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read
> in.read();
> in.seek(2 * blockSize + 10) // block 1 gets saved to disk
> .. and so on
>
> The in memory cache is bounded, and by default has a limit of 72MB (9
> blocks). When a block is fully read, and a seek is issued it's released
> [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109].
> We can also delete the on disk file for the block here if it exists.
>
> Also maybe add an upper limit on disk space, and delete the file which stores
> data of the block furthest from the current block (similar to the in memory
> cache) when this limit is reached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5795: Update Configuration.java
hadoop-yetus commented on PR #5795:
URL: https://github.com/apache/hadoop/pull/5795#issuecomment-1614173495
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 5s | | trunk passed |
| +1 :green_heart: | compile | 20m 51s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 17m 50s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 25s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 0s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 44s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 17s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 58s | | the patch passed |
| +1 :green_heart: | compile | 16m 25s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 16m 25s | | the patch passed |
| +1 :green_heart: | compile | 16m 13s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 16m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 21s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 45s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 18s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 46s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 19m 5s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | asflicense | 1m 10s | | The patch does not
generate ASF License warnings. |
| | | 233m 24s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5795/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5795 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux caaee5f88eb8 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 1d3619939d73bb4564d13a6d207e74174626d9a6 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5795/1/testReport/ |
| Max. process+thread count | 1673 (vs. ulimit of 5500) |
| modules | C: hadoop-common-project/hadoop-common U:
hadoop-common-project/hadoop-common |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5795/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
virajjasani commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247472501
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
+ // Don't read block 0 completely
+ in.read(buffer, 0, blockSize - S_1K * 10);
+
+ // Seek to block 1 and don't read completely
+ in.seek(blockSize);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 2 and don't read completely
+ in.seek(blockSize * 2L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 3 and don't read completely
+ in.seek(blockSize * 3L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 4 and don't read completely
+ in.seek(blockSize * 4L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 5 and don't read completely
+ in.seek(blockSize * 5L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // backward seek, can't use block 0 as it is evicted
+ in.seek(S_1K * 5);
+ in.read();
+
Review Comment:
for the num of assertions, i was thinking of adding metric as next sub-task
so that this patch doesn't become too complicated to review. is that fine with
you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738891#comment-17738891
]
ASF GitHub Bot commented on HADOOP-18291:
-
mehakmeet commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247471235
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -247,9 +305,46 @@ private Entry getEntry(int blockNumber) {
throw new IllegalStateException(String.format("block %d not found in
cache", blockNumber));
}
numGets++;
+addToHeadOfLinkedList(entry);
return entry;
}
+ /**
+ * Add the given entry to the head of the linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToHeadOfLinkedList(Entry entry) {
+blocksLock.writeLock().lock();
+try {
+ if (head == null) {
+head = entry;
+tail = entry;
+ }
+ if (entry != head) {
+Entry prev = entry.getPrevious();
+Entry nxt = entry.getNext();
+if (prev != null) {
+ prev.setNext(nxt);
+}
+if (nxt != null) {
+ nxt.setPrevious(prev);
+}
+entry.setPrevious(null);
+entry.setNext(head);
+head.setPrevious(entry);
+head = entry;
+ }
+ if (tail != null) {
+while (tail.getNext() != null) {
+ tail = tail.getNext();
+}
+ }
Review Comment:
True, I was thinking, would it be possible via a simple UT as well, where we
pass in the entries as we desire and access them in our preferences to test
functionality, might be easier if we directly test the LRU logic than via the
stream.
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Ahmar Suhail
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Currently there is no limit on the size of disk cache. This means we could
> have a large number of files on files, especially for access patterns that
> are very random and do not always read the block fully.
>
> eg:
> in.seek(5);
> in.read();
> in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read
> in.read();
> in.seek(2 * blockSize + 10) // block 1 gets saved to disk
> .. and so on
>
> The in memory cache is bounded, and by default has a limit of 72MB (9
> blocks). When a block is fully read, and a seek is issued it's released
> [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109].
> We can also delete the on disk file for the block here if it exists.
>
> Also maybe add an upper limit on disk space, and delete the file which stores
> data of the block furthest from the current block (similar to the in memory
> cache) when this limit is reached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a diff in pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
mehakmeet commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247471235
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -247,9 +305,46 @@ private Entry getEntry(int blockNumber) {
throw new IllegalStateException(String.format("block %d not found in
cache", blockNumber));
}
numGets++;
+addToHeadOfLinkedList(entry);
return entry;
}
+ /**
+ * Add the given entry to the head of the linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToHeadOfLinkedList(Entry entry) {
+blocksLock.writeLock().lock();
+try {
+ if (head == null) {
+head = entry;
+tail = entry;
+ }
+ if (entry != head) {
+Entry prev = entry.getPrevious();
+Entry nxt = entry.getNext();
+if (prev != null) {
+ prev.setNext(nxt);
+}
+if (nxt != null) {
+ nxt.setPrevious(prev);
+}
+entry.setPrevious(null);
+entry.setNext(head);
+head.setPrevious(entry);
+head = entry;
+ }
+ if (tail != null) {
+while (tail.getNext() != null) {
+ tail = tail.getNext();
+}
+ }
Review Comment:
True, I was thinking, would it be possible via a simple UT as well, where we
pass in the entries as we desire and access them in our preferences to test
functionality, might be easier if we directly test the LRU logic than via the
stream.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738890#comment-17738890
]
ASF GitHub Bot commented on HADOOP-18291:
-
mehakmeet commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247470283
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -299,9 +395,62 @@ public void put(int blockNumber, ByteBuffer buffer,
Configuration conf,
// Update stream_read_blocks_in_cache stats only after blocks map is
updated with new file
// entry to avoid any discrepancy related to the value of
stream_read_blocks_in_cache.
// If stream_read_blocks_in_cache is updated before updating the blocks
map here, closing of
-// the input stream can lead to the removal of the cache file even before
blocks is added with
-// the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
+// the input stream can lead to the removal of the cache file even before
blocks is added
+// with the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
prefetchingStatistics.blockAddedToFileCache();
+addToLinkedListAndEvictIfRequired(entry);
+ }
+
+ /**
+ * Add the given entry to the head of the linked list and if the LRU cache
size
+ * exceeds the max limit, evict tail of the LRU linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToLinkedListAndEvictIfRequired(Entry entry) {
+addToHeadOfLinkedList(entry);
+blocksLock.writeLock().lock();
+try {
+ if (blocks.size() > maxBlocksCount && !closed.get()) {
+Entry elementToPurge = tail;
+tail = tail.getPrevious();
+if (tail == null) {
+ tail = head;
+}
+tail.setNext(null);
+elementToPurge.setPrevious(null);
+deleteBlockFileAndEvictCache(elementToPurge);
+ }
+} finally {
+ blocksLock.writeLock().unlock();
+}
+ }
+
+ /**
+ * Delete cache file as part of the block cache LRU eviction.
+ *
+ * @param elementToPurge Block entry to evict.
+ */
+ private void deleteBlockFileAndEvictCache(Entry elementToPurge) {
+boolean lockAcquired =
+elementToPurge.takeLock(Entry.LockType.WRITE,
PREFETCH_WRITE_LOCK_TIMEOUT,
+PREFETCH_WRITE_LOCK_TIMEOUT_UNIT);
+if (!lockAcquired) {
+ LOG.error("Cache file {} deletion would not be attempted as write lock
could not"
Review Comment:
Okay, sounds good
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Ahmar Suhail
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Currently there is no limit on the size of disk cache. This means we could
> have a large number of files on files, especially for access patterns that
> are very random and do not always read the block fully.
>
> eg:
> in.seek(5);
> in.read();
> in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read
> in.read();
> in.seek(2 * blockSize + 10) // block 1 gets saved to disk
> .. and so on
>
> The in memory cache is bounded, and by default has a limit of 72MB (9
> blocks). When a block is fully read, and a seek is issued it's released
> [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109].
> We can also delete the on disk file for the block here if it exists.
>
> Also maybe add an upper limit on disk space, and delete the file which stores
> data of the block furthest from the current block (similar to the in memory
> cache) when this limit is reached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a diff in pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
mehakmeet commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247470283
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -299,9 +395,62 @@ public void put(int blockNumber, ByteBuffer buffer,
Configuration conf,
// Update stream_read_blocks_in_cache stats only after blocks map is
updated with new file
// entry to avoid any discrepancy related to the value of
stream_read_blocks_in_cache.
// If stream_read_blocks_in_cache is updated before updating the blocks
map here, closing of
-// the input stream can lead to the removal of the cache file even before
blocks is added with
-// the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
+// the input stream can lead to the removal of the cache file even before
blocks is added
+// with the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
prefetchingStatistics.blockAddedToFileCache();
+addToLinkedListAndEvictIfRequired(entry);
+ }
+
+ /**
+ * Add the given entry to the head of the linked list and if the LRU cache
size
+ * exceeds the max limit, evict tail of the LRU linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToLinkedListAndEvictIfRequired(Entry entry) {
+addToHeadOfLinkedList(entry);
+blocksLock.writeLock().lock();
+try {
+ if (blocks.size() > maxBlocksCount && !closed.get()) {
+Entry elementToPurge = tail;
+tail = tail.getPrevious();
+if (tail == null) {
+ tail = head;
+}
+tail.setNext(null);
+elementToPurge.setPrevious(null);
+deleteBlockFileAndEvictCache(elementToPurge);
+ }
+} finally {
+ blocksLock.writeLock().unlock();
+}
+ }
+
+ /**
+ * Delete cache file as part of the block cache LRU eviction.
+ *
+ * @param elementToPurge Block entry to evict.
+ */
+ private void deleteBlockFileAndEvictCache(Entry elementToPurge) {
+boolean lockAcquired =
+elementToPurge.takeLock(Entry.LockType.WRITE,
PREFETCH_WRITE_LOCK_TIMEOUT,
+PREFETCH_WRITE_LOCK_TIMEOUT_UNIT);
+if (!lockAcquired) {
+ LOG.error("Cache file {} deletion would not be attempted as write lock
could not"
Review Comment:
Okay, sounds good
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738889#comment-17738889
]
ASF GitHub Bot commented on HADOOP-18291:
-
mehakmeet commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247458556
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -77,13 +78,14 @@ public ITestS3APrefetchingInputStream() {
private static final int TIMEOUT_MILLIS = 5000;
private static final int INTERVAL_MILLIS = 500;
-
+ private static final int PREFETCH_MAX_NUM_BLOCKS = 3;
@Override
public Configuration createConfiguration() {
Configuration conf = super.createConfiguration();
S3ATestUtils.removeBaseAndBucketOverrides(conf, PREFETCH_ENABLED_KEY);
conf.setBoolean(PREFETCH_ENABLED_KEY, true);
+conf.setInt(FS_PREFETCH_MAX_BLOCKS_COUNT, PREFETCH_MAX_NUM_BLOCKS);
Review Comment:
nit: Remove base and bucket config for this property in L86, just so that
test is consistent in diff env.
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
Review Comment:
add a comment here explaining what the test is doing on a high level, so
that it's easier to understand the flow of how LRU is happening.
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
+ // Don't read block 0 completely
+ in.read(buffer, 0, blockSize - S_1K * 10);
+
+ // Seek to block 1 and don't read completely
+ in.seek(blockSize);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 2 and don't read completely
+ in.seek(blockSize * 2L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 3 and don't read completely
+ in.seek(blockSize * 3L);
+ in.read(buffer, 0, 2 * S_1K);
+
Review Comment:
Add a seek to the intersection point of two blocks(eg: 4*blockSize - 10KB)
and read some bytes(>10KBs) to read both blocks and assert if the head of the
list is the correct block.
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
+ // Don't read block 0 completely
+ in.read(buffer, 0, blockSize - S_1K * 10);
+
+ // Seek to block 1 and don't read completely
+ in.seek(blockSize);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 2 and don't read completely
+ in.seek(blockSize * 2L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 3 and don't read completely
+ in.seek(blockSize * 3L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 4 and don't read completely
+ in.seek(blockSize * 4L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 5 and don't read completely
+ in.seek(blockSize * 5L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // backward seek, can't use block 0 as it is evicted
+ in.seek(S_1K * 5);
+ in.read();
+
Review Comment:
Assert the number of evictions being done or blocks being removed from the
list. At certain points, test what the capacity of the list is to keep it
consistent.
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Ahmar Suhail
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Currently there is no limit on the size of disk cache. This
[GitHub] [hadoop] mehakmeet commented on a diff in pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
mehakmeet commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247458556
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -77,13 +78,14 @@ public ITestS3APrefetchingInputStream() {
private static final int TIMEOUT_MILLIS = 5000;
private static final int INTERVAL_MILLIS = 500;
-
+ private static final int PREFETCH_MAX_NUM_BLOCKS = 3;
@Override
public Configuration createConfiguration() {
Configuration conf = super.createConfiguration();
S3ATestUtils.removeBaseAndBucketOverrides(conf, PREFETCH_ENABLED_KEY);
conf.setBoolean(PREFETCH_ENABLED_KEY, true);
+conf.setInt(FS_PREFETCH_MAX_BLOCKS_COUNT, PREFETCH_MAX_NUM_BLOCKS);
Review Comment:
nit: Remove base and bucket config for this property in L86, just so that
test is consistent in diff env.
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
Review Comment:
add a comment here explaining what the test is doing on a high level, so
that it's easier to understand the flow of how LRU is happening.
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
+ // Don't read block 0 completely
+ in.read(buffer, 0, blockSize - S_1K * 10);
+
+ // Seek to block 1 and don't read completely
+ in.seek(blockSize);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 2 and don't read completely
+ in.seek(blockSize * 2L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 3 and don't read completely
+ in.seek(blockSize * 3L);
+ in.read(buffer, 0, 2 * S_1K);
+
Review Comment:
Add a seek to the intersection point of two blocks(eg: 4*blockSize - 10KB)
and read some bytes(>10KBs) to read both blocks and assert if the head of the
list is the correct block.
##
hadoop-tools/hadoop-aws/src/test/java/org/apache/hadoop/fs/s3a/ITestS3APrefetchingInputStream.java:
##
@@ -301,4 +303,56 @@ public void testStatusProbesAfterClosingStream() throws
Throwable {
}
+ @Test
+ public void testSeeksWithLruEviction() throws Throwable {
+IOStatistics ioStats;
+openFS();
+
+try (FSDataInputStream in = largeFileFS.open(largeFile)) {
+ ioStats = in.getIOStatistics();
+
+ byte[] buffer = new byte[blockSize];
+
+ // Don't read block 0 completely
+ in.read(buffer, 0, blockSize - S_1K * 10);
+
+ // Seek to block 1 and don't read completely
+ in.seek(blockSize);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 2 and don't read completely
+ in.seek(blockSize * 2L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 3 and don't read completely
+ in.seek(blockSize * 3L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 4 and don't read completely
+ in.seek(blockSize * 4L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // Seek to block 5 and don't read completely
+ in.seek(blockSize * 5L);
+ in.read(buffer, 0, 2 * S_1K);
+
+ // backward seek, can't use block 0 as it is evicted
+ in.seek(S_1K * 5);
+ in.read();
+
Review Comment:
Assert the number of evictions being done or blocks being removed from the
list. At certain points, test what the capacity of the list is to keep it
consistent.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738882#comment-17738882
]
ASF GitHub Bot commented on HADOOP-18291:
-
virajjasani commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247450167
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -299,9 +395,62 @@ public void put(int blockNumber, ByteBuffer buffer,
Configuration conf,
// Update stream_read_blocks_in_cache stats only after blocks map is
updated with new file
// entry to avoid any discrepancy related to the value of
stream_read_blocks_in_cache.
// If stream_read_blocks_in_cache is updated before updating the blocks
map here, closing of
-// the input stream can lead to the removal of the cache file even before
blocks is added with
-// the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
+// the input stream can lead to the removal of the cache file even before
blocks is added
+// with the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
prefetchingStatistics.blockAddedToFileCache();
+addToLinkedListAndEvictIfRequired(entry);
+ }
+
+ /**
+ * Add the given entry to the head of the linked list and if the LRU cache
size
+ * exceeds the max limit, evict tail of the LRU linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToLinkedListAndEvictIfRequired(Entry entry) {
+addToHeadOfLinkedList(entry);
+blocksLock.writeLock().lock();
+try {
+ if (blocks.size() > maxBlocksCount && !closed.get()) {
+Entry elementToPurge = tail;
+tail = tail.getPrevious();
+if (tail == null) {
+ tail = head;
+}
+tail.setNext(null);
+elementToPurge.setPrevious(null);
+deleteBlockFileAndEvictCache(elementToPurge);
+ }
+} finally {
+ blocksLock.writeLock().unlock();
+}
+ }
+
+ /**
+ * Delete cache file as part of the block cache LRU eviction.
+ *
+ * @param elementToPurge Block entry to evict.
+ */
+ private void deleteBlockFileAndEvictCache(Entry elementToPurge) {
+boolean lockAcquired =
+elementToPurge.takeLock(Entry.LockType.WRITE,
PREFETCH_WRITE_LOCK_TIMEOUT,
+PREFETCH_WRITE_LOCK_TIMEOUT_UNIT);
+if (!lockAcquired) {
+ LOG.error("Cache file {} deletion would not be attempted as write lock
could not"
Review Comment:
if eviction misses it, stream close would be able to clean it up.
if stream close misses it, then it stays on disk and we might eventually
also come up with some "file last accessed" based check and maybe some crons
removing them eventually. not a bad idea IMO.
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Ahmar Suhail
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Currently there is no limit on the size of disk cache. This means we could
> have a large number of files on files, especially for access patterns that
> are very random and do not always read the block fully.
>
> eg:
> in.seek(5);
> in.read();
> in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read
> in.read();
> in.seek(2 * blockSize + 10) // block 1 gets saved to disk
> .. and so on
>
> The in memory cache is bounded, and by default has a limit of 72MB (9
> blocks). When a block is fully read, and a seek is issued it's released
> [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109].
> We can also delete the on disk file for the block here if it exists.
>
> Also maybe add an upper limit on disk space, and delete the file which stores
> data of the block furthest from the current block (similar to the in memory
> cache) when this limit is reached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
virajjasani commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247450167
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -299,9 +395,62 @@ public void put(int blockNumber, ByteBuffer buffer,
Configuration conf,
// Update stream_read_blocks_in_cache stats only after blocks map is
updated with new file
// entry to avoid any discrepancy related to the value of
stream_read_blocks_in_cache.
// If stream_read_blocks_in_cache is updated before updating the blocks
map here, closing of
-// the input stream can lead to the removal of the cache file even before
blocks is added with
-// the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
+// the input stream can lead to the removal of the cache file even before
blocks is added
+// with the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
prefetchingStatistics.blockAddedToFileCache();
+addToLinkedListAndEvictIfRequired(entry);
+ }
+
+ /**
+ * Add the given entry to the head of the linked list and if the LRU cache
size
+ * exceeds the max limit, evict tail of the LRU linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToLinkedListAndEvictIfRequired(Entry entry) {
+addToHeadOfLinkedList(entry);
+blocksLock.writeLock().lock();
+try {
+ if (blocks.size() > maxBlocksCount && !closed.get()) {
+Entry elementToPurge = tail;
+tail = tail.getPrevious();
+if (tail == null) {
+ tail = head;
+}
+tail.setNext(null);
+elementToPurge.setPrevious(null);
+deleteBlockFileAndEvictCache(elementToPurge);
+ }
+} finally {
+ blocksLock.writeLock().unlock();
+}
+ }
+
+ /**
+ * Delete cache file as part of the block cache LRU eviction.
+ *
+ * @param elementToPurge Block entry to evict.
+ */
+ private void deleteBlockFileAndEvictCache(Entry elementToPurge) {
+boolean lockAcquired =
+elementToPurge.takeLock(Entry.LockType.WRITE,
PREFETCH_WRITE_LOCK_TIMEOUT,
+PREFETCH_WRITE_LOCK_TIMEOUT_UNIT);
+if (!lockAcquired) {
+ LOG.error("Cache file {} deletion would not be attempted as write lock
could not"
Review Comment:
if eviction misses it, stream close would be able to clean it up.
if stream close misses it, then it stays on disk and we might eventually
also come up with some "file last accessed" based check and maybe some crons
removing them eventually. not a bad idea IMO.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738881#comment-17738881
]
ASF GitHub Bot commented on HADOOP-18291:
-
virajjasani commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247449167
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -299,9 +395,62 @@ public void put(int blockNumber, ByteBuffer buffer,
Configuration conf,
// Update stream_read_blocks_in_cache stats only after blocks map is
updated with new file
// entry to avoid any discrepancy related to the value of
stream_read_blocks_in_cache.
// If stream_read_blocks_in_cache is updated before updating the blocks
map here, closing of
-// the input stream can lead to the removal of the cache file even before
blocks is added with
-// the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
+// the input stream can lead to the removal of the cache file even before
blocks is added
+// with the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
prefetchingStatistics.blockAddedToFileCache();
+addToLinkedListAndEvictIfRequired(entry);
+ }
+
+ /**
+ * Add the given entry to the head of the linked list and if the LRU cache
size
+ * exceeds the max limit, evict tail of the LRU linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToLinkedListAndEvictIfRequired(Entry entry) {
+addToHeadOfLinkedList(entry);
+blocksLock.writeLock().lock();
+try {
+ if (blocks.size() > maxBlocksCount && !closed.get()) {
+Entry elementToPurge = tail;
+tail = tail.getPrevious();
+if (tail == null) {
+ tail = head;
+}
+tail.setNext(null);
+elementToPurge.setPrevious(null);
+deleteBlockFileAndEvictCache(elementToPurge);
+ }
+} finally {
+ blocksLock.writeLock().unlock();
+}
+ }
+
+ /**
+ * Delete cache file as part of the block cache LRU eviction.
+ *
+ * @param elementToPurge Block entry to evict.
+ */
+ private void deleteBlockFileAndEvictCache(Entry elementToPurge) {
+boolean lockAcquired =
+elementToPurge.takeLock(Entry.LockType.WRITE,
PREFETCH_WRITE_LOCK_TIMEOUT,
+PREFETCH_WRITE_LOCK_TIMEOUT_UNIT);
+if (!lockAcquired) {
+ LOG.error("Cache file {} deletion would not be attempted as write lock
could not"
Review Comment:
it should be okay, in fact we have same logic for input stream close as
well, if eviction or removal of disk block is unsuccessful, we are leaving them
with a fat warning.
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Ahmar Suhail
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Currently there is no limit on the size of disk cache. This means we could
> have a large number of files on files, especially for access patterns that
> are very random and do not always read the block fully.
>
> eg:
> in.seek(5);
> in.read();
> in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read
> in.read();
> in.seek(2 * blockSize + 10) // block 1 gets saved to disk
> .. and so on
>
> The in memory cache is bounded, and by default has a limit of 72MB (9
> blocks). When a block is fully read, and a seek is issued it's released
> [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109].
> We can also delete the on disk file for the block here if it exists.
>
> Also maybe add an upper limit on disk space, and delete the file which stores
> data of the block furthest from the current block (similar to the in memory
> cache) when this limit is reached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
virajjasani commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247449167
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -299,9 +395,62 @@ public void put(int blockNumber, ByteBuffer buffer,
Configuration conf,
// Update stream_read_blocks_in_cache stats only after blocks map is
updated with new file
// entry to avoid any discrepancy related to the value of
stream_read_blocks_in_cache.
// If stream_read_blocks_in_cache is updated before updating the blocks
map here, closing of
-// the input stream can lead to the removal of the cache file even before
blocks is added with
-// the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
+// the input stream can lead to the removal of the cache file even before
blocks is added
+// with the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
prefetchingStatistics.blockAddedToFileCache();
+addToLinkedListAndEvictIfRequired(entry);
+ }
+
+ /**
+ * Add the given entry to the head of the linked list and if the LRU cache
size
+ * exceeds the max limit, evict tail of the LRU linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToLinkedListAndEvictIfRequired(Entry entry) {
+addToHeadOfLinkedList(entry);
+blocksLock.writeLock().lock();
+try {
+ if (blocks.size() > maxBlocksCount && !closed.get()) {
+Entry elementToPurge = tail;
+tail = tail.getPrevious();
+if (tail == null) {
+ tail = head;
+}
+tail.setNext(null);
+elementToPurge.setPrevious(null);
+deleteBlockFileAndEvictCache(elementToPurge);
+ }
+} finally {
+ blocksLock.writeLock().unlock();
+}
+ }
+
+ /**
+ * Delete cache file as part of the block cache LRU eviction.
+ *
+ * @param elementToPurge Block entry to evict.
+ */
+ private void deleteBlockFileAndEvictCache(Entry elementToPurge) {
+boolean lockAcquired =
+elementToPurge.takeLock(Entry.LockType.WRITE,
PREFETCH_WRITE_LOCK_TIMEOUT,
+PREFETCH_WRITE_LOCK_TIMEOUT_UNIT);
+if (!lockAcquired) {
+ LOG.error("Cache file {} deletion would not be attempted as write lock
could not"
Review Comment:
it should be okay, in fact we have same logic for input stream close as
well, if eviction or removal of disk block is unsuccessful, we are leaving them
with a fat warning.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomicooler commented on pull request #5783: YARN-11000. Replace queue resource calculation logic in updateClusterResource
tomicooler commented on PR #5783: URL: https://github.com/apache/hadoop/pull/5783#issuecomment-1614145314 Green with Legacy mode as well. 👍 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738879#comment-17738879
]
ASF GitHub Bot commented on HADOOP-18291:
-
virajjasani commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247448062
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -247,9 +305,46 @@ private Entry getEntry(int blockNumber) {
throw new IllegalStateException(String.format("block %d not found in
cache", blockNumber));
}
numGets++;
+addToHeadOfLinkedList(entry);
return entry;
}
+ /**
+ * Add the given entry to the head of the linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToHeadOfLinkedList(Entry entry) {
+blocksLock.writeLock().lock();
+try {
+ if (head == null) {
+head = entry;
+tail = entry;
+ }
+ if (entry != head) {
+Entry prev = entry.getPrevious();
+Entry nxt = entry.getNext();
+if (prev != null) {
+ prev.setNext(nxt);
+}
+if (nxt != null) {
+ nxt.setPrevious(prev);
+}
+entry.setPrevious(null);
+entry.setNext(head);
+head.setPrevious(entry);
+head = entry;
+ }
+ if (tail != null) {
+while (tail.getNext() != null) {
+ tail = tail.getNext();
+}
+ }
Review Comment:
thanks, i was actually thinking about it but there is just not clean way of
doing so, hence what i have been able to do so far was by "logging" head and
tail nodes (as you also mentioned earlier) with all other nodes, so that i
could track the exact nodes sequence. that's the best way i could find so far,
but extracting that info in IT is really difficult (if we were to do it in
clean way).
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Ahmar Suhail
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Currently there is no limit on the size of disk cache. This means we could
> have a large number of files on files, especially for access patterns that
> are very random and do not always read the block fully.
>
> eg:
> in.seek(5);
> in.read();
> in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read
> in.read();
> in.seek(2 * blockSize + 10) // block 1 gets saved to disk
> .. and so on
>
> The in memory cache is bounded, and by default has a limit of 72MB (9
> blocks). When a block is fully read, and a seek is issued it's released
> [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109].
> We can also delete the on disk file for the block here if it exists.
>
> Also maybe add an upper limit on disk space, and delete the file which stores
> data of the block furthest from the current block (similar to the in memory
> cache) when this limit is reached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on a diff in pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
virajjasani commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247448062
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -247,9 +305,46 @@ private Entry getEntry(int blockNumber) {
throw new IllegalStateException(String.format("block %d not found in
cache", blockNumber));
}
numGets++;
+addToHeadOfLinkedList(entry);
return entry;
}
+ /**
+ * Add the given entry to the head of the linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToHeadOfLinkedList(Entry entry) {
+blocksLock.writeLock().lock();
+try {
+ if (head == null) {
+head = entry;
+tail = entry;
+ }
+ if (entry != head) {
+Entry prev = entry.getPrevious();
+Entry nxt = entry.getNext();
+if (prev != null) {
+ prev.setNext(nxt);
+}
+if (nxt != null) {
+ nxt.setPrevious(prev);
+}
+entry.setPrevious(null);
+entry.setNext(head);
+head.setPrevious(entry);
+head = entry;
+ }
+ if (tail != null) {
+while (tail.getNext() != null) {
+ tail = tail.getNext();
+}
+ }
Review Comment:
thanks, i was actually thinking about it but there is just not clean way of
doing so, hence what i have been able to do so far was by "logging" head and
tail nodes (as you also mentioned earlier) with all other nodes, so that i
could track the exact nodes sequence. that's the best way i could find so far,
but extracting that info in IT is really difficult (if we were to do it in
clean way).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738877#comment-17738877
]
ASF GitHub Bot commented on HADOOP-18291:
-
mehakmeet commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247447055
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -299,9 +395,62 @@ public void put(int blockNumber, ByteBuffer buffer,
Configuration conf,
// Update stream_read_blocks_in_cache stats only after blocks map is
updated with new file
// entry to avoid any discrepancy related to the value of
stream_read_blocks_in_cache.
// If stream_read_blocks_in_cache is updated before updating the blocks
map here, closing of
-// the input stream can lead to the removal of the cache file even before
blocks is added with
-// the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
+// the input stream can lead to the removal of the cache file even before
blocks is added
+// with the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
prefetchingStatistics.blockAddedToFileCache();
+addToLinkedListAndEvictIfRequired(entry);
+ }
+
+ /**
+ * Add the given entry to the head of the linked list and if the LRU cache
size
+ * exceeds the max limit, evict tail of the LRU linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToLinkedListAndEvictIfRequired(Entry entry) {
+addToHeadOfLinkedList(entry);
+blocksLock.writeLock().lock();
+try {
+ if (blocks.size() > maxBlocksCount && !closed.get()) {
+Entry elementToPurge = tail;
+tail = tail.getPrevious();
+if (tail == null) {
+ tail = head;
+}
+tail.setNext(null);
+elementToPurge.setPrevious(null);
+deleteBlockFileAndEvictCache(elementToPurge);
+ }
+} finally {
+ blocksLock.writeLock().unlock();
+}
+ }
+
+ /**
+ * Delete cache file as part of the block cache LRU eviction.
+ *
+ * @param elementToPurge Block entry to evict.
+ */
+ private void deleteBlockFileAndEvictCache(Entry elementToPurge) {
+boolean lockAcquired =
+elementToPurge.takeLock(Entry.LockType.WRITE,
PREFETCH_WRITE_LOCK_TIMEOUT,
+PREFETCH_WRITE_LOCK_TIMEOUT_UNIT);
+if (!lockAcquired) {
+ LOG.error("Cache file {} deletion would not be attempted as write lock
could not"
Review Comment:
It seems like we are okay with things not blowing up if eviction is not
successful, are we okay with it? Can this hurt in the long run?
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Ahmar Suhail
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Currently there is no limit on the size of disk cache. This means we could
> have a large number of files on files, especially for access patterns that
> are very random and do not always read the block fully.
>
> eg:
> in.seek(5);
> in.read();
> in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read
> in.read();
> in.seek(2 * blockSize + 10) // block 1 gets saved to disk
> .. and so on
>
> The in memory cache is bounded, and by default has a limit of 72MB (9
> blocks). When a block is fully read, and a seek is issued it's released
> [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109].
> We can also delete the on disk file for the block here if it exists.
>
> Also maybe add an upper limit on disk space, and delete the file which stores
> data of the block furthest from the current block (similar to the in memory
> cache) when this limit is reached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a diff in pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
mehakmeet commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247447055
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -299,9 +395,62 @@ public void put(int blockNumber, ByteBuffer buffer,
Configuration conf,
// Update stream_read_blocks_in_cache stats only after blocks map is
updated with new file
// entry to avoid any discrepancy related to the value of
stream_read_blocks_in_cache.
// If stream_read_blocks_in_cache is updated before updating the blocks
map here, closing of
-// the input stream can lead to the removal of the cache file even before
blocks is added with
-// the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
+// the input stream can lead to the removal of the cache file even before
blocks is added
+// with the new cache file, leading to incorrect value of
stream_read_blocks_in_cache.
prefetchingStatistics.blockAddedToFileCache();
+addToLinkedListAndEvictIfRequired(entry);
+ }
+
+ /**
+ * Add the given entry to the head of the linked list and if the LRU cache
size
+ * exceeds the max limit, evict tail of the LRU linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToLinkedListAndEvictIfRequired(Entry entry) {
+addToHeadOfLinkedList(entry);
+blocksLock.writeLock().lock();
+try {
+ if (blocks.size() > maxBlocksCount && !closed.get()) {
+Entry elementToPurge = tail;
+tail = tail.getPrevious();
+if (tail == null) {
+ tail = head;
+}
+tail.setNext(null);
+elementToPurge.setPrevious(null);
+deleteBlockFileAndEvictCache(elementToPurge);
+ }
+} finally {
+ blocksLock.writeLock().unlock();
+}
+ }
+
+ /**
+ * Delete cache file as part of the block cache LRU eviction.
+ *
+ * @param elementToPurge Block entry to evict.
+ */
+ private void deleteBlockFileAndEvictCache(Entry elementToPurge) {
+boolean lockAcquired =
+elementToPurge.takeLock(Entry.LockType.WRITE,
PREFETCH_WRITE_LOCK_TIMEOUT,
+PREFETCH_WRITE_LOCK_TIMEOUT_UNIT);
+if (!lockAcquired) {
+ LOG.error("Cache file {} deletion would not be attempted as write lock
could not"
Review Comment:
It seems like we are okay with things not blowing up if eviction is not
successful, are we okay with it? Can this hurt in the long run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18291) S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
[
https://issues.apache.org/jira/browse/HADOOP-18291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738876#comment-17738876
]
ASF GitHub Bot commented on HADOOP-18291:
-
mehakmeet commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247446174
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -247,9 +305,46 @@ private Entry getEntry(int blockNumber) {
throw new IllegalStateException(String.format("block %d not found in
cache", blockNumber));
}
numGets++;
+addToHeadOfLinkedList(entry);
return entry;
}
+ /**
+ * Add the given entry to the head of the linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToHeadOfLinkedList(Entry entry) {
+blocksLock.writeLock().lock();
+try {
+ if (head == null) {
+head = entry;
+tail = entry;
+ }
+ if (entry != head) {
+Entry prev = entry.getPrevious();
+Entry nxt = entry.getNext();
+if (prev != null) {
+ prev.setNext(nxt);
+}
+if (nxt != null) {
+ nxt.setPrevious(prev);
+}
+entry.setPrevious(null);
+entry.setNext(head);
+head.setPrevious(entry);
+head = entry;
+ }
+ if (tail != null) {
+while (tail.getNext() != null) {
+ tail = tail.getNext();
+}
+ }
Review Comment:
Yes, happy with the new change I think. Would be good to explicitly test
certain tail changing scenarios in the IT like you mentioned above if we are
not already doing it.
> S3A prefetch - Implement LRU cache for SingleFilePerBlockCache
> --
>
> Key: HADOOP-18291
> URL: https://issues.apache.org/jira/browse/HADOOP-18291
> Project: Hadoop Common
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Ahmar Suhail
>Assignee: Viraj Jasani
>Priority: Major
> Labels: pull-request-available
>
> Currently there is no limit on the size of disk cache. This means we could
> have a large number of files on files, especially for access patterns that
> are very random and do not always read the block fully.
>
> eg:
> in.seek(5);
> in.read();
> in.seek(blockSize + 10) // block 0 gets saved to disk as it's not fully read
> in.read();
> in.seek(2 * blockSize + 10) // block 1 gets saved to disk
> .. and so on
>
> The in memory cache is bounded, and by default has a limit of 72MB (9
> blocks). When a block is fully read, and a seek is issued it's released
> [here|https://github.com/apache/hadoop/blob/feature-HADOOP-18028-s3a-prefetch/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/read/S3CachingInputStream.java#L109].
> We can also delete the on disk file for the block here if it exists.
>
> Also maybe add an upper limit on disk space, and delete the file which stores
> data of the block furthest from the current block (similar to the in memory
> cache) when this limit is reached.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] mehakmeet commented on a diff in pull request #5754: HADOOP-18291. S3A prefetch - Implement thread-safe LRU cache for SingleFilePerBlockCache
mehakmeet commented on code in PR #5754:
URL: https://github.com/apache/hadoop/pull/5754#discussion_r1247446174
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/impl/prefetch/SingleFilePerBlockCache.java:
##
@@ -247,9 +305,46 @@ private Entry getEntry(int blockNumber) {
throw new IllegalStateException(String.format("block %d not found in
cache", blockNumber));
}
numGets++;
+addToHeadOfLinkedList(entry);
return entry;
}
+ /**
+ * Add the given entry to the head of the linked list.
+ *
+ * @param entry Block entry to add.
+ */
+ private void addToHeadOfLinkedList(Entry entry) {
+blocksLock.writeLock().lock();
+try {
+ if (head == null) {
+head = entry;
+tail = entry;
+ }
+ if (entry != head) {
+Entry prev = entry.getPrevious();
+Entry nxt = entry.getNext();
+if (prev != null) {
+ prev.setNext(nxt);
+}
+if (nxt != null) {
+ nxt.setPrevious(prev);
+}
+entry.setPrevious(null);
+entry.setNext(head);
+head.setPrevious(entry);
+head = entry;
+ }
+ if (tail != null) {
+while (tail.getNext() != null) {
+ tail = tail.getNext();
+}
+ }
Review Comment:
Yes, happy with the new change I think. Would be good to explicitly test
certain tail changing scenarios in the IT like you mentioned above if we are
not already doing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] zhtttylz commented on a diff in pull request #5759: HDFS-17052. Improve BlockPlacementPolicyRackFaultTolerant to avoid choose nodes failed when no enough Rack.
zhtttylz commented on code in PR #5759:
URL: https://github.com/apache/hadoop/pull/5759#discussion_r1247441720
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java:
##
@@ -192,11 +192,23 @@ private void chooseEvenlyFromRemainingRacks(Node writer,
} finally {
excludedNodes.addAll(newExcludeNodes);
}
+ if (numResultsOflastChoose == results.size()) {
Review Comment:
Thanks for your suggestion, I'll make the required code changes promptly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] Hexiaoqiao commented on a diff in pull request #5759: HDFS-17052. Erasure coding reconstruction failed when num of storageT…
Hexiaoqiao commented on code in PR #5759:
URL: https://github.com/apache/hadoop/pull/5759#discussion_r1247427463
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java:
##
@@ -192,11 +192,23 @@ private void chooseEvenlyFromRemainingRacks(Node writer,
} finally {
excludedNodes.addAll(newExcludeNodes);
}
+ if (numResultsOflastChoose == results.size()) {
Review Comment:
It will be better to add some annotation here.
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestReconstructStripedBlocks.java:
##
@@ -515,4 +517,63 @@ public void testReconstrutionWithBusyBlock1() throws
Exception {
assertEquals(9, bm.countNodes(blockInfo).liveReplicas());
}
+ @Test
+ public void testReconstructionWithStorageTypeNotEnough() throws Exception {
+final HdfsConfiguration conf = new HdfsConfiguration();
+conf.setInt(DFSConfigKeys.DFS_NAMENODE_REDUNDANCY_INTERVAL_SECONDS_KEY, 1);
+
+// nine disk node eleven archive node
Review Comment:
Please use a capital letter at the beginning of the sentences and period at
the end of annotation. (L525,L533,L556,L563)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18207) Introduce hadoop-logging module
[ https://issues.apache.org/jira/browse/HADOOP-18207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738863#comment-17738863 ] ASF GitHub Bot commented on HADOOP-18207: - virajjasani commented on PR #5717: URL: https://github.com/apache/hadoop/pull/5717#issuecomment-1614073367 resolved merge conflicts. Humble request if anyone can provide additional review of the PR in addition to @Apache9's review. > Introduce hadoop-logging module > --- > > Key: HADOOP-18207 > URL: https://issues.apache.org/jira/browse/HADOOP-18207 > Project: Hadoop Common > Issue Type: Sub-task >Reporter: Duo Zhang >Assignee: Viraj Jasani >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > There are several goals here: > # Provide the ability to change log level, get log level, etc. > # Place all the appender implementation(?) > # Hide the real logging implementation. > # Later we could remove all the log4j references in other hadoop module. > # Move as much log4j usage to the module as possible. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] virajjasani commented on pull request #5717: HADOOP-18207. Introduce hadoop-logging module
virajjasani commented on PR #5717: URL: https://github.com/apache/hadoop/pull/5717#issuecomment-1614073367 resolved merge conflicts. Humble request if anyone can provide additional review of the PR in addition to @Apache9's review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn merged pull request #5791: HDFS-17062. HA NameNode Web UI should show last HA Transition time
ayushtkn merged PR #5791: URL: https://github.com/apache/hadoop/pull/5791 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn closed pull request #5795: Update Configuration.java
ayushtkn closed pull request #5795: Update Configuration.java URL: https://github.com/apache/hadoop/pull/5795 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on pull request #5795: Update Configuration.java
ayushtkn commented on PR #5795: URL: https://github.com/apache/hadoop/pull/5795#issuecomment-1614062928 irrelevant closing -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] chioWong commented on pull request #5795: Update Configuration.java
chioWong commented on PR #5795: URL: https://github.com/apache/hadoop/pull/5795#issuecomment-1614018620 5255 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] chioWong opened a new pull request, #5795: Update Configuration.java
chioWong opened a new pull request, #5795: URL: https://github.com/apache/hadoop/pull/5795 ### Description of PR ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hfutatzhanghb commented on pull request #5753: HDFS-17050. Erasure coding: fix bug for invalidating duplicated block when two ec block at the same datanode but different storage.
hfutatzhanghb commented on PR #5753: URL: https://github.com/apache/hadoop/pull/5753#issuecomment-1614002174 > the problem makes sense, I am not able to decode the solution though in one pass and I saw there is a comment saying there is some problem in the current solution, so would circle back once it is ready for review state. > > but if the problem is for real, then it is indeed a nice catch :) Sir, thanks for your replying. Yes, current solution still have some problems. I will fix it soonly and we can discuss then. Thanks again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5783: YARN-11000. Replace queue resource calculation logic in updateClusterResource
hadoop-yetus commented on PR #5783:
URL: https://github.com/apache/hadoop/pull/5783#issuecomment-1613891353
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 7s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | jsonlint | 0m 0s | | jsonlint was not available. |
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 49 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 18s | | trunk passed |
| +1 :green_heart: | compile | 1m 2s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 1s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 59s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 54s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 50s | | the patch passed |
| +1 :green_heart: | compile | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 0m 47s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 47s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 49s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/4/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 8 new + 857 unchanged - 23 fixed = 865 total (was 880) |
| +1 :green_heart: | mvnsite | 0m 49s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 44s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 58s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 42s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 102m 27s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 250m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5783 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint
xmllint |
| uname | Linux 7e92986c05c8 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 84d1bb34fcb83fd9708a279216f66275a4cd1098 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/4/testReport/ |
| Max. process+thread count | 909 (vs. ulimit of 5500
[jira] [Commented] (HADOOP-18786) Hadoop build depends on archives.apache.org
[ https://issues.apache.org/jira/browse/HADOOP-18786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738791#comment-17738791 ] ASF GitHub Bot commented on HADOOP-18786: - ctubbsii commented on PR #5789: URL: https://github.com/apache/hadoop/pull/5789#issuecomment-1613822991 The version bumps are merely incidental to the actual issue this PR and JIRA are intending to expose: that the use of the archives should be avoided. If committers want to bump the versions first, I can rebase this PR once that is done. However, these changes are pretty trivial, so I'm not really needed for that part. Once the committers decide to act on this, they can rebase or merge however they see fit. Up to now, these "should" comments about what to do about the version bumps have been very unclear to me... it looks like discussion among yourselves... but if it's a request for me to change something in this PR, please state the request clearly. > Hadoop build depends on archives.apache.org > --- > > Key: HADOOP-18786 > URL: https://issues.apache.org/jira/browse/HADOOP-18786 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.6 >Reporter: Christopher Tubbs >Priority: Critical > Labels: pull-request-available > > Several times throughout Hadoop's source, the ASF archive is referenced, > including part of the build that downloads Yetus. > Building a release from source should not require access to the ASF archives, > as that contributes to end users being subject to throttling and blocking by > INFRA, for "abuse" of the archives, even though they are merely building a > current ASF release from source. This is particularly problematic for > downstream packagers who must build from Hadoop's source, or for CI/CD > situations that depend on Hadoop's source, and particularly problematic for > those end users behind a NAT gateway, because even if Hadoop's use of the > archive is modest, it adds up for multiple users. > The build should be modified, so that it does not require access to fixed > versions in the archives (or should work with the upstream of those dependent > projects to publish their releases elsewhere, for routine consumptions). In > the interim, the source could be updated to point to the current dependency > versions available on downloads.apache.org. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ctubbsii commented on pull request #5789: HADOOP-18786. Use CDN instead of ASF archive
ctubbsii commented on PR #5789: URL: https://github.com/apache/hadoop/pull/5789#issuecomment-1613822991 The version bumps are merely incidental to the actual issue this PR and JIRA are intending to expose: that the use of the archives should be avoided. If committers want to bump the versions first, I can rebase this PR once that is done. However, these changes are pretty trivial, so I'm not really needed for that part. Once the committers decide to act on this, they can rebase or merge however they see fit. Up to now, these "should" comments about what to do about the version bumps have been very unclear to me... it looks like discussion among yourselves... but if it's a request for me to change something in this PR, please state the request clearly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ayushtkn commented on a diff in pull request #5753: HDFS-17050. Erasure coding: fix bug for invalidating duplicated block when two ec block at the same datanode but different storage
ayushtkn commented on code in PR #5753: URL: https://github.com/apache/hadoop/pull/5753#discussion_r1246991248 ## hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockInfoStriped.java: ## @@ -6,9 +6,9 @@ * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License. You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * + * + * http://www.apache.org/licenses/LICENSE-2.0 + * Review Comment: don't play with the licence -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5794: HDFS-17064. Document the usage of the new Balancer "sortTopNodes" and "hotBlockTimeInterval" parameter
hadoop-yetus commented on PR #5794:
URL: https://github.com/apache/hadoop/pull/5794#issuecomment-1613634442
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | markdownlint | 0m 1s | | markdownlint was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 48m 12s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5794/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 14s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 38s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 3m 25s | | trunk passed |
| +1 :green_heart: | shadedclient | 42m 44s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 1m 27s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 1m 27s | | the patch passed |
| +1 :green_heart: | compile | 1m 18s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 1m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 8s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 21s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 36s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 3m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 27s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 239m 14s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5794/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 396m 44s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestObserverNode |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5794/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5794 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets markdownlint
|
| uname | Linux 1b911298e317 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / ef7dafe7d6ea6171036ce0027fda390eefcf1976 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5794/1/testReport/ |
| Max. process+thread count | 2
[jira] [Commented] (HADOOP-18750) Spark History Server 3.3.1 fails to starts with Hadoop 3.3.x
[
https://issues.apache.org/jira/browse/HADOOP-18750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738737#comment-17738737
]
ASF GitHub Bot commented on HADOOP-18750:
-
ayushtkn commented on PR #5695:
URL: https://github.com/apache/hadoop/pull/5695#issuecomment-1613632813
I am not sure about it, In my personal experience: shading without
relocation often leads to strange issues if there are different versions of the
same dependency is the classpath and this jar should be widely used.
Does the solution pointed by @sunchao helps? or at worst if there is only
few classes which creates issues, can we exclude just those classes from
relocation?
In any case, I am not sure about this. I will pass to other more experienced
folks to take a call on this.
> Spark History Server 3.3.1 fails to starts with Hadoop 3.3.x
>
>
> Key: HADOOP-18750
> URL: https://issues.apache.org/jira/browse/HADOOP-18750
> Project: Hadoop Common
> Issue Type: Bug
>Reporter: Aman Raj
>Assignee: Kamal Sharma
>Priority: Major
> Labels: pull-request-available
>
> When Spark History Server tries to start with Hadoop 3.3.4 (Happens only in
> Kerberos scenarios), it fails to do so with the following exception:
> {code:java}
> 23/05/23 03:14:15 ERROR HistoryServer [main]: Failed to bind HistoryServer
> java.lang.IllegalStateException: class
> org.apache.hadoop.security.authentication.server.AuthenticationFilter is not
> a javax.servlet.Filter
> at
> org.sparkproject.jetty.servlet.FilterHolder.doStart(FilterHolder.java:103)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]
> at
> org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:749)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]
> at
> java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
> ~[?:1.8.0_372]
> at
> java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742)
> ~[?:1.8.0_372]
> at
> java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:647)
> ~[?:1.8.0_372]
> at
> org.sparkproject.jetty.servlet.ServletHandler.initialize(ServletHandler.java:774)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:379)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]
> at
> org.sparkproject.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:916)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]
> at
> org.sparkproject.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:288)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]
> at
> org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:73)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]
> at org.apache.spark.ui.ServerInfo.addHandler(JettyUtils.scala:491)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]
> at org.apache.spark.ui.WebUI.$anonfun$bind$3(WebUI.scala:154)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]
> at org.apache.spark.ui.WebUI.$anonfun$bind$3$adapted(WebUI.scala:154)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]
> at
> scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> ~[scala-library-2.12.15.jar:?]
> at
> scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> ~[scala-library-2.12.15.jar:?]
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
> ~[scala-library-2.12.15.jar:?]
> at org.apache.spark.ui.WebUI.bind(WebUI.scala:154)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]
> at
> org.apache.spark.deploy.history.HistoryServer.bind(HistoryServer.scala:164)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]
> at
> org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:310)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT]
> at
> org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
> ~[spark-core_2.12-3.3.1.5.1-SNAPSHOT.jar:3.3.1.5.1-SNAPSHOT] {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional co
[GitHub] [hadoop] ayushtkn commented on pull request #5695: HADOOP-18750. Remove javax/servlet shading in hadoop-client-api
ayushtkn commented on PR #5695: URL: https://github.com/apache/hadoop/pull/5695#issuecomment-1613632813 I am not sure about it, In my personal experience: shading without relocation often leads to strange issues if there are different versions of the same dependency is the classpath and this jar should be widely used. Does the solution pointed by @sunchao helps? or at worst if there is only few classes which creates issues, can we exclude just those classes from relocation? In any case, I am not sure about this. I will pass to other more experienced folks to take a call on this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] github-code-scanning[bot] commented on pull request #5779: Add CodeQL workflow
github-code-scanning[bot] commented on PR #5779: URL: https://github.com/apache/hadoop/pull/5779#issuecomment-1613614657 This pull request sets up GitHub code scanning for this repository. Once the scans have completed and the checks have passed, the analysis results for this pull request branch will appear on [this overview](/apache/hadoop/security/code-scanning?query=pr%3A5779+is%3Aopen). Once you merge this pull request, the 'Security' tab will show more code scanning analysis results (for example, for the default branch). Depending on your configuration and choice of analysis tool, future pull requests will be annotated with code scanning analysis results. For more information about GitHub code scanning, check out [the documentation](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] tomicooler commented on pull request #5783: YARN-11000. Replace queue resource calculation logic in updateClusterResource
tomicooler commented on PR #5783: URL: https://github.com/apache/hadoop/pull/5783#issuecomment-1613588641 Note: +1 Overall with the new queue calculation mode enabled. 💚 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5783: YARN-11000. Replace queue resource calculation logic in updateClusterResource
hadoop-yetus commented on PR #5783:
URL: https://github.com/apache/hadoop/pull/5783#issuecomment-1613585628
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | jsonlint | 0m 1s | | jsonlint was not available. |
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 49 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 3s | | trunk passed |
| +1 :green_heart: | compile | 1m 4s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 0m 53s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 57s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 59s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 39m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 49s | | the patch passed |
| +1 :green_heart: | compile | 0m 57s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 57s | | the patch passed |
| +1 :green_heart: | compile | 0m 46s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 46s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 49s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/3/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
|
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 12 new + 856 unchanged - 24 fixed = 868 total (was 880) |
| +1 :green_heart: | mvnsite | 0m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 40s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 57s | | the patch passed |
| +1 :green_heart: | shadedclient | 40m 36s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 102m 40s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 251m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5783 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets jsonlint
xmllint |
| uname | Linux c52bda787775 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3fc52164af9f96c5a3d18ea5ea84b85e74037485 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5783/3/testReport/ |
| Max. process+thread count | 863 (vs. ulimit of 550
[jira] [Created] (HADOOP-18788) Support LZ4 compressionLevel
wenweijian created HADOOP-18788: --- Summary: Support LZ4 compressionLevel Key: HADOOP-18788 URL: https://issues.apache.org/jira/browse/HADOOP-18788 Project: Hadoop Common Issue Type: New Feature Components: common Reporter: wenweijian The Hadoop's LZ4 compression codec now depends on lz4-java. There two type of compressor in lz4, `fastCompressor` and `highCompressor`. The default lz4 compressor in hadoop is fastCompressor, we also can use highCompressor by using config `IO_COMPRESSION_CODEC_LZ4_USELZ4HC_DEFAULT` When we want to use highCompressor in hadoop, we only can use the default compression level, which is level 9. while highCompressor in lz4-java supports compression level from 1 to 17. I think we can set a configuration to let users to choose different compresssionLevel for lz4 highCompressor. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] hadoop-yetus commented on pull request #5793: HDFS-17063. Datanode configures different Capacity Reserved for each …
hadoop-yetus commented on PR #5793:
URL: https://github.com/apache/hadoop/pull/5793#issuecomment-1613522017
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 5s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 51m 26s | | trunk passed |
| +1 :green_heart: | compile | 1m 26s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 10s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 32s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 3m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 19s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5793/2/artifact/out/blanks-eol.txt)
| The patch has 11 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 0s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5793/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 11 unchanged -
0 fixed = 13 total (was 11) |
| +1 :green_heart: | mvnsite | 1m 17s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| -1 :x: | spotbugs | 3m 24s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5793/2/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html)
| hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) |
| +1 :green_heart: | shadedclient | 40m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 253m 58s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 48s | | The patch does not
generate ASF License warnings. |
| | | 413m 31s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | baseURI isn't initialized in
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ProvidedVolumeImpl.toString()
when invoked from constructor for superclass At ProvidedVolumeImpl.java:when
invoked from constructor for superclass At ProvidedVolumeImpl.java:[line 556] |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5793/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5793 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux babadb2e233e 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven
[GitHub] [hadoop] xinglin commented on pull request #5790: HDFS-17055 Export HAState as a metric from Namenode for monitoring
xinglin commented on PR #5790: URL: https://github.com/apache/hadoop/pull/5790#issuecomment-1613505813 Hi @goiri , Could you help commit this PR into branch-3.3? It is the same change which you helped merge into trunk this Monday. This PR contains only one code-change commit and the rest are empty commits to trigger multiple builds. We have one clean build and that should be good enough to merge. If you have different thoughts, please let me know. thanks, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-18112) Implement paging during S3 multi object delete.
[
https://issues.apache.org/jira/browse/HADOOP-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HADOOP-18112:
Description:
{*}Error{*}:
Rename operation fails during multi object delete of size more than 1000. We
see an exception during multi object delete of more than 1000 keys in one go
during rename operation.
{noformat}
org.apache.hadoop.fs.s3a.AWSBadRequestException: rename
com.amazonaws.services.s3.model.AmazonS3Exception
: The XML you provided was not well-formed or did not validate against our
published schema (Service: Amazon S3; Status Code: 400; Error Code:
MalformedXML; Request ID: XZ8PGAQHP0FGHPYS; S3 Extended Request ID:
vTG8c+koukzQ8yMRGd9BvWfmRwkCZ3fAs/EOiAV5S9E
JjLqFTNCgDOKokuus5W600Z5iOa/iQBI=; Proxy: null), S3 Extended Request ID:
vTG8c+koukzQ8yMRGd9BvWfmRwkCZ3fAs/EOiAV5S9EJjLqFTNCgDOKokuus5W600Z5iOa/iQBI=:MalformedXML:
The XML you provided was not well-formed or did not validate against our
published schema
(Service: Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID:
XZ8PGAQHP0FGHPYS; S3 Extended Request ID:
vTG8c+koukzQ8yMRGd9BvWfmRwkCZ3fAs/EOiAV5S9EJjLqFTNCgDOKokuus5W600Z5iOa/iQBI=;
Proxy: null)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:247)
at
org.apache.hadoop.fs.s3a.s3guard.RenameTracker.convertToIOException(RenameTracker.java:267)
at
org.apache.hadoop.fs.s3a.s3guard.RenameTracker.deleteFailed(RenameTracker.java:198)
at
org.apache.hadoop.fs.s3a.impl.RenameOperation.removeSourceObjects(RenameOperation.java:706)
at
org.apache.hadoop.fs.s3a.impl.RenameOperation.completeActiveCopiesAndDeleteSources(RenameOperation.java:274)
at
org.apache.hadoop.fs.s3a.impl.RenameOperation.recursiveDirectoryRename(RenameOperation.java:484)
at
org.apache.hadoop.fs.s3a.impl.RenameOperation.execute(RenameOperation.java:312)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.innerRename(S3AFileSystem.java:1912)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$rename$7(S3AFileSystem.java:1759)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499)
at
org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:444)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2250)
at
org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:1757)
at org.apache.hadoop.fs.FileSystem.rename(FileSystem.java:1605)
at
org.apache.hadoop.fs.TrashPolicyDefault.moveToTrash(TrashPolicyDefault.java:186)
at org.apache.hadoop.fs.Trash.moveToTrash(Trash.java:110){noformat}
{*}Solution{*}:
So implementing paging of requests to reduce the number of keys in a single
request. Page size can be configured
using "fs.s3a.bulk.delete.page.size"
was:
{*}Error{*}:
Rename operation fails during multi object delete of size more than 1000. We
see below exception during multi object delete of more than 1000 keys in one go
during rename operation.
{noformat}
org.apache.hadoop.fs.s3a.AWSBadRequestException: rename
s3a://ms-targeting-prod-cdp-aws-dr-bkt/data/ms-targeting-prod-hbase/hbase/.tmp/data/default/dr-productionL.Address
to
s3a://ms-targeting-prod-cdp-aws-dr-bkt/user/root/.Trash/Current/data/ms-targetin
g-prod-hbase/hbase/.tmp/data/default/dr-productionL.Address16438377847941643837797901
on
s3a://ms-targeting-prod-cdp-aws-dr-bkt/data/ms-targeting-prod-hbase/hbase/.tmp/data/default/dr-productionL.Address:
com.amazonaws.services.s3.model.AmazonS3Exception
: The XML you provided was not well-formed or did not validate against our
published schema (Service: Amazon S3; Status Code: 400; Error Code:
MalformedXML; Request ID: XZ8PGAQHP0FGHPYS; S3 Extended Request ID:
vTG8c+koukzQ8yMRGd9BvWfmRwkCZ3fAs/EOiAV5S9E
JjLqFTNCgDOKokuus5W600Z5iOa/iQBI=; Proxy: null), S3 Extended Request ID:
vTG8c+koukzQ8yMRGd9BvWfmRwkCZ3fAs/EOiAV5S9EJjLqFTNCgDOKokuus5W600Z5iOa/iQBI=:MalformedXML:
The XML you provided was not well-formed or did not validate against our
published schema
(Service: Amazon S3; Status Code: 400; Error Code: MalformedXML; Request ID:
XZ8PGAQHP0FGHPYS; S3 Extended Request ID:
vTG8c+koukzQ8yMRGd9BvWfmRwkCZ3fAs/EOiAV5S9EJjLqFTNCgDOKokuus5W600Z5iOa/iQBI=;
Proxy: null)
at
org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:247)
at
org.apache.hadoop.fs.s3a.s3guard.RenameTracker.convertToIOException(RenameTracker.java:267)
at
org.apache.hadoop.fs.s3a.s3guard.RenameTracker.deleteFailed(RenameTracker.java:198)
at
org.apache.hadoop.fs.s3a.impl.RenameOperation.removeSourceObjects(RenameOperation.java:706)
at
org.apache.hadoop.fs.s3a.impl.RenameOperation.completeActiveCopiesAndDeleteSources(RenameOperation.java:274)
at
org.apach
[GitHub] [hadoop] hadoop-yetus commented on pull request #5793: HDFS-17063. Datanode configures different Capacity Reserved for each …
hadoop-yetus commented on PR #5793:
URL: https://github.com/apache/hadoop/pull/5793#issuecomment-1613412279
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 53s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 50m 55s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 1m 15s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 23s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 3m 23s | | trunk passed |
| +1 :green_heart: | shadedclient | 40m 11s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| -1 :x: | blanks | 0m 0s |
[/blanks-eol.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5793/1/artifact/out/blanks-eol.txt)
| The patch has 4 line(s) that end in blanks. Use git apply --whitespace=fix
<>. Refer https://git-scm.com/docs/git-apply |
| -0 :warning: | checkstyle | 1m 1s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5793/1/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 2 new + 11 unchanged -
0 fixed = 13 total (was 11) |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 56s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 1m 30s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| -1 :x: | spotbugs | 3m 24s |
[/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5793/1/artifact/out/new-spotbugs-hadoop-hdfs-project_hadoop-hdfs.html)
| hadoop-hdfs-project/hadoop-hdfs generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) |
| +1 :green_heart: | shadedclient | 40m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 237m 42s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5793/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 46s | | The patch does not
generate ASF License warnings. |
| | | 394m 25s | | |
| Reason | Tests |
|---:|:--|
| SpotBugs | module:hadoop-hdfs-project/hadoop-hdfs |
| | baseURI isn't initialized in
org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.ProvidedVolumeImpl.toString()
when invoked from constructor for superclass At ProvidedVolumeImpl.java:when
invoked from constructor for superclass At ProvidedVolumeImpl.java:[line 556] |
| Failed junit tests |
hadoop.hdfs.server.balancer.TestBalancerWithHANameNodes |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5793/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5793 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codes
[GitHub] [hadoop] ashutoshcipher commented on pull request #4932: MAPREDUCE-7406. Add aggregated webservice endpoints to fetch all tasks including their attempts
ashutoshcipher commented on PR #4932: URL: https://github.com/apache/hadoop/pull/4932#issuecomment-1613175672 @slfan1989 , As you have looked at PR in past, Its opened for while. Please help in reviewing it. Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ashutoshcipher commented on pull request #5293: YARN-11414. ClusterMetricsInfo shows wrong availableMB when node labels enabled
ashutoshcipher commented on PR #5293: URL: https://github.com/apache/hadoop/pull/5293#issuecomment-1613172372 The PR seems to be open a while now. @tomicooler @cnauroth @aajisaka Please help in reviewing it Thank you so much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18786) Hadoop build depends on archives.apache.org
[ https://issues.apache.org/jira/browse/HADOOP-18786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738568#comment-17738568 ] ASF GitHub Bot commented on HADOOP-18786: - ctubbsii commented on PR #5789: URL: https://github.com/apache/hadoop/pull/5789#issuecomment-1613163547 > yetus is special compared to gradle as it is asf, but then so would thriftc and mvn be. From https://www.apache.org/legal/release-policy.html#source-packages: `A source release SHOULD not contain compiled code` So, it's fine if you're just bundling and redistributing Yetus scripts, which are also source. I'm not sure if Yetus contains compiled code, or if it's just scripts, and I'm not sure which parts Hadoop's build uses. > Hadoop build depends on archives.apache.org > --- > > Key: HADOOP-18786 > URL: https://issues.apache.org/jira/browse/HADOOP-18786 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.6 >Reporter: Christopher Tubbs >Priority: Critical > Labels: pull-request-available > > Several times throughout Hadoop's source, the ASF archive is referenced, > including part of the build that downloads Yetus. > Building a release from source should not require access to the ASF archives, > as that contributes to end users being subject to throttling and blocking by > INFRA, for "abuse" of the archives, even though they are merely building a > current ASF release from source. This is particularly problematic for > downstream packagers who must build from Hadoop's source, or for CI/CD > situations that depend on Hadoop's source, and particularly problematic for > those end users behind a NAT gateway, because even if Hadoop's use of the > archive is modest, it adds up for multiple users. > The build should be modified, so that it does not require access to fixed > versions in the archives (or should work with the upstream of those dependent > projects to publish their releases elsewhere, for routine consumptions). In > the interim, the source could be updated to point to the current dependency > versions available on downloads.apache.org. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] ctubbsii commented on pull request #5789: [HADOOP-18786] Use CDN instead of ASF archive
ctubbsii commented on PR #5789: URL: https://github.com/apache/hadoop/pull/5789#issuecomment-1613163547 > yetus is special compared to gradle as it is asf, but then so would thriftc and mvn be. From https://www.apache.org/legal/release-policy.html#source-packages: `A source release SHOULD not contain compiled code` So, it's fine if you're just bundling and redistributing Yetus scripts, which are also source. I'm not sure if Yetus contains compiled code, or if it's just scripts, and I'm not sure which parts Hadoop's build uses. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-18786) Hadoop build depends on archives.apache.org
[ https://issues.apache.org/jira/browse/HADOOP-18786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738566#comment-17738566 ] ASF GitHub Bot commented on HADOOP-18786: - steveloughran commented on PR #5789: URL: https://github.com/apache/hadoop/pull/5789#issuecomment-1613156364 yetus is special compared to gradle as it is asf, but then so would thriftc and mvn be. > Hadoop build depends on archives.apache.org > --- > > Key: HADOOP-18786 > URL: https://issues.apache.org/jira/browse/HADOOP-18786 > Project: Hadoop Common > Issue Type: Bug > Components: build >Affects Versions: 3.3.6 >Reporter: Christopher Tubbs >Priority: Critical > Labels: pull-request-available > > Several times throughout Hadoop's source, the ASF archive is referenced, > including part of the build that downloads Yetus. > Building a release from source should not require access to the ASF archives, > as that contributes to end users being subject to throttling and blocking by > INFRA, for "abuse" of the archives, even though they are merely building a > current ASF release from source. This is particularly problematic for > downstream packagers who must build from Hadoop's source, or for CI/CD > situations that depend on Hadoop's source, and particularly problematic for > those end users behind a NAT gateway, because even if Hadoop's use of the > archive is modest, it adds up for multiple users. > The build should be modified, so that it does not require access to fixed > versions in the archives (or should work with the upstream of those dependent > projects to publish their releases elsewhere, for routine consumptions). In > the interim, the source could be updated to point to the current dependency > versions available on downloads.apache.org. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[GitHub] [hadoop] steveloughran commented on pull request #5789: [HADOOP-18786] Use CDN instead of ASF archive
steveloughran commented on PR #5789: URL: https://github.com/apache/hadoop/pull/5789#issuecomment-1613156364 yetus is special compared to gradle as it is asf, but then so would thriftc and mvn be. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15984) Update jersey from 1.19 to 2.x
[
https://issues.apache.org/jira/browse/HADOOP-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738535#comment-17738535
]
ASF GitHub Bot commented on HADOOP-15984:
-
hadoop-yetus commented on PR #5768:
URL: https://github.com/apache/hadoop/pull/5768#issuecomment-1613047628
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 18m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 46s | | trunk passed |
| +1 :green_heart: | compile | 19m 4s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 17m 37s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 4m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 22m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 19m 48s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 18m 41s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 47s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 39s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 34m 5s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 35s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 24s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 33s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/7/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
| hadoop-yarn-common in the patch failed. |
| -1 :x: | mvninstall | 0m 22s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/7/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt)
| hadoop-yarn-server-applicationhistoryservice in the patch failed. |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/7/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt)
| hadoop-yarn-server-timelineservice in the patch failed. |
| -1 :x: | mvninstall | 0m 35s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/7/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 28s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/7/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/7/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-client in the patch failed. |
| -1 :x: | mvninstall | 0m 33s |
[/patch-mvninstall-hado
[jira] [Commented] (HADOOP-15984) Update jersey from 1.19 to 2.x
[
https://issues.apache.org/jira/browse/HADOOP-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738534#comment-17738534
]
ASF GitHub Bot commented on HADOOP-15984:
-
hadoop-yetus commented on PR #5768:
URL: https://github.com/apache/hadoop/pull/5768#issuecomment-1613044997
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 32s | | trunk passed |
| +1 :green_heart: | compile | 18m 47s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 17m 40s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 4m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 22m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 19m 49s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 18m 40s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 46s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 43s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 34m 5s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 33s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 3m 22s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 33s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/8/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
| hadoop-yarn-common in the patch failed. |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/8/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt)
| hadoop-yarn-server-applicationhistoryservice in the patch failed. |
| -1 :x: | mvninstall | 0m 22s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/8/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt)
| hadoop-yarn-server-timelineservice in the patch failed. |
| -1 :x: | mvninstall | 0m 34s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/8/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 31s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/8/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 21s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/8/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-client in the patch failed. |
| -1 :x: | mvninstall | 0m 35s |
[/patch-mvninstall-hado
[GitHub] [hadoop] hadoop-yetus commented on pull request #5768: HADOOP-15984. Jersey 2.x upgrade
hadoop-yetus commented on PR #5768:
URL: https://github.com/apache/hadoop/pull/5768#issuecomment-1613047628
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 42s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 18m 11s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 46s | | trunk passed |
| +1 :green_heart: | compile | 19m 4s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 17m 37s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 4m 49s | | trunk passed |
| +1 :green_heart: | mvnsite | 22m 22s | | trunk passed |
| +1 :green_heart: | javadoc | 19m 48s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 18m 41s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 47s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 39s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 34m 5s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 35s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 1m 24s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 33s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/7/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
| hadoop-yarn-common in the patch failed. |
| -1 :x: | mvninstall | 0m 22s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/7/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt)
| hadoop-yarn-server-applicationhistoryservice in the patch failed. |
| -1 :x: | mvninstall | 0m 20s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/7/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt)
| hadoop-yarn-server-timelineservice in the patch failed. |
| -1 :x: | mvninstall | 0m 35s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/7/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 28s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/7/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/7/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-client in the patch failed. |
| -1 :x: | mvninstall | 0m 33s |
[/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/7/artifact/out/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
| hadoop-mapreduce-client in the patch failed. |
| -1 :
[GitHub] [hadoop] hadoop-yetus commented on pull request #5768: HADOOP-15984. Jersey 2.x upgrade
hadoop-yetus commented on PR #5768:
URL: https://github.com/apache/hadoop/pull/5768#issuecomment-1613044997
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 39s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 23s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 32m 32s | | trunk passed |
| +1 :green_heart: | compile | 18m 47s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 17m 40s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 4m 47s | | trunk passed |
| +1 :green_heart: | mvnsite | 22m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 19m 49s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 18m 40s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +0 :ok: | spotbugs | 0m 46s | | branch/hadoop-project no spotbugs
output file (spotbugsXml.xml) |
| +0 :ok: | spotbugs | 0m 43s | |
branch/hadoop-client-modules/hadoop-client-minicluster no spotbugs output file
(spotbugsXml.xml) |
| +1 :green_heart: | shadedclient | 34m 5s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 34m 33s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 3m 22s | | Maven dependency ordering for patch |
| -1 :x: | mvninstall | 0m 33s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/8/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-common.txt)
| hadoop-yarn-common in the patch failed. |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/8/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-applicationhistoryservice.txt)
| hadoop-yarn-server-applicationhistoryservice in the patch failed. |
| -1 :x: | mvninstall | 0m 22s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/8/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-timelineservice.txt)
| hadoop-yarn-server-timelineservice in the patch failed. |
| -1 :x: | mvninstall | 0m 34s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/8/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 31s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/8/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt)
| hadoop-yarn-server-nodemanager in the patch failed. |
| -1 :x: | mvninstall | 0m 21s |
[/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/8/artifact/out/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-client in the patch failed. |
| -1 :x: | mvninstall | 0m 35s |
[/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/8/artifact/out/patch-mvninstall-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
| hadoop-mapreduce-client in the patch failed. |
| -1 :
[GitHub] [hadoop] haiyang1987 opened a new pull request, #5794: HDFS-17064. Document the usage of the new Balancer "sortTopNodes" and "hotBlockTimeInterval" parameter
haiyang1987 opened a new pull request, #5794: URL: https://github.com/apache/hadoop/pull/5794 ### Description of PR https://issues.apache.org/jira/browse/HDFS-17064 Document the usage of the new Balancer "sortTopNodes" and "hotBlockTimeInterval" parameter -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15984) Update jersey from 1.19 to 2.x
[
https://issues.apache.org/jira/browse/HADOOP-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738512#comment-17738512
]
ASF GitHub Bot commented on HADOOP-15984:
-
hadoop-yetus commented on PR #5768:
URL: https://github.com/apache/hadoop/pull/5768#issuecomment-1612966805
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 1s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 39m 14s | | trunk passed |
| +1 :green_heart: | compile | 22m 24s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| -1 :x: | compile | 1m 36s |
[/branch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
| root in trunk failed with JDK Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09. |
| -0 :warning: | checkstyle | 1m 6s |
[/buildtool-branch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/buildtool-branch-checkstyle-root.txt)
| The patch fails to run checkstyle in root |
| -1 :x: | mvnsite | 0m 48s |
[/branch-mvnsite-hadoop-common-project_hadoop-kms.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-kms.txt)
| hadoop-kms in trunk failed. |
| -1 :x: | mvnsite | 0m 47s |
[/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-client in trunk failed. |
| -1 :x: | mvnsite | 0m 46s |
[/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
| hadoop-mapreduce-client in trunk failed. |
| -1 :x: | mvnsite | 1m 8s |
[/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-catalog_hadoop-yarn-applications-catalog-webapp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-catalog_hadoop-yarn-applications-catalog-webapp.txt)
| hadoop-yarn-applications-catalog-webapp in trunk failed. |
| -1 :x: | mvnsite | 0m 47s |
[/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-examples.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-examples.txt)
| hadoop-mapreduce-examples in trunk failed. |
| -1 :x: | javadoc | 0m 48s |
[/branch-javadoc-hadoop-project-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-javadoc-hadoop-project-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| hadoop-project in trunk failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javadoc | 0m 47s |
[/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| hadoop-common in trunk failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javadoc | 0m 47s |
[/branch-javadoc-hadoop-common-project_hadoop-kms-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-javadoc-hadoop-common-project_hadoop-kms-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| hadoop-kms in trunk failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javad
[GitHub] [hadoop] hadoop-yetus commented on pull request #5768: HADOOP-15984. Jersey 2.x upgrade
hadoop-yetus commented on PR #5768:
URL: https://github.com/apache/hadoop/pull/5768#issuecomment-1612966805
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 44s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 16m 1s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 39m 14s | | trunk passed |
| +1 :green_heart: | compile | 22m 24s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| -1 :x: | compile | 1m 36s |
[/branch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-compile-root-jdkPrivateBuild-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09.txt)
| root in trunk failed with JDK Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09. |
| -0 :warning: | checkstyle | 1m 6s |
[/buildtool-branch-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/buildtool-branch-checkstyle-root.txt)
| The patch fails to run checkstyle in root |
| -1 :x: | mvnsite | 0m 48s |
[/branch-mvnsite-hadoop-common-project_hadoop-kms.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-mvnsite-hadoop-common-project_hadoop-kms.txt)
| hadoop-kms in trunk failed. |
| -1 :x: | mvnsite | 0m 47s |
[/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt)
| hadoop-yarn-client in trunk failed. |
| -1 :x: | mvnsite | 0m 46s |
[/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-client.txt)
| hadoop-mapreduce-client in trunk failed. |
| -1 :x: | mvnsite | 1m 8s |
[/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-catalog_hadoop-yarn-applications-catalog-webapp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-applications_hadoop-yarn-applications-catalog_hadoop-yarn-applications-catalog-webapp.txt)
| hadoop-yarn-applications-catalog-webapp in trunk failed. |
| -1 :x: | mvnsite | 0m 47s |
[/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-examples.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-mvnsite-hadoop-mapreduce-project_hadoop-mapreduce-examples.txt)
| hadoop-mapreduce-examples in trunk failed. |
| -1 :x: | javadoc | 0m 48s |
[/branch-javadoc-hadoop-project-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-javadoc-hadoop-project-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| hadoop-project in trunk failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javadoc | 0m 47s |
[/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| hadoop-common in trunk failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javadoc | 0m 47s |
[/branch-javadoc-hadoop-common-project_hadoop-kms-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-javadoc-hadoop-common-project_hadoop-kms-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt)
| hadoop-kms in trunk failed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1. |
| -1 :x: | javadoc | 0m 47s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5768/6/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.
[GitHub] [hadoop] hadoop-yetus commented on pull request #5792: YARN-11464: queue element is added to any other leaf queue, it's queueType becomes QueueType.PARENT_QUEUE
hadoop-yetus commented on PR #5792:
URL: https://github.com/apache/hadoop/pull/5792#issuecomment-1612873046
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 37s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 17s | | trunk passed |
| +1 :green_heart: | compile | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | compile | 1m 1s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | checkstyle | 1m 0s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 3s | | trunk passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 50s | | trunk passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 2m 9s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 56s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 52s | | the patch passed |
| +1 :green_heart: | compile | 0m 55s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javac | 0m 55s | | the patch passed |
| +1 :green_heart: | compile | 0m 49s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | javac | 0m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 44s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 50s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 45s | | the patch passed with JDK
Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1 |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK
Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| +1 :green_heart: | spotbugs | 1m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 34m 22s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 93m 35s |
[/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5792/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt)
| hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 40s | | The patch does not
generate ASF License warnings. |
| | | 230m 57s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.yarn.server.resourcemanager.reservation.TestFairSchedulerPlanFollower |
| | hadoop.yarn.server.resourcemanager.reservation.TestReservationSystem |
| | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesReservation
|
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
| |
hadoop.yarn.server.resourcemanager.scheduler.fair.TestAllocationFileLoaderService
|
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.43 ServerAPI=1.43 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-5792/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/5792 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 373acf1dde32 4.15.0-212-generic #223-Ubuntu SMP Tue May 23
13:09:22 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / beb456f4e2cd65712ad90ee4c7a907ad3fc986a0 |
| Default Java | Private Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.19+7-post-Ubuntu-0ubuntu120.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_362-8u372-ga~us1-0ubuntu1~20.04-b09 |
|
[GitHub] [hadoop] qijiale76 opened a new pull request, #5793: HDFS-17063. Datanode configures different Capacity Reserved for each …
qijiale76 opened a new pull request, #5793: URL: https://github.com/apache/hadoop/pull/5793 …disk ### Description of PR Now `dfs.datanode.du.reserved` takes effect for all directory of a datanode. This issue allows cluster administrator to configure `dfs.datanode.du.reserved./data/hdfs1/data`, which only take effect for a specific directory. ### How was this patch tested? UT & tested in our cluster ### For code changes: - [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
