[jira] [Commented] (HDFS-17526) getMetadataInputStream should use getShareDeleteFileInputStream for windows
[
https://issues.apache.org/jira/browse/HDFS-17526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883476#comment-17883476
]
ASF GitHub Bot commented on HDFS-17526:
---
hadoop-yetus commented on PR #6826:
URL: https://github.com/apache/hadoop/pull/6826#issuecomment-2365178894
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 33m 15s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 22s | | trunk passed |
| +1 :green_heart: | compile | 1m 20s | | trunk passed |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 30s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 56s | | trunk passed |
| +1 :green_heart: | spotbugs | 3m 22s | | trunk passed |
| +1 :green_heart: | shadedclient | 37m 7s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 59s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed |
| +1 :green_heart: | spotbugs | 3m 19s | | the patch passed |
| +1 :green_heart: | shadedclient | 36m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 231m 37s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 50s | | The patch does not
generate ASF License warnings. |
| | | 401m 25s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6826/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6826 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f62e5f56f761 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 5f042ded4b7bb9ce826cd9e8b65c08a16e79aea9 |
| Default Java | Red Hat, Inc.-1.8.0_412-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6826/2/testReport/ |
| Max. process+thread count | 3701 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6826/2/console |
| versions | git=2.9.5 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> getMetadataInputStream should use getShareDeleteFileInputStream for windows
> ---
>
> Key: HDFS-17526
> URL: https://issues.apache.org/jira/browse/HDFS-17526
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.3.4
>Reporter: Danny Becker
>Priority: Major
> Labels: pull-request-available
>
> In HDFS-10636, the getDataInputStream method uses the
> getShareDeleteFileInputStream for windows, but the getMetaDataInputStream
> does not use this. The following error can happen when a DataNode is trying
> to update the genstamp on a block in Windows.
> DataNode Logs:
> {{Caused by: java.io.IOException: Failed to rename
> G:\data\hdfs\data\current\BP-1\current\finalized\subdir5\subdir16\blk_1_1.meta
> to
> G:\data\hdfs
[jira] [Commented] (HDFS-17254) DataNode httpServer has too many worker threads
[
https://issues.apache.org/jira/browse/HDFS-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883462#comment-17883462
]
ASF GitHub Bot commented on HDFS-17254:
---
ayushtkn commented on code in PR #6307:
URL: https://github.com/apache/hadoop/pull/6307#discussion_r1769487525
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java:
##
@@ -144,7 +147,16 @@ public DatanodeHttpServer(final Configuration conf,
confForCreate.set(FsPermission.UMASK_LABEL, "000");
this.bossGroup = new NioEventLoopGroup();
-this.workerGroup = new NioEventLoopGroup();
+int workerCount = conf.getInt(DFS_DATANODE_NETTY_WORKER_NUM_THREADS_KEY,
+DFS_DATANODE_NETTY_WORKER_NUM_THREADS_DEFAULT);
+if (workerCount < 0) {
+ LOG.warn("The value of " +
+ DFS_DATANODE_NETTY_WORKER_NUM_THREADS_KEY + " is less than 0, will
use default value: " +
+ DFS_DATANODE_NETTY_WORKER_NUM_THREADS_DEFAULT);
+ workerCount = DFS_DATANODE_NETTY_WORKER_NUM_THREADS_DEFAULT;
Review Comment:
Use logger format {} instead of concat
> DataNode httpServer has too many worker threads
> ---
>
> Key: HDFS-17254
> URL: https://issues.apache.org/jira/browse/HDFS-17254
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Liangjun He
>Assignee: Liangjun He
>Priority: Minor
> Labels: pull-request-available
>
> When optimizing the thread number of high-density storage DN, we found the
> number of worker threads for the DataNode httpServer is twice the number of
> available cores on node , resulting in too many threads. We can change this
> to be configurable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17565) EC: dfs.datanode.ec.reconstruction.threads should support dynamic reconfigured.
[
https://issues.apache.org/jira/browse/HDFS-17565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883450#comment-17883450
]
ASF GitHub Bot commented on HDFS-17565:
---
ayushtkn commented on code in PR #6928:
URL: https://github.com/apache/hadoop/pull/6928#discussion_r1769491905
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/erasurecode/ErasureCodingWorker.java:
##
@@ -172,4 +175,19 @@ public void shutDown() {
public float getXmitWeight() {
return xmitWeight;
}
+
+ public void setStripedReconstructionPoolSize(int size) {
+Preconditions.checkArgument(size > 0,
+DFS_DN_EC_RECONSTRUCTION_THREADS_KEY + " should be greater than 0");
+this.stripedReconstructionPool.setCorePoolSize(size);
+this.stripedReconstructionPool.setMaximumPoolSize(size);
+ }
+
+ @VisibleForTesting
+ public int getStripedReconstructionPoolSize() {
+int poolSize = this.stripedReconstructionPool.getCorePoolSize();
+Preconditions.checkArgument(poolSize ==
this.stripedReconstructionPool.getMaximumPoolSize(),
+"The maximum pool size should be equal to core pool size");
Review Comment:
I think this isn't required, this looks like validating the
``ThreadPoolExecutor``, that ain't a hadoop thing & isn't required here in the
scope either
> EC: dfs.datanode.ec.reconstruction.threads should support dynamic
> reconfigured.
> ---
>
> Key: HDFS-17565
> URL: https://issues.apache.org/jira/browse/HDFS-17565
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Chenyu Zheng
>Assignee: Chenyu Zheng
>Priority: Major
> Labels: pull-request-available
>
> dfs.datanode.ec.reconstruction.threads should support dynamic reconfigured,
> then we can adjust the speed of ec block copy. Especially HDFS-17550 wanna
> decommissioning DataNode by EC block reconstruction.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17254) DataNode httpServer has too many worker threads
[
https://issues.apache.org/jira/browse/HDFS-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883449#comment-17883449
]
ASF GitHub Bot commented on HDFS-17254:
---
ayushtkn commented on code in PR #6307:
URL: https://github.com/apache/hadoop/pull/6307#discussion_r1769487525
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/web/DatanodeHttpServer.java:
##
@@ -144,7 +147,16 @@ public DatanodeHttpServer(final Configuration conf,
confForCreate.set(FsPermission.UMASK_LABEL, "000");
this.bossGroup = new NioEventLoopGroup();
-this.workerGroup = new NioEventLoopGroup();
+int workerCount = conf.getInt(DFS_DATANODE_NETTY_WORKER_NUM_THREADS_KEY,
+DFS_DATANODE_NETTY_WORKER_NUM_THREADS_DEFAULT);
+if (workerCount < 0) {
+ LOG.warn("The value of " +
+ DFS_DATANODE_NETTY_WORKER_NUM_THREADS_KEY + " is less than 0, will
use default value: " +
+ DFS_DATANODE_NETTY_WORKER_NUM_THREADS_DEFAULT);
+ workerCount = DFS_DATANODE_NETTY_WORKER_NUM_THREADS_DEFAULT;
Review Comment:
you logger format {} instead of concat
> DataNode httpServer has too many worker threads
> ---
>
> Key: HDFS-17254
> URL: https://issues.apache.org/jira/browse/HDFS-17254
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Liangjun He
>Assignee: Liangjun He
>Priority: Minor
> Labels: pull-request-available
>
> When optimizing the thread number of high-density storage DN, we found the
> number of worker threads for the DataNode httpServer is twice the number of
> available cores on node , resulting in too many threads. We can change this
> to be configurable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17626) Reduce lock contention at datanode startup
[
https://issues.apache.org/jira/browse/HDFS-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883384#comment-17883384
]
ASF GitHub Bot commented on HDFS-17626:
---
virajjasani commented on code in PR #7053:
URL: https://github.com/apache/hadoop/pull/7053#discussion_r1769078801
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java:
##
@@ -258,7 +258,7 @@ NamespaceInfo retrieveNamespaceInfo() throws IOException {
while (shouldRun()) {
try {
nsInfo = bpNamenode.versionRequest();
-LOG.debug(this + " received versionRequest response: " + nsInfo);
+LOG.debug("{} received versionRequest response: {}", this, nsInfo);
Review Comment:
LOG.debug already does isDebugEnabled() internally, the original issue is
String concatenation with heavy object.
This change is sufficient IMO.
> Reduce lock contention at datanode startup
> --
>
> Key: HDFS-17626
> URL: https://issues.apache.org/jira/browse/HDFS-17626
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2024-09-18-20-45-56-999.png
>
>
> During the datanode startup process, there is a debug log, because there is
> no LOG.isDebugEnabled() guard, so even if debug is not enabled, the read lock
> will be obtained. The guard should be added here to reduce lock contention.
> !image-2024-09-18-20-45-56-999.png|width=333,height=263!
> !https://docs.corp.vipshop.com/uploader/f/4DSEukZKf6cV5VRY.png?accessToken=eyJhbGciOiJIUzI1NiIsImtpZCI6ImRlZmF1bHQiLCJ0eXAiOiJKV1QifQ.eyJleHAiOjE3MjY2NjQxNjYsImZpbGVHVUlEIjoiQWxvNE5uOU9OYko2aDJ4WCIsImlhdCI6MTcyNjY2MzU2NiwiaXNzIjoidXBsb2FkZXJfYWNjZXNzX3Jlc291cmNlIiwidXNlcklkIjo2MTYyMTQwfQ.DwDBnJ6I8vCFd14A-wsq2oLU5a0rcPoUvq49Z4aWg2A|width=334,height=133!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17569) Setup Effective Work Number when Generating Block Reconstruction Work
[
https://issues.apache.org/jira/browse/HDFS-17569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17883372#comment-17883372
]
ASF GitHub Bot commented on HDFS-17569:
---
hadoop-yetus commented on PR #6924:
URL: https://github.com/apache/hadoop/pull/6924#issuecomment-2364311859
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 5 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 9s | | trunk passed |
| +1 :green_heart: | compile | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 0m 38s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 40s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 49s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 1m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 24m 31s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 41s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 0m 41s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 31s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6924/16/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 18 new + 274 unchanged
- 0 fixed = 292 total (was 274) |
| +1 :green_heart: | mvnsite | 0m 37s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 38s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 11s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 1m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 197m 13s | | hadoop-hdfs in the patch
passed. |
| -1 :x: | asflicense | 0m 32s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6924/16/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 290m 16s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6924/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6924 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 0ee5e8423f4c 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 749a810dcd7c3f48cd02342656bc7fbc4b47dfa7 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6924/16/testReport/ |
| Max. process+thread count | 4900 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-h
[jira] [Commented] (HDFS-17603) [FGL] Abstract a LockManager to manage locks
[ https://issues.apache.org/jira/browse/HDFS-17603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882957#comment-17882957 ] ASF GitHub Bot commented on HDFS-17603: --- hadoop-yetus commented on PR #7054: URL: https://github.com/apache/hadoop/pull/7054#issuecomment-2360539733 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Logfile | Comment | |::|--:|:|::|:---:| | +0 :ok: | reexec | 0m 0s | | Docker mode activated. | | -1 :x: | patch | 0m 20s | | https://github.com/apache/hadoop/pull/7054 does not apply to HDFS-17384. Rebase required? Wrong Branch? See https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help. | | Subsystem | Report/Notes | |--:|:-| | GITHUB PR | https://github.com/apache/hadoop/pull/7054 | | Console output | https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7054/1/console | | versions | git=2.34.1 | | Powered by | Apache Yetus 0.14.0 https://yetus.apache.org | This message was automatically generated. > [FGL] Abstract a LockManager to manage locks > - > > Key: HDFS-17603 > URL: https://issues.apache.org/jira/browse/HDFS-17603 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: Felix N >Priority: Major > Labels: pull-request-available > Attachments: image-2024-08-14-17-17-28-792.png > > > Abstract a LockManager to manage locks. > Some requirements for this LockManager: > * Cached a fixed number lock instances, such as: 1000 > * Assign a Lock instance to a key and keep this mapping until the key > released this instance > * This LockManager needs a high performance, such as: QPS 1000w > > Some implementations that we can refer to: > * alluxio.collections.LockPool in Alluxio > * Implementation in MEITUAN > !image-2024-08-14-17-17-28-792.png|width=205,height=196! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17603) [FGL] Abstract a LockManager to manage locks
[
https://issues.apache.org/jira/browse/HDFS-17603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882966#comment-17882966
]
ASF GitHub Bot commented on HDFS-17603:
---
hadoop-yetus commented on PR #7054:
URL: https://github.com/apache/hadoop/pull/7054#issuecomment-2360628592
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 18m 2s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ HDFS-17384 Compile Tests _ |
| -1 :x: | mvninstall | 5m 48s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7054/2/artifact/out/branch-mvninstall-root.txt)
| root in HDFS-17384 failed. |
| -1 :x: | compile | 1m 37s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7054/2/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
| hadoop-hdfs in HDFS-17384 failed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | compile | 0m 43s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7054/2/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt)
| hadoop-hdfs in HDFS-17384 failed with JDK Private
Build-1.8.0_422-8u422-b05-1~20.04-b05. |
| +1 :green_heart: | checkstyle | 1m 8s | | HDFS-17384 passed |
| -1 :x: | mvnsite | 0m 49s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7054/2/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in HDFS-17384 failed. |
| -1 :x: | javadoc | 0m 44s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7054/2/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
| hadoop-hdfs in HDFS-17384 failed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | javadoc | 0m 35s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7054/2/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt)
| hadoop-hdfs in HDFS-17384 failed with JDK Private
Build-1.8.0_422-8u422-b05-1~20.04-b05. |
| -1 :x: | spotbugs | 0m 45s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7054/2/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in HDFS-17384 failed. |
| -1 :x: | shadedclient | 5m 0s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 23s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7054/2/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 0m 24s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7054/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | javac | 0m 24s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7054/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | compile | 0m 24s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt](https://ci-hadoop.apache.org/job/ha
[jira] [Commented] (HDFS-17603) [FGL] Abstract a LockManager to manage locks
[ https://issues.apache.org/jira/browse/HDFS-17603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882958#comment-17882958 ] ASF GitHub Bot commented on HDFS-17603: --- kokon191 commented on PR #7054: URL: https://github.com/apache/hadoop/pull/7054#issuecomment-2360542315 @ZanderXu Can you take a look when you're free? Thanks! > [FGL] Abstract a LockManager to manage locks > - > > Key: HDFS-17603 > URL: https://issues.apache.org/jira/browse/HDFS-17603 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: Felix N >Priority: Major > Labels: pull-request-available > Attachments: image-2024-08-14-17-17-28-792.png > > > Abstract a LockManager to manage locks. > Some requirements for this LockManager: > * Cached a fixed number lock instances, such as: 1000 > * Assign a Lock instance to a key and keep this mapping until the key > released this instance > * This LockManager needs a high performance, such as: QPS 1000w > > Some implementations that we can refer to: > * alluxio.collections.LockPool in Alluxio > * Implementation in MEITUAN > !image-2024-08-14-17-17-28-792.png|width=205,height=196! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17603) [FGL] Abstract a LockManager to manage locks
[ https://issues.apache.org/jira/browse/HDFS-17603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17603: -- Labels: pull-request-available (was: ) > [FGL] Abstract a LockManager to manage locks > - > > Key: HDFS-17603 > URL: https://issues.apache.org/jira/browse/HDFS-17603 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: Felix N >Priority: Major > Labels: pull-request-available > Attachments: image-2024-08-14-17-17-28-792.png > > > Abstract a LockManager to manage locks. > Some requirements for this LockManager: > * Cached a fixed number lock instances, such as: 1000 > * Assign a Lock instance to a key and keep this mapping until the key > released this instance > * This LockManager needs a high performance, such as: QPS 1000w > > Some implementations that we can refer to: > * alluxio.collections.LockPool in Alluxio > * Implementation in MEITUAN > !image-2024-08-14-17-17-28-792.png|width=205,height=196! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17603) [FGL] Abstract a LockManager to manage locks
[ https://issues.apache.org/jira/browse/HDFS-17603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882955#comment-17882955 ] ASF GitHub Bot commented on HDFS-17603: --- kokon191 opened a new pull request, #7054: URL: https://github.com/apache/hadoop/pull/7054 A `LockPoolManager` that stores locks and a ref count. Ref count increments upon lock acquisition, decrements upon lock release. Lock is ejected when ref count hits 0. Cached locks get an extra ref count and won't be ejected by lock releases. Cached locks are updated periodically with a `PromotionService`, engined by the `PromotionMonitor` that tracks which locks are the most active. > [FGL] Abstract a LockManager to manage locks > - > > Key: HDFS-17603 > URL: https://issues.apache.org/jira/browse/HDFS-17603 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: Felix N >Priority: Major > Attachments: image-2024-08-14-17-17-28-792.png > > > Abstract a LockManager to manage locks. > Some requirements for this LockManager: > * Cached a fixed number lock instances, such as: 1000 > * Assign a Lock instance to a key and keep this mapping until the key > released this instance > * This LockManager needs a high performance, such as: QPS 1000w > > Some implementations that we can refer to: > * alluxio.collections.LockPool in Alluxio > * Implementation in MEITUAN > !image-2024-08-14-17-17-28-792.png|width=205,height=196! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17626) Reduce lock contention at datanode startup
[
https://issues.apache.org/jira/browse/HDFS-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882934#comment-17882934
]
ASF GitHub Bot commented on HDFS-17626:
---
hadoop-yetus commented on PR #7053:
URL: https://github.com/apache/hadoop/pull/7053#issuecomment-2360299403
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 52s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 42s | | trunk passed |
| +1 :green_heart: | compile | 1m 24s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 1m 12s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 26s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 16s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 20s | | trunk passed |
| +1 :green_heart: | shadedclient | 37m 10s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 17s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 58s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 36s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 2s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 265m 38s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 407m 37s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7053/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/7053 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 0f87bfa7b622 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / fc9d069d22cf1d0ef94e0f6f70841e84ab20c4de |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7053/2/testReport/ |
| Max. process+thread count | 3146 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7053/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
[jira] [Commented] (HDFS-17626) Reduce lock contention at datanode startup
[
https://issues.apache.org/jira/browse/HDFS-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882923#comment-17882923
]
ASF GitHub Bot commented on HDFS-17626:
---
KeeProMise commented on code in PR #7053:
URL: https://github.com/apache/hadoop/pull/7053#discussion_r1766306651
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java:
##
@@ -258,7 +258,7 @@ NamespaceInfo retrieveNamespaceInfo() throws IOException {
while (shouldRun()) {
try {
nsInfo = bpNamenode.versionRequest();
-LOG.debug(this + " received versionRequest response: " + nsInfo);
+LOG.debug("{} received versionRequest response: {}", this, nsInfo);
Review Comment:
```java
if (LOG.isDebugEnabled()) {
LOG.debug("{} received versionRequest response: {}", this, nsInfo);
}
```
> Reduce lock contention at datanode startup
> --
>
> Key: HDFS-17626
> URL: https://issues.apache.org/jira/browse/HDFS-17626
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2024-09-18-20-45-56-999.png
>
>
> During the datanode startup process, there is a debug log, because there is
> no LOG.isDebugEnabled() guard, so even if debug is not enabled, the read lock
> will be obtained. The guard should be added here to reduce lock contention.
> !image-2024-09-18-20-45-56-999.png|width=333,height=263!
> !https://docs.corp.vipshop.com/uploader/f/4DSEukZKf6cV5VRY.png?accessToken=eyJhbGciOiJIUzI1NiIsImtpZCI6ImRlZmF1bHQiLCJ0eXAiOiJKV1QifQ.eyJleHAiOjE3MjY2NjQxNjYsImZpbGVHVUlEIjoiQWxvNE5uOU9OYko2aDJ4WCIsImlhdCI6MTcyNjY2MzU2NiwiaXNzIjoidXBsb2FkZXJfYWNjZXNzX3Jlc291cmNlIiwidXNlcklkIjo2MTYyMTQwfQ.DwDBnJ6I8vCFd14A-wsq2oLU5a0rcPoUvq49Z4aWg2A|width=334,height=133!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17626) Reduce lock contention at datanode startup
[
https://issues.apache.org/jira/browse/HDFS-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882922#comment-17882922
]
ASF GitHub Bot commented on HDFS-17626:
---
KeeProMise commented on code in PR #7053:
URL: https://github.com/apache/hadoop/pull/7053#discussion_r1766301368
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java:
##
@@ -258,7 +258,7 @@ NamespaceInfo retrieveNamespaceInfo() throws IOException {
while (shouldRun()) {
try {
nsInfo = bpNamenode.versionRequest();
-LOG.debug(this + " received versionRequest response: " + nsInfo);
+LOG.debug("{} received versionRequest response: {}", this, nsInfo);
Review Comment:
HI, IMO, "if (LOG.isDebugEnabled()) {...}" is better.
> Reduce lock contention at datanode startup
> --
>
> Key: HDFS-17626
> URL: https://issues.apache.org/jira/browse/HDFS-17626
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2024-09-18-20-45-56-999.png
>
>
> During the datanode startup process, there is a debug log, because there is
> no LOG.isDebugEnabled() guard, so even if debug is not enabled, the read lock
> will be obtained. The guard should be added here to reduce lock contention.
> !image-2024-09-18-20-45-56-999.png|width=333,height=263!
> !https://docs.corp.vipshop.com/uploader/f/4DSEukZKf6cV5VRY.png?accessToken=eyJhbGciOiJIUzI1NiIsImtpZCI6ImRlZmF1bHQiLCJ0eXAiOiJKV1QifQ.eyJleHAiOjE3MjY2NjQxNjYsImZpbGVHVUlEIjoiQWxvNE5uOU9OYko2aDJ4WCIsImlhdCI6MTcyNjY2MzU2NiwiaXNzIjoidXBsb2FkZXJfYWNjZXNzX3Jlc291cmNlIiwidXNlcklkIjo2MTYyMTQwfQ.DwDBnJ6I8vCFd14A-wsq2oLU5a0rcPoUvq49Z4aWg2A|width=334,height=133!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17626) Reduce lock contention at datanode startup
[
https://issues.apache.org/jira/browse/HDFS-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882841#comment-17882841
]
ASF GitHub Bot commented on HDFS-17626:
---
tomscut commented on code in PR #7053:
URL: https://github.com/apache/hadoop/pull/7053#discussion_r1765980808
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java:
##
@@ -258,7 +258,9 @@ NamespaceInfo retrieveNamespaceInfo() throws IOException {
while (shouldRun()) {
try {
nsInfo = bpNamenode.versionRequest();
-LOG.debug(this + " received versionRequest response: " + nsInfo);
+if (LOG.isDebugEnabled()) {
+ LOG.debug(this + " received versionRequest response: " + nsInfo);
+}
Review Comment:
Thank you for your advice, this way can also achieve the same effect.
> Reduce lock contention at datanode startup
> --
>
> Key: HDFS-17626
> URL: https://issues.apache.org/jira/browse/HDFS-17626
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2024-09-18-20-45-56-999.png
>
>
> During the datanode startup process, there is a debug log, because there is
> no LOG.isDebugEnabled() guard, so even if debug is not enabled, the read lock
> will be obtained. The guard should be added here to reduce lock contention.
> !image-2024-09-18-20-45-56-999.png|width=333,height=263!
> !https://docs.corp.vipshop.com/uploader/f/4DSEukZKf6cV5VRY.png?accessToken=eyJhbGciOiJIUzI1NiIsImtpZCI6ImRlZmF1bHQiLCJ0eXAiOiJKV1QifQ.eyJleHAiOjE3MjY2NjQxNjYsImZpbGVHVUlEIjoiQWxvNE5uOU9OYko2aDJ4WCIsImlhdCI6MTcyNjY2MzU2NiwiaXNzIjoidXBsb2FkZXJfYWNjZXNzX3Jlc291cmNlIiwidXNlcklkIjo2MTYyMTQwfQ.DwDBnJ6I8vCFd14A-wsq2oLU5a0rcPoUvq49Z4aWg2A|width=334,height=133!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17626) Reduce lock contention at datanode startup
[
https://issues.apache.org/jira/browse/HDFS-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882793#comment-17882793
]
ASF GitHub Bot commented on HDFS-17626:
---
hadoop-yetus commented on PR #7053:
URL: https://github.com/apache/hadoop/pull/7053#issuecomment-2359267690
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 17s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 14s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 17s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 14s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 1m 14s | | the patch passed |
| +1 :green_heart: | compile | 1m 9s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 1m 9s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 57s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 14s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 2s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 17s | | the patch passed |
| +1 :green_heart: | shadedclient | 37m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 267m 8s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 45s | | The patch does not
generate ASF License warnings. |
| | | 408m 19s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7053/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/7053 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 75a8e6b6150d 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 11aa442354fef3ea355d2bd33aff5fb221060ea8 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7053/1/testReport/ |
| Max. process+thread count | 3144 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7053/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
[jira] [Commented] (HDFS-17626) Reduce lock contention at datanode startup
[
https://issues.apache.org/jira/browse/HDFS-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882717#comment-17882717
]
ASF GitHub Bot commented on HDFS-17626:
---
ayushtkn commented on code in PR #7053:
URL: https://github.com/apache/hadoop/pull/7053#discussion_r1765089605
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BPServiceActor.java:
##
@@ -258,7 +258,9 @@ NamespaceInfo retrieveNamespaceInfo() throws IOException {
while (shouldRun()) {
try {
nsInfo = bpNamenode.versionRequest();
-LOG.debug(this + " received versionRequest response: " + nsInfo);
+if (LOG.isDebugEnabled()) {
+ LOG.debug(this + " received versionRequest response: " + nsInfo);
+}
Review Comment:
I think if we change this to
```
LOG.debug("{} received versionRequest response: {}", this, nsInfo);
```
in this case as well, ``toString()`` won't be invoked & we can get away
without having this ``isDebugEnabled`` gaurd
> Reduce lock contention at datanode startup
> --
>
> Key: HDFS-17626
> URL: https://issues.apache.org/jira/browse/HDFS-17626
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Minor
> Labels: pull-request-available
> Attachments: image-2024-09-18-20-45-56-999.png
>
>
> During the datanode startup process, there is a debug log, because there is
> no LOG.isDebugEnabled() guard, so even if debug is not enabled, the read lock
> will be obtained. The guard should be added here to reduce lock contention.
> !image-2024-09-18-20-45-56-999.png|width=333,height=263!
> !https://docs.corp.vipshop.com/uploader/f/4DSEukZKf6cV5VRY.png?accessToken=eyJhbGciOiJIUzI1NiIsImtpZCI6ImRlZmF1bHQiLCJ0eXAiOiJKV1QifQ.eyJleHAiOjE3MjY2NjQxNjYsImZpbGVHVUlEIjoiQWxvNE5uOU9OYko2aDJ4WCIsImlhdCI6MTcyNjY2MzU2NiwiaXNzIjoidXBsb2FkZXJfYWNjZXNzX3Jlc291cmNlIiwidXNlcklkIjo2MTYyMTQwfQ.DwDBnJ6I8vCFd14A-wsq2oLU5a0rcPoUvq49Z4aWg2A|width=334,height=133!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17626) Reduce lock contention at datanode startup
[ https://issues.apache.org/jira/browse/HDFS-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17626: -- Labels: pull-request-available (was: ) > Reduce lock contention at datanode startup > -- > > Key: HDFS-17626 > URL: https://issues.apache.org/jira/browse/HDFS-17626 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Tao Li >Assignee: Tao Li >Priority: Minor > Labels: pull-request-available > Attachments: image-2024-09-18-20-45-56-999.png > > > During the datanode startup process, there is a debug log, because there is > no LOG.isDebugEnabled() guard, so even if debug is not enabled, the read lock > will be obtained. The guard should be added here to reduce lock contention. > !image-2024-09-18-20-45-56-999.png|width=333,height=263! > !https://docs.corp.vipshop.com/uploader/f/4DSEukZKf6cV5VRY.png?accessToken=eyJhbGciOiJIUzI1NiIsImtpZCI6ImRlZmF1bHQiLCJ0eXAiOiJKV1QifQ.eyJleHAiOjE3MjY2NjQxNjYsImZpbGVHVUlEIjoiQWxvNE5uOU9OYko2aDJ4WCIsImlhdCI6MTcyNjY2MzU2NiwiaXNzIjoidXBsb2FkZXJfYWNjZXNzX3Jlc291cmNlIiwidXNlcklkIjo2MTYyMTQwfQ.DwDBnJ6I8vCFd14A-wsq2oLU5a0rcPoUvq49Z4aWg2A|width=334,height=133! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17626) Reduce lock contention at datanode startup
[ https://issues.apache.org/jira/browse/HDFS-17626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17882703#comment-17882703 ] ASF GitHub Bot commented on HDFS-17626: --- tomscut opened a new pull request, #7053: URL: https://github.com/apache/hadoop/pull/7053 JIRA: HDFS-17626 ### Description of PR During the datanode startup process, there is a debug log, because there is no LOG.isDebugEnabled() guard, so even if debug is not enabled, the read lock will be obtained. The guard should be added here to reduce lock contention. ### How was this patch tested? Not required > Reduce lock contention at datanode startup > -- > > Key: HDFS-17626 > URL: https://issues.apache.org/jira/browse/HDFS-17626 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Tao Li >Assignee: Tao Li >Priority: Minor > Attachments: image-2024-09-18-20-45-56-999.png > > > During the datanode startup process, there is a debug log, because there is > no LOG.isDebugEnabled() guard, so even if debug is not enabled, the read lock > will be obtained. The guard should be added here to reduce lock contention. > !image-2024-09-18-20-45-56-999.png|width=333,height=263! > !https://docs.corp.vipshop.com/uploader/f/4DSEukZKf6cV5VRY.png?accessToken=eyJhbGciOiJIUzI1NiIsImtpZCI6ImRlZmF1bHQiLCJ0eXAiOiJKV1QifQ.eyJleHAiOjE3MjY2NjQxNjYsImZpbGVHVUlEIjoiQWxvNE5uOU9OYko2aDJ4WCIsImlhdCI6MTcyNjY2MzU2NiwiaXNzIjoidXBsb2FkZXJfYWNjZXNzX3Jlc291cmNlIiwidXNlcklkIjo2MTYyMTQwfQ.DwDBnJ6I8vCFd14A-wsq2oLU5a0rcPoUvq49Z4aWg2A|width=334,height=133! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17624) The availableCount will be deducted only if the excludedNode is included in the selected scope.
[
https://issues.apache.org/jira/browse/HDFS-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881537#comment-17881537
]
ASF GitHub Bot commented on HDFS-17624:
---
hadoop-yetus commented on PR #7042:
URL: https://github.com/apache/hadoop/pull/7042#issuecomment-2348829791
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 58s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 31s | | trunk passed |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 46s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 1m 48s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 22s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 35s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 0m 35s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 30s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 35s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 3s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 1m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 199m 19s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7042/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 294m 17s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.net.TestDFSNetworkTopology |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7042/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/7042 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 8e6d8c50326f 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6c4fa42327348ac515c7065f20996fbc1fe66a3b |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7042/1/testReport/ |
| Max. process+thread count | 4419 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibra
[jira] [Updated] (HDFS-17624) The availableCount will be deducted only if the excludedNode is included in the selected scope.
[ https://issues.apache.org/jira/browse/HDFS-17624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17624: -- Labels: pull-request-available (was: ) > The availableCount will be deducted only if the excludedNode is included in > the selected scope. > --- > > Key: HDFS-17624 > URL: https://issues.apache.org/jira/browse/HDFS-17624 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: fuchaohong >Priority: Major > Labels: pull-request-available > > Presently if chosen scope is /default/rack1 and excluded node is > /default/rack2/host2. Then the available count will be deducted. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17620) Better block placement for small EC files
[
https://issues.apache.org/jira/browse/HDFS-17620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881473#comment-17881473
]
ASF GitHub Bot commented on HDFS-17620:
---
hadoop-yetus commented on PR #7035:
URL: https://github.com/apache/hadoop/pull/7035#issuecomment-2348089297
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 55s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 58s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 1m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 12s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 48s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 34s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 50s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7035/2/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 0m 54s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7035/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | javac | 0m 54s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7035/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | compile | 0m 49s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7035/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt)
| hadoop-hdfs in the patch failed with JDK Private
Build-1.8.0_422-8u422-b05-1~20.04-b05. |
| -1 :x: | javac | 0m 49s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7035/2/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt)
| hadoop-hdfs in the patch failed with JDK Private
Build-1.8.0_422-8u422-b05-1~20.04-b05. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 58s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7035/2/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 8 new + 13 unchanged -
0 fixed = 21 total (was 13) |
| -1 :x: | mvnsite | 0m 53s |
[/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7035/2/artifact/out/patch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| +1 :green_heart: | javadoc | 1m 0s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 34s | | the patch passed with JDK
Private Build-1.8.0_4
[jira] [Commented] (HDFS-17545) [ARR] router async rpc client.
[
https://issues.apache.org/jira/browse/HDFS-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881356#comment-17881356
]
ASF GitHub Bot commented on HDFS-17545:
---
hadoop-yetus commented on PR #6871:
URL: https://github.com/apache/hadoop/pull/6871#issuecomment-2346698787
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ HDFS-17531 Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 48s | | HDFS-17531 passed |
| +1 :green_heart: | compile | 0m 30s | | HDFS-17531 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 0m 25s | | HDFS-17531 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 22s | | HDFS-17531 passed |
| +1 :green_heart: | mvnsite | 0m 29s | | HDFS-17531 passed |
| +1 :green_heart: | javadoc | 0m 31s | | HDFS-17531 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 20s | | HDFS-17531 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 55s | | HDFS-17531 passed |
| +1 :green_heart: | shadedclient | 20m 36s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 49s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 19s | | the patch passed |
| +1 :green_heart: | compile | 0m 24s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 0m 24s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 14s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 20s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 54s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 27m 16s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 26s | | The patch does not
generate ASF License warnings. |
| | | 109m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6871/19/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6871 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 13d151344b1a 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HDFS-17531 / 46255050b486a73d12458651c4b6390e534576cd |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6871/19/testReport/ |
| Max. process+thread count | 3856 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
htt
[jira] [Commented] (HDFS-17545) [ARR] router async rpc client.
[
https://issues.apache.org/jira/browse/HDFS-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881268#comment-17881268
]
ASF GitHub Bot commented on HDFS-17545:
---
KeeProMise opened a new pull request, #6871:
URL: https://github.com/apache/hadoop/pull/6871
### Description of PR
please see: https://issues.apache.org/jira/browse/HDFS-17545
NOTE: This is a sub-pull request (PR) related to
[HDFS-17531](https://issues.apache.org/jira/browse/HDFS-17531)(Asynchronous
router RPC). For more details or context, please refer to the main issue
[HDFS-17531](https://issues.apache.org/jira/browse/HDFS-17531)
More detailed documentation:
[HDFS-17531](https://issues.apache.org/jira/browse/HDFS-17531) Router
asynchronous rpc implementation.pdf and Aynchronous router.pdf
**Main modifications:**
- RouterRpcClient.java: The original functionality remains unchanged; common
methods have been extracted from the original methods.
- RouterAsyncRpcClient.java: An asynchronous implementation of
RouterRpcClient, inheriting from RouterRpcClient.
- Added configuration for asynchronous feature toggle, as well as the number
of asynchronous handlers and responders.
- Using ThreadLocalContext to maintain thread local variables, ensuring that
thread local variables can be correctly passed between handler,
asyncRouterHandler, and asyncRouterResponder.
### How was this patch tested?
new UT TestRouterAsyncRpcClient
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> [ARR] router async rpc client.
> --
>
> Key: HDFS-17545
> URL: https://issues.apache.org/jira/browse/HDFS-17545
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Labels: pull-request-available
>
> *Describe*
> 1. Mainly using AsyncUtil to implement {*}RouterAsyncRpcClient{*}, this class
> extends RouterRpcClient, enabling the {*}invoiceAll{*}, {*}invoiceMethod{*},
> {*}invoiceSequential{*}, {*}invoiceConcurrent{*}, and *invoiceSingle* methods
> to support asynchrony.
> 2. Use two thread pools, *asyncRouterHandler* and {*}asyncRouterResponder{*},
> to handle asynchronous requests and responses, respectively.
> 3. Added {*}DFS_ROUTER_RPC_ENABLE_ASYNC{*},
> {*}DFS_ROUTER_RPC_ASYNC_HANDLER_COUNT{*},
> *DFS_ROUTER_RPC_ASYNC_RESPONDER_COUNT_DEFAULT* to configure whether to use
> async router, as well as the number of asyncRouterHandlers and
> asyncRouterResponders.
> 4. Using *ThreadLocalContext* to maintain thread local variables, ensuring
> that thread local variables can be correctly passed between handler,
> asyncRouterHandler, and asyncRouterResponder.
>
> *Test*
> new UT TestRouterAsyncRpcClient
> Note: For discussions on *AsyncUtil* and client {*}protocolPB{*}, please
> refer to HDFS-17543 and HDFS-17544.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17545) [ARR] router async rpc client.
[
https://issues.apache.org/jira/browse/HDFS-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881267#comment-17881267
]
ASF GitHub Bot commented on HDFS-17545:
---
KeeProMise closed pull request #6871: HDFS-17545. [ARR] router async rpc client.
URL: https://github.com/apache/hadoop/pull/6871
> [ARR] router async rpc client.
> --
>
> Key: HDFS-17545
> URL: https://issues.apache.org/jira/browse/HDFS-17545
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Labels: pull-request-available
>
> *Describe*
> 1. Mainly using AsyncUtil to implement {*}RouterAsyncRpcClient{*}, this class
> extends RouterRpcClient, enabling the {*}invoiceAll{*}, {*}invoiceMethod{*},
> {*}invoiceSequential{*}, {*}invoiceConcurrent{*}, and *invoiceSingle* methods
> to support asynchrony.
> 2. Use two thread pools, *asyncRouterHandler* and {*}asyncRouterResponder{*},
> to handle asynchronous requests and responses, respectively.
> 3. Added {*}DFS_ROUTER_RPC_ENABLE_ASYNC{*},
> {*}DFS_ROUTER_RPC_ASYNC_HANDLER_COUNT{*},
> *DFS_ROUTER_RPC_ASYNC_RESPONDER_COUNT_DEFAULT* to configure whether to use
> async router, as well as the number of asyncRouterHandlers and
> asyncRouterResponders.
> 4. Using *ThreadLocalContext* to maintain thread local variables, ensuring
> that thread local variables can be correctly passed between handler,
> asyncRouterHandler, and asyncRouterResponder.
>
> *Test*
> new UT TestRouterAsyncRpcClient
> Note: For discussions on *AsyncUtil* and client {*}protocolPB{*}, please
> refer to HDFS-17543 and HDFS-17544.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17545) [ARR] router async rpc client.
[
https://issues.apache.org/jira/browse/HDFS-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881261#comment-17881261
]
ASF GitHub Bot commented on HDFS-17545:
---
hadoop-yetus commented on PR #6871:
URL: https://github.com/apache/hadoop/pull/6871#issuecomment-2345847507
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 1s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ HDFS-17531 Compile Tests _ |
| -1 :x: | mvninstall | 35m 28s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6871/18/artifact/out/branch-mvninstall-root.txt)
| root in HDFS-17531 failed. |
| +1 :green_heart: | compile | 0m 28s | | HDFS-17531 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 0m 25s | | HDFS-17531 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 21s | | HDFS-17531 passed |
| +1 :green_heart: | mvnsite | 0m 28s | | HDFS-17531 passed |
| +1 :green_heart: | javadoc | 0m 31s | | HDFS-17531 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 22s | | HDFS-17531 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 55s | | HDFS-17531 passed |
| +1 :green_heart: | shadedclient | 20m 20s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 20m 32s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 0m 18s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 19s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 16s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 49s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 10s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 27m 19s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 27s | | The patch does not
generate ASF License warnings. |
| | | 111m 48s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6871/18/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6871 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux a0ae766166d6 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HDFS-17531 / 4e7e35471cd5fbad975e3549dd8129ee6d5f791b |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6871/18/testReport/ |
| Max. process+thread count | 4229 (vs. uli
[jira] [Commented] (HDFS-17401) EC: Excess internal block may not be able to be deleted correctly when it's stored in fallback storage
[ https://issues.apache.org/jira/browse/HDFS-17401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17881223#comment-17881223 ] ASF GitHub Bot commented on HDFS-17401: --- RuinanGu commented on PR #6597: URL: https://github.com/apache/hadoop/pull/6597#issuecomment-2345638573 @haiyang1987 Could you please take a look? > EC: Excess internal block may not be able to be deleted correctly when it's > stored in fallback storage > -- > > Key: HDFS-17401 > URL: https://issues.apache.org/jira/browse/HDFS-17401 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.3.6 >Reporter: Ruinan Gu >Assignee: Ruinan Gu >Priority: Major > Labels: pull-request-available > > Excess internal block can't be deleted correctly when it's stored in fallback > storage. > Simple case: > EC-RS-6-3-1024k file is stored using ALL_SSD storage policy(SSD is default > storage type and DISK is fallback storage type), if the block group is as > follows > [0(SSD), 0(SSD), 1(SSD), 2(SSD), 3(SSD), 4(SSD), 5(SSD), 6(SSD), 7(SSD), > 8(DISK)] > The are two index 0 internal block and one of them should be chosen to > delete.But the current implement chooses the index 0 internal blocks as > candidates but DISK as exess storage type.As a result, the exess storage > type(DISK) can not correspond to the exess internal blocks' storage type(SSD) > correctly, and the exess internal block can not be deleted correctly. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17556) Avoid adding block to neededReconstruction repeatedly in decommission
[
https://issues.apache.org/jira/browse/HDFS-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880927#comment-17880927
]
ASF GitHub Bot commented on HDFS-17556:
---
lfxy commented on PR #6896:
URL: https://github.com/apache/hadoop/pull/6896#issuecomment-2343274503
@Hexiaoqiao Could you help to merge this commit?
> Avoid adding block to neededReconstruction repeatedly in decommission
> -
>
> Key: HDFS-17556
> URL: https://issues.apache.org/jira/browse/HDFS-17556
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.5.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Major
> Labels: pull-request-available
>
> In decommission and maintenance process, before added to
> BlockManager::neededReconstruction block will be check if it has been added.
> The check contains if block is in BlockManager::neededReconstruction or in
> PendingReconstructionBlocks::pendingReconstructions as below code.
> But it also need to check if it is in
> PendingReconstructionBlocks::timedOutItems. Or else
> DatanodeAdminDefaultMonitor will add block to
> BlockManager::neededReconstruction repeatedly if block time out in
> PendingReconstructionBlocks::pendingReconstructions.
>
> {code:java}
> if (!blockManager.neededReconstruction.contains(block) &&
> blockManager.pendingReconstruction.getNumReplicas(block) == 0 &&
> blockManager.isPopulatingReplQueues()) {
> // Process these blocks only when active NN is out of safe mode.
> blockManager.neededReconstruction.add(block,
> liveReplicas, num.readOnlyReplicas(),
> num.outOfServiceReplicas(),
> blockManager.getExpectedRedundancyNum(block));
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17545) [ARR] router async rpc client.
[
https://issues.apache.org/jira/browse/HDFS-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880868#comment-17880868
]
ASF GitHub Bot commented on HDFS-17545:
---
KeeProMise commented on PR #6871:
URL: https://github.com/apache/hadoop/pull/6871#issuecomment-2342733716
Hi, @goiri @simbadzina @Hexiaoqiao @sjlee @ayushtkn @haiyang1987 @ZanderXu
the PR has been blocked for 2 months, and if everyone has time, please help to
review it because several subtask PRs need to depend on this PR. This PR does
not modify the existing synchronous router logic. To facilitate review, my
**main modifications** are as follows:
- RouterRpcClient.java: The original functionality and logic have not been
changed; I only have extracted some common methods.
- RouterAsyncRpcClient.java: An asynchronous implementation of
RouterRpcClient.
- Added configuration for the asynchronous feature toggle, as well as the
number of asynchronous handlers and responders.
- Using ThreadLocalContext to maintain thread local variables, ensuring that
thread local variables can be correctly passed between handler,
asyncRouterHandler, and asyncRouterResponder.
The PRs that depend on this PR are:
https://github.com/apache/hadoop/pull/6994
https://github.com/apache/hadoop/pull/6988
https://github.com/apache/hadoop/pull/6986
https://github.com/apache/hadoop/pull/6983
> [ARR] router async rpc client.
> --
>
> Key: HDFS-17545
> URL: https://issues.apache.org/jira/browse/HDFS-17545
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Labels: pull-request-available
>
> *Describe*
> 1. Mainly using AsyncUtil to implement {*}RouterAsyncRpcClient{*}, this class
> extends RouterRpcClient, enabling the {*}invoiceAll{*}, {*}invoiceMethod{*},
> {*}invoiceSequential{*}, {*}invoiceConcurrent{*}, and *invoiceSingle* methods
> to support asynchrony.
> 2. Use two thread pools, *asyncRouterHandler* and {*}asyncRouterResponder{*},
> to handle asynchronous requests and responses, respectively.
> 3. Added {*}DFS_ROUTER_RPC_ENABLE_ASYNC{*},
> {*}DFS_ROUTER_RPC_ASYNC_HANDLER_COUNT{*},
> *DFS_ROUTER_RPC_ASYNC_RESPONDER_COUNT_DEFAULT* to configure whether to use
> async router, as well as the number of asyncRouterHandlers and
> asyncRouterResponders.
> 4. Using *ThreadLocalContext* to maintain thread local variables, ensuring
> that thread local variables can be correctly passed between handler,
> asyncRouterHandler, and asyncRouterResponder.
>
> *Test*
> new UT TestRouterAsyncRpcClient
> Note: For discussions on *AsyncUtil* and client {*}protocolPB{*}, please
> refer to HDFS-17543 and HDFS-17544.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[
https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880792#comment-17880792
]
ASF GitHub Bot commented on HDFS-17381:
---
hadoop-yetus commented on PR #6551:
URL: https://github.com/apache/hadoop/pull/6551#issuecomment-2342056697
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 26s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 20s | | trunk passed |
| +1 :green_heart: | compile | 9m 1s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 8m 34s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 2m 5s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 13s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 56s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 41s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 43s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 22s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 8m 33s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 8m 33s | | the patch passed |
| +1 :green_heart: | compile | 8m 25s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 8m 25s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 5s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/12/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 8 new + 142 unchanged - 0 fixed = 150 total (was
142) |
| +1 :green_heart: | mvnsite | 2m 16s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 51s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 43s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 4m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 5s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 13s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 3s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 24m 24s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 41s | | The patch does not
generate ASF License warnings. |
| | | 183m 39s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/12/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6551 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux ad0713a08e22 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 3c0b1fb6757a53640167af52c3826258bd7cf21b |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-m
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[
https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880684#comment-17880684
]
ASF GitHub Bot commented on HDFS-17381:
---
hadoop-yetus commented on PR #6551:
URL: https://github.com/apache/hadoop/pull/6551#issuecomment-2341197524
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 1m 15s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 10s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 15s | | trunk passed |
| +1 :green_heart: | compile | 10m 29s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 9m 45s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 2m 18s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 55s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 33s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 22s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 42s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 23s | | the patch passed |
| +1 :green_heart: | compile | 9m 54s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 9m 54s | | the patch passed |
| +1 :green_heart: | compile | 9m 57s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 9m 57s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 11s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/10/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 9 new + 142 unchanged - 0 fixed = 151 total (was
142) |
| +1 :green_heart: | mvnsite | 2m 6s | | the patch passed |
| -1 :x: | javadoc | 0m 37s |
[/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/10/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
|
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 generated 4 new + 0
unchanged - 0 fixed = 4 total (was 0) |
| +1 :green_heart: | javadoc | 1m 20s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 4m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 43s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 22s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 6s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 24m 31s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 190m 35s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/10/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6551 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 202a1bb972c8 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-17609) [FGL] Fix lock mode in some RPC
[
https://issues.apache.org/jira/browse/HDFS-17609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880581#comment-17880581
]
ASF GitHub Bot commented on HDFS-17609:
---
hadoop-yetus commented on PR #7037:
URL: https://github.com/apache/hadoop/pull/7037#issuecomment-2340247121
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 32s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ HDFS-17384 Compile Tests _ |
| -1 :x: | mvninstall | 5m 54s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/3/artifact/out/branch-mvninstall-root.txt)
| root in HDFS-17384 failed. |
| -1 :x: | compile | 1m 34s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/3/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
| hadoop-hdfs in HDFS-17384 failed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | compile | 0m 41s |
[/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/3/artifact/out/branch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt)
| hadoop-hdfs in HDFS-17384 failed with JDK Private
Build-1.8.0_422-8u422-b05-1~20.04-b05. |
| +1 :green_heart: | checkstyle | 1m 6s | | HDFS-17384 passed |
| -1 :x: | mvnsite | 0m 47s |
[/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/3/artifact/out/branch-mvnsite-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in HDFS-17384 failed. |
| -1 :x: | javadoc | 0m 44s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/3/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
| hadoop-hdfs in HDFS-17384 failed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | javadoc | 0m 35s |
[/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/3/artifact/out/branch-javadoc-hadoop-hdfs-project_hadoop-hdfs-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt)
| hadoop-hdfs in HDFS-17384 failed with JDK Private
Build-1.8.0_422-8u422-b05-1~20.04-b05. |
| -1 :x: | spotbugs | 0m 43s |
[/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/3/artifact/out/branch-spotbugs-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in HDFS-17384 failed. |
| -1 :x: | shadedclient | 4m 55s | | branch has errors when building
and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 41s |
[/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/3/artifact/out/patch-mvninstall-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch failed. |
| -1 :x: | compile | 0m 44s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/3/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | javac | 0m 44s |
[/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7037/3/artifact/out/patch-compile-hadoop-hdfs-project_hadoop-hdfs-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
| hadoop-hdfs in the patch failed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | compile | 0m 42s |
[/patch-c
[jira] [Commented] (HDFS-17609) [FGL] Fix lock mode in some RPC
[
https://issues.apache.org/jira/browse/HDFS-17609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880560#comment-17880560
]
ASF GitHub Bot commented on HDFS-17609:
---
hfutatzhanghb commented on PR #7037:
URL: https://github.com/apache/hadoop/pull/7037#issuecomment-2340112640
@ZanderXu @ferhui Sir, PTAL if you have free time~ Thanks a lot.
> [FGL] Fix lock mode in some RPC
> ---
>
> Key: HDFS-17609
> URL: https://issues.apache.org/jira/browse/HDFS-17609
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> * FSNamesystem#getFilesBlockingDecom ： use GLOBAL mode rather than FS mode
> because of below codes:
> ```java
> for (DatanodeDescriptor dataNode :
> blockManager.getDatanodeManager().getDatanodes()) {
> }
> ```
>
> * FSNamesystem#listOpenFiles: use GLOBAL mode because it calls
> getFilesBlockingDecom method.
> * FSNamesystem#getContentSummary： use GLOBAL mode rather than FS mode
> because it calls computeFileSize method.
> * BlockManagerSafeMode#leaveSafeMode： use GLOBAL mode rather than BM mode
> because it calls startSecretManagerIfNecessary which depended upon FS lock.
> Change to GLOBAL mode is safe here.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17609) [FGL] Fix lock mode in some RPC
[
https://issues.apache.org/jira/browse/HDFS-17609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDFS-17609:
--
Labels: pull-request-available (was: )
> [FGL] Fix lock mode in some RPC
> ---
>
> Key: HDFS-17609
> URL: https://issues.apache.org/jira/browse/HDFS-17609
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> * FSNamesystem#getFilesBlockingDecom ： use GLOBAL mode rather than FS mode
> because of below codes:
> ```java
> for (DatanodeDescriptor dataNode :
> blockManager.getDatanodeManager().getDatanodes()) {
> }
> ```
>
> * FSNamesystem#listOpenFiles: use GLOBAL mode because it calls
> getFilesBlockingDecom method.
> * FSNamesystem#getContentSummary： use GLOBAL mode rather than FS mode
> because it calls computeFileSize method.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17609) [FGL] Fix lock mode in some RPC
[
https://issues.apache.org/jira/browse/HDFS-17609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880555#comment-17880555
]
ASF GitHub Bot commented on HDFS-17609:
---
hfutatzhanghb opened a new pull request, #7037:
URL: https://github.com/apache/hadoop/pull/7037
### Description of PR
Fix lock mode in some RPCs and add some assert statement.
> [FGL] Fix lock mode in some RPC
> ---
>
> Key: HDFS-17609
> URL: https://issues.apache.org/jira/browse/HDFS-17609
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
>
> * FSNamesystem#getFilesBlockingDecom ： use GLOBAL mode rather than FS mode
> because of below codes:
> ```java
> for (DatanodeDescriptor dataNode :
> blockManager.getDatanodeManager().getDatanodes()) {
> }
> ```
>
> * FSNamesystem#listOpenFiles: use GLOBAL mode because it calls
> getFilesBlockingDecom method.
> * FSNamesystem#getContentSummary： use GLOBAL mode rather than FS mode
> because it calls computeFileSize method.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16757) Add a new method copyBlockCrossNamespace to DataNode
[
https://issues.apache.org/jira/browse/HDFS-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880542#comment-17880542
]
ASF GitHub Bot commented on HDFS-16757:
---
LiuGuH commented on code in PR #6926:
URL: https://github.com/apache/hadoop/pull/6926#discussion_r1751479459
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:
##
@@ -3863,5 +3864,27 @@ public void setLastDirScannerFinishTime(long time) {
public long getPendingAsyncDeletions() {
return asyncDiskService.countPendingDeletions();
}
+
+ @Override
+ public void hardLinkOneBlock(ExtendedBlock srcBlock, ExtendedBlock dstBlock)
throws IOException {
+BlockLocalPathInfo blpi = getBlockLocalPathInfo(srcBlock);
+FsVolumeImpl v = getVolume(srcBlock);
+
+try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.VOLUME,
dstBlock.getBlockPoolId(),
+v.getStorageID())) {
+ File src = new File(blpi.getBlockPath());
+ File srcMeta = new File(blpi.getMetaPath());
+ BlockPoolSlice dstBPS = v.getBlockPoolSlice(dstBlock.getBlockPoolId());
+ File dstBlockFile = dstBPS.hardLinkOneBlock(src, srcMeta,
dstBlock.getLocalBlock());
+
+ ReplicaInfo replicaInfo =
+ new LocalReplicaInPipeline(dstBlock.getBlockId(),
dstBlock.getGenerationStamp(), v,
+ dstBlockFile.getParentFile(),
dstBlock.getLocalBlock().getNumBytes());
+ dstBlockFile = dstBPS.addFinalizedBlock(dstBlock.getLocalBlock(),
replicaInfo);
+ replicaInfo = new FinalizedReplica(dstBlock.getLocalBlock(),
getVolume(srcBlock),
+ dstBlockFile.getParentFile());
+ volumeMap.add(dstBlock.getBlockPoolId(), replicaInfo);
Review Comment:
dst volume write lock protects hardlink file atomicity and volumeMap.add().
The best case scenario we should use src volume read lock and dst volume write
lock to protect src and dst file operation.
And in DataXceiver, writeBlock() readBlock() transferBlock() , is as same
as this only require lock with block meta operation.
So, I think this is enough for now. @Hexiaoqiao he @KeeProMise
> Add a new method copyBlockCrossNamespace to DataNode
>
>
> Key: HDFS-16757
> URL: https://issues.apache.org/jira/browse/HDFS-16757
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: ZanderXu
>Assignee: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> Add a new method copyBlockCrossNamespace in DataTransferProtocol at the
> DataNode Side.
> This method will copy a source block from one namespace to a target block
> from a different namespace. If the target DN is the same with the current DN,
> this method will copy the block via HardLink. If the target DN is different
> with the current DN, this method will copy the block via TransferBlock.
> This method will contains some parameters:
> * ExtendedBlock sourceBlock
> * Token sourceBlockToken
> * ExtendedBlock targetBlock
> * Token targetBlockToken
> * DatanodeInfo targetDN
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17620) Better block placement for small EC files
[
https://issues.apache.org/jira/browse/HDFS-17620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880475#comment-17880475
]
ASF GitHub Bot commented on HDFS-17620:
---
junegunn opened a new pull request, #7035:
URL: https://github.com/apache/hadoop/pull/7035
### Description of PR
Fixes block placement for small EC files. Detailed description is in the
JIRA issue.
### How was this patch tested?
Tested on our production HBase cluster running Hadoop 3.3.6.
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Better block placement for small EC files
> -
>
> Key: HDFS-17620
> URL: https://issues.apache.org/jira/browse/HDFS-17620
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: erasure-coding, namenode
>Affects Versions: 3.3.6
>Reporter: Junegunn Choi
>Priority: Major
>
> h2. Problem description
> If an erasure-coded file is not large enough to fill the stripe width of the
> EC policy, the block distribution can be suboptimal.
> For example, an RS-6-3-1024K EC file smaller than 1024K will have 1 data
> block and 3 parity blocks. While all 9 (6 + 3) storage locations are chosen
> by the block placement policy, only 4 of them are used, and the last 3
> locations are for parity blocks. If the cluster has a very small number of
> racks (e.g. 3), with the current scheme to find a pipeline with the shortest
> path, the last nodes are likely to be in the same rack, resulting in a
> suboptimal rack distribution.
> {noformat}
> Locations: N1 N2 N3 N4 N5 N6 N7 N8 N9
> Racks: R1 R1 R1 R2 R2 R2 R3 R3 R3
>Blocks: D1P1 P2 P3
> {noformat}
> We can see that blocks are stored in only 2 racks, not 3.
> Because the block does not have enough racks, {{ErasureCodingWork}} will
> later be created to replicate the block to a new rack, however, the current
> code tries to copy the block to the first node in the chosen locations,
> regardless of its rack. So it is not guaranteed to improve the situation, and
> we constantly see {{PendingReconstructionMonitor timed out}} messages in the
> log.
> h2. Proposed solution
> 1. Reorder the chosen locations by rack so that the parity blocks are stored
> in as many racks as possible.
> 2. Make {{ErasureCodingWork}} try to find a target on a new rack
> h2. Real-world test result
> We first noticed the problem on our HBase cluster running Hadoop 3.3.6 on 18
> nodes across 3 racks. After setting RS-6-3-1024K policy on the HBase data
> directory, we noticed that
> 1. FSCK reports "Unsatisfactory placement block groups" for small EC files.
> {noformat}
> /hbase/***: Replica placement policy is violated for ***. Block should be
> additionally replicated on 2 more rack(s). Total number of racks in the
> cluster: 3
> ...
> Erasure Coded Block Groups:
> ...
> Unsatisfactory placement block groups: 1475 (2.5252092 %)
> {noformat}
> 2. Namenode keeps logging "PendingReconstructionMonitor timed out" messages
> every recheck-interval (5 minutes).
> 3. and {{FSNamesystem.UnderReplicatedBlocks}} metric bumps and clears every
> recheck-interval.
> After applying the patch, all the problems are gone. "Unsatisfactory
> placement block groups" is now zero. No metrics bumps or "timed out" logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17531) RBF: Asynchronous router RPC
[
https://issues.apache.org/jira/browse/HDFS-17531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880388#comment-17880388
]
ASF GitHub Bot commented on HDFS-17531:
---
KeeProMise closed pull request #7034: [ARR] Rebase the HDFS-17531 branch onto
the trunk branch.
URL: https://github.com/apache/hadoop/pull/7034
> RBF: Asynchronous router RPC
>
>
> Key: HDFS-17531
> URL: https://issues.apache.org/jira/browse/HDFS-17531
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: rbf
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: Async router single ns performance test.pdf, Aynchronous
> router.pdf, Comparison of Async router & sync router performance.pdf,
> HDFS-17531.001.patch, Router asynchronous rpc implementation .pdf,
> image-2024-05-19-18-07-51-282.png
>
>
> *Description*
> Currently, the main function of the Router service is to accept client
> requests, forward the requests to the corresponding downstream ns, and then
> return the results of the downstream ns to the client. The link is as follows:
> *!image-2024-05-19-18-07-51-282.png|width=900,height=300!*
> The main threads involved in the rpc link are:
> {*}Read{*}: Get the client request and put it into the call queue *(1)*
> {*}Handler{*}:
> Extract call *(2)* from the call queue, process the call, generate a new
> call, place it in the call of the connection thread, and wait for the call
> processing to complete *(3)*
> After being awakened by the connection thread, process the response and put
> it into the response queue *(5)*
> *Connection:*
> Hold the link with downstream ns, send the call from the call to the
> downstream ns (via {*}rpcRequestThread{*}), and obtain a response from ns.
> Based on the call in the response, notify the call to complete processing
> *(4)*
> *Responder:*
> Retrieve the response queue from the queue *(6)* and return it to the client
>
> *Shortcoming*
> Even if the *connection* thread can send more requests to downstream
> nameservices, since *(3)* and *(4)* are synchronous, when the *handler*
> thread adds the call to connection.calls, it needs to wait until the
> *connection* notifies the call to complete, and then Only after the response
> is put into the response queue can a new call be obtained from the call queue
> and processed. Therefore, the concurrency performance of the router is
> limited by the number of handlers; a simple example is as follows: If the
> number of handlers is 1 and the maximum number of calls in the connection
> thread is 10, then even if the connection thread can send 10 requests to the
> downstream ns, since the number of handlers is 1, the router can only process
> one request after another.
>
> Since the performance of router rpc is mainly limited by the number of
> handlers, the most effective way to improve rpc performance currently is to
> increase the number of handlers. Letting the router create a large number of
> handler threads will also increase the number of thread switches and cannot
> maximize the use of machine performance.
>
> There are usually multiple ns downstream of the router. If the handler
> forwards the request to an ns with poor performance, it will cause the
> handler to wait for a long time. Due to the reduction of available handlers,
> the router's ability to handle ns requests with normal performance will be
> reduced. From the perspective of the client, the performance of the
> downstream ns of the router has deteriorated at this time. We often find that
> the call queue of the downstream ns is not high, but the call queue of the
> router is very high.
>
> Therefore, although the main function of the router is to federate and handle
> requests from multiple NSs, the current synchronous RPC performance cannot
> satisfy the scenario where there are many NSs downstream of the router. Even
> if the concurrent performance of the router can be improved by increasing the
> number of handlers, it is still relatively slow. More threads will increase
> the CPU context switching time, and in fact many of the handler threads are
> in a blocked state, which is undoubtedly a waste of thread resources. When a
> request enters the router, there is no guarantee that there will be a running
> handler at this time.
>
> Therefore, I consider asynchronous router rpc. Please view the *pdf* for the
> complete solution.
>
> Welcome everyone to exchange and discuss!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apach
[jira] [Commented] (HDFS-17531) RBF: Asynchronous router RPC
[
https://issues.apache.org/jira/browse/HDFS-17531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880387#comment-17880387
]
ASF GitHub Bot commented on HDFS-17531:
---
hadoop-yetus commented on PR #7034:
URL: https://github.com/apache/hadoop/pull/7034#issuecomment-2338560952
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 0s | | Docker mode activated. |
| -1 :x: | patch | 0m 21s | |
https://github.com/apache/hadoop/pull/7034 does not apply to HDFS-17531. Rebase
required? Wrong Branch? See
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute for help.
|
| Subsystem | Report/Notes |
|--:|:-|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7034/1/console |
| versions | git=2.34.1 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> RBF: Asynchronous router RPC
>
>
> Key: HDFS-17531
> URL: https://issues.apache.org/jira/browse/HDFS-17531
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: rbf
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: Async router single ns performance test.pdf, Aynchronous
> router.pdf, Comparison of Async router & sync router performance.pdf,
> HDFS-17531.001.patch, Router asynchronous rpc implementation .pdf,
> image-2024-05-19-18-07-51-282.png
>
>
> *Description*
> Currently, the main function of the Router service is to accept client
> requests, forward the requests to the corresponding downstream ns, and then
> return the results of the downstream ns to the client. The link is as follows:
> *!image-2024-05-19-18-07-51-282.png|width=900,height=300!*
> The main threads involved in the rpc link are:
> {*}Read{*}: Get the client request and put it into the call queue *(1)*
> {*}Handler{*}:
> Extract call *(2)* from the call queue, process the call, generate a new
> call, place it in the call of the connection thread, and wait for the call
> processing to complete *(3)*
> After being awakened by the connection thread, process the response and put
> it into the response queue *(5)*
> *Connection:*
> Hold the link with downstream ns, send the call from the call to the
> downstream ns (via {*}rpcRequestThread{*}), and obtain a response from ns.
> Based on the call in the response, notify the call to complete processing
> *(4)*
> *Responder:*
> Retrieve the response queue from the queue *(6)* and return it to the client
>
> *Shortcoming*
> Even if the *connection* thread can send more requests to downstream
> nameservices, since *(3)* and *(4)* are synchronous, when the *handler*
> thread adds the call to connection.calls, it needs to wait until the
> *connection* notifies the call to complete, and then Only after the response
> is put into the response queue can a new call be obtained from the call queue
> and processed. Therefore, the concurrency performance of the router is
> limited by the number of handlers; a simple example is as follows: If the
> number of handlers is 1 and the maximum number of calls in the connection
> thread is 10, then even if the connection thread can send 10 requests to the
> downstream ns, since the number of handlers is 1, the router can only process
> one request after another.
>
> Since the performance of router rpc is mainly limited by the number of
> handlers, the most effective way to improve rpc performance currently is to
> increase the number of handlers. Letting the router create a large number of
> handler threads will also increase the number of thread switches and cannot
> maximize the use of machine performance.
>
> There are usually multiple ns downstream of the router. If the handler
> forwards the request to an ns with poor performance, it will cause the
> handler to wait for a long time. Due to the reduction of available handlers,
> the router's ability to handle ns requests with normal performance will be
> reduced. From the perspective of the client, the performance of the
> downstream ns of the router has deteriorated at this time. We often find that
> the call queue of the downstream ns is not high, but the call queue of the
> router is very high.
>
> Therefore, although the main function of the router is to federate and handle
> requests from multiple NSs, the current synchronous RPC performance cannot
> satisfy the scenario where there are many NSs downstream of the router. Even
> if the concurrent performance of the router can be improved by increasing the
>
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[
https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880364#comment-17880364
]
ASF GitHub Bot commented on HDFS-17381:
---
steveloughran commented on code in PR #6551:
URL: https://github.com/apache/hadoop/pull/6551#discussion_r1750385001
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/WithErasureCoding.java:
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs;
+
+import java.io.IOException;
+
+/**
+ * Filesystems that support EC can implement this interface.
+ */
+public interface WithErasureCoding {
+
+ /**
+ * Get the EC Policy name of the given file's fileStatus.
+ * If the file is not erasure coded, this should return null.
Review Comment:
Use RFC2119 terminology. Here: SHALL
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/Options.java:
##
@@ -641,5 +641,10 @@ private OpenFileOptions() {
FS_OPTION_OPENFILE_READ_POLICY_WHOLE_FILE)
.collect(Collectors.toSet()));
+/**
+ * EC policy to be set on the file that needs to be created.
Review Comment:
add an {@value} tag for IDE/javadoc preview of value
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/WithErasureCoding.java:
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs;
+
+import java.io.IOException;
+
+/**
+ * Filesystems that support EC can implement this interface.
+ */
+public interface WithErasureCoding {
+
+ /**
+ * Get the EC Policy name of the given file's fileStatus.
+ * If the file is not erasure coded, this should return null.
+ * Callers will make sure to check if the FS schema of the fileStatus
Review Comment:
hard to parse this. What are callers meant to do? Filesystem schema (hdfs,
o3, ...) can be changed. Do you mean use `instanceof` to probe a filesystem
instance to see if it implementas.
##
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableDirectoryCreateCommand.java:
##
@@ -18,15 +18,20 @@
package org.apache.hadoop.tools.mapred;
+import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.hdfs.DistributedFileSystem;
+import org.apache.hadoop.fs.Options;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.WithErasureCoding;
import org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy;
-import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
+import org.apache.hadoop.hdfs.protocol.SystemErasureCodingPolicies;
import org.apache.hadoop.tools.DistCpOptions;
import org.apache.hadoop.tools.util.RetriableCommand;
-import org.apache.hadoop.fs.Path;
-import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.mapreduce.Mapper;
+import org.slf4j.Logger;
Review Comment:
put in a separate import block
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FileStatus.java:
##
@@ -351,7 +352,12 @@ public String getGroup() {
public Path getPath() {
return path;
}
-
+
+ @VisibleForTesting
Review Comment:
no. commented on the test.
##
hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/TestDistCpWithRawXAttrs.jav
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[ https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880363#comment-17880363 ] ASF GitHub Bot commented on HDFS-17381: --- sadanand48 commented on PR #6551: URL: https://github.com/apache/hadoop/pull/6551#issuecomment-2338339586 > +can i encourage you to use git merge to resync with with trunk once people have started reviewing? its a lot easier to review with the "changes since your last review" option. thanks Sure , I will use git merge instead of force push next time. I have still maintained the commit history here even with force push. For "changes since your last review" please refer to this [commit ](https://github.com/apache/hadoop/pull/6551/commits/292e110f05630c6df1e68912911a44e85f8300ef) > Distcp of EC files should not be limited to DFS. > > > Key: HDFS-17381 > URL: https://issues.apache.org/jira/browse/HDFS-17381 > Project: Hadoop HDFS > Issue Type: Bug > Components: distcp >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > Labels: pull-request-available > > Currently EC file support in distcp is limited to DFS and the code checks if > the given FS instance is DistributedFileSystem, In Ozone, EC is supported > now, and this limitation can be removed and a general contract for any > filesystem that supports EC files should be allowed by implementing few > interfaces/methods. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[ https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880357#comment-17880357 ] ASF GitHub Bot commented on HDFS-17381: --- steveloughran commented on PR #6551: URL: https://github.com/apache/hadoop/pull/6551#issuecomment-2338279078 +can i encourage you to use git merge to resync with with trunk once people have started reviewing? its a lot easier to review with the "changes since your last review" option. thanks > Distcp of EC files should not be limited to DFS. > > > Key: HDFS-17381 > URL: https://issues.apache.org/jira/browse/HDFS-17381 > Project: Hadoop HDFS > Issue Type: Bug > Components: distcp >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > Labels: pull-request-available > > Currently EC file support in distcp is limited to DFS and the code checks if > the given FS instance is DistributedFileSystem, In Ozone, EC is supported > now, and this limitation can be removed and a general contract for any > filesystem that supports EC files should be allowed by implementing few > interfaces/methods. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[ https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880356#comment-17880356 ] ASF GitHub Bot commented on HDFS-17381: --- steveloughran commented on PR #6551: URL: https://github.com/apache/hadoop/pull/6551#issuecomment-2338275838 > I don't see many usages of the createFile() api that returns the builder object though and I see this jira ([HADOOP-14384](https://issues.apache.org/jira/browse/HADOOP-14384)) that reduces usage of this API, I hope we can use it now. that was before hadoop 3.0 shipped. it is now stable and while not used much in those libraries that still want to run on on java2 -and because they're confident we aren't going to cut the old ones. The HDFS subclass provides many more options; S3A supports extra options such as "fs.s3a.create.performance" which turns off the safety check of a LIST for child objects. We've just been working hard to make those same libraries able to use the builder based openFile() API as it has tangible performance benefits. > Distcp of EC files should not be limited to DFS. > > > Key: HDFS-17381 > URL: https://issues.apache.org/jira/browse/HDFS-17381 > Project: Hadoop HDFS > Issue Type: Bug > Components: distcp >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > Labels: pull-request-available > > Currently EC file support in distcp is limited to DFS and the code checks if > the given FS instance is DistributedFileSystem, In Ozone, EC is supported > now, and this limitation can be removed and a general contract for any > filesystem that supports EC files should be allowed by implementing few > interfaces/methods. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16757) Add a new method copyBlockCrossNamespace to DataNode
[
https://issues.apache.org/jira/browse/HDFS-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880232#comment-17880232
]
ASF GitHub Bot commented on HDFS-16757:
---
KeeProMise commented on code in PR #6926:
URL: https://github.com/apache/hadoop/pull/6926#discussion_r1749875879
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:
##
@@ -4388,4 +4409,94 @@ boolean isSlownode() {
public BlockPoolManager getBlockPoolManager() {
return blockPoolManager;
}
+
+ public void copyBlockCrossNamespace(ExtendedBlock sourceBlk, ExtendedBlock
targetBlk,
+ DatanodeInfo targetDn) throws IOException {
+if (!data.isValidBlock(sourceBlk)) {
+ // block does not exist or is under-construction
+ String errStr =
+ "copyBlock:(" + this.getInfoPort() + ") Can't send invalid block " +
sourceBlk + " "
+ + data.getReplicaString(sourceBlk.getBlockPoolId(),
sourceBlk.getBlockId());
+ LOG.info(errStr);
+ throw new IOException(errStr);
+}
+long onDiskLength = data.getLength(sourceBlk);
+if (sourceBlk.getNumBytes() > onDiskLength) {
+ // Shorter on-disk len indicates corruption so report NN the corrupt
block
+ String msg = "copyBlock: Can't replicate block " + sourceBlk + " because
on-disk length "
+ + onDiskLength + " is shorter than provided length " +
sourceBlk.getNumBytes();
+ LOG.info(msg);
+ throw new IOException(msg);
+}
+LOG.info(getDatanodeInfo() + " copyBlock: Starting thread to transfer: " +
"block:"
++ sourceBlk + " from " + this.getDatanodeUuid() + " to " +
targetDn.getDatanodeUuid()
++ "(" + targetDn + ")");
Review Comment:
Hi, @LiuGuH , can use log parameterized messages ("{} ", a) instead of
string concatenation (a + xxx).
> Add a new method copyBlockCrossNamespace to DataNode
>
>
> Key: HDFS-16757
> URL: https://issues.apache.org/jira/browse/HDFS-16757
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: ZanderXu
>Assignee: liuguanghua
>Priority: Minor
> Labels: pull-request-available
>
> Add a new method copyBlockCrossNamespace in DataTransferProtocol at the
> DataNode Side.
> This method will copy a source block from one namespace to a target block
> from a different namespace. If the target DN is the same with the current DN,
> this method will copy the block via HardLink. If the target DN is different
> with the current DN, this method will copy the block via TransferBlock.
> This method will contains some parameters:
> * ExtendedBlock sourceBlock
> * Token sourceBlockToken
> * ExtendedBlock targetBlock
> * Token targetBlockToken
> * DatanodeInfo targetDN
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17602) RBF mount point with SPACE order can not find the most available namespace, it will choose an irrelevant namespace
[
https://issues.apache.org/jira/browse/HDFS-17602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880188#comment-17880188
]
ASF GitHub Bot commented on HDFS-17602:
---
wzk784533 commented on code in PR #6991:
URL: https://github.com/apache/hadoop/pull/6991#discussion_r1749481916
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/order/AvailableSpaceResolver.java:
##
@@ -118,6 +121,20 @@ protected String chooseFirstNamespace(String path,
PathLocation loc) {
getSubclusterMapping();
List subclusterList = new LinkedList<>(
subclusterInfo.values());
+
+if (loc != null && loc.getDestinations() != null) {
+Set locSet = new HashSet();
+for (RemoteLocation dest : loc.getDestinations()) {
+locSet.add(dest.getNameserviceId());
+}
+List filteredSubclusterList = new
LinkedList<>();
+for (SubclusterAvailableSpace cluster : subclusterList) {
+if (locSet.contains(cluster.getNameserviceId())) {
+filteredSubclusterList.add(cluster);
+}
+}
+subclusterList = filteredSubclusterList;
Review Comment:
Ok, I will fix it later
> RBF mount point with SPACE order can not find the most available namespace,
> it will choose an irrelevant namespace
> --
>
> Key: HDFS-17602
> URL: https://issues.apache.org/jira/browse/HDFS-17602
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: router
>Reporter: Zhongkun Wu
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2024-08-12-10-08-54-031.png,
> image-2024-08-12-10-08-58-271.png, image-2024-08-12-10-12-48-428.png,
> image-2024-08-12-10-14-20-580.png, image-2024-08-12-10-25-26-003.png,
> image-2024-08-12-10-25-42-863.png
>
>
> !image-2024-08-12-10-08-58-271.png!
> In the
> MultipleDestinationMountTableResolver.java we invoke
> orderedResolver.getFirstNamespace(path, mountTableResult);
> It will then invoke this function in RouterResolver.java
> !image-2024-08-12-10-12-48-428.png!
> and now we are in
> chooseFirstNamespace function in AvailableSpaceResolver.java
> !image-2024-08-12-10-14-20-580.png!
>
> The path parameter is the destination where we want to create a file
> the loc parameter is the mount point we set
>
> this function will choose the most available namespace in all the namespace
> we have in StateStore, which is not the same as the mount point we set for
> our destination.
>
> As a result we will get a namespace irrelevant to the namespaces we set for
> the destination path
>
> !image-2024-08-12-10-25-42-863.png!
> in the log above:
> we get the namespace we don't set with our destination path, So the it will
> choose the first namespace it sees and it's not really the most available
> namespace among the namespaces we set for our destination
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16084) getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv() failed
[
https://issues.apache.org/jira/browse/HDFS-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880108#comment-17880108
]
ASF GitHub Bot commented on HDFS-16084:
---
Hexiaoqiao commented on PR #6969:
URL: https://github.com/apache/hadoop/pull/6969#issuecomment-2336603872
> > Not see any place marked 3.4.2, would you mind to confirm? Thanks.
>
> branch-3.4 is going to be where 3.4.2 is going to be forked from -if you
do a PR for that branch it'll be out on the relevant release
Hi @steveloughran , have cherrypick to branch-3.4, Please double check.
> getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv()
> failed
>
>
> Key: HDFS-16084
> URL: https://issues.apache.org/jira/browse/HDFS-16084
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.2.1
>Reporter: Antoine Pitrou
>Assignee: kevin cai
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0, 3.4.2
>
>
> First reported in ARROW-13011: when a libhdfs API call fails because
> CLASSPATH isn't set, calling the API a second time leads to a crash.
> *Backtrace*
> This was obtained from the ARROW-13011 reproducer:
> {code:java}
> #0 globalClassReference (className=className@entry=0x7f75883c13b0
> "org/apache/hadoop/conf/Configuration", env=env@entry=0x6c2f2f3a73666468,
> out=out@entry=0x7fffd86e3020) at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:279
> #1 0x7f75883b9511 in constructNewObjectOfClass
> (env=env@entry=0x6c2f2f3a73666468, out=out@entry=0x7fffd86e3148,
> className=className@entry=0x7f75883c13b0
> "org/apache/hadoop/conf/Configuration",
> ctorSignature=ctorSignature@entry=0x7f75883c1180 "()V")
> at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:212
> #2 0x7f75883bb6d0 in hdfsBuilderConnect (bld=0x5562e4bbb3e0) at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:700
> #3 0x7f758de31ef3 in arrow::io::internal::LibHdfsShim::BuilderConnect
> (this=0x7f758e768240 namespace)::libhdfs_shim>,
> bld=0x5562e4bbb3e0) at /arrow/cpp/src/arrow/io/hdfs_internal.cc:366
> #4 0x7f758de2d098 in
> arrow::io::HadoopFileSystem::HadoopFileSystemImpl::Connect
> (this=0x5562e4a9f750, config=0x5562e46edc30)
> at /arrow/cpp/src/arrow/io/hdfs.cc:372
> #5 0x7f758de2e646 in arrow::io::HadoopFileSystem::Connect
> (config=0x5562e46edc30, fs=0x5562e46edd08) at
> /arrow/cpp/src/arrow/io/hdfs.cc:590
> #6 0x7f758d532d2a in arrow::fs::HadoopFileSystem::Impl::Init
> (this=0x5562e46edc30) at /arrow/cpp/src/arrow/filesystem/hdfs.cc:59
> #7 0x7f758d536931 in arrow::fs::HadoopFileSystem::Make (options=...,
> io_context=...) at /arrow/cpp/src/arrow/filesystem/hdfs.cc:409
> #8 0x7f75885d7445 in
> __pyx_pf_7pyarrow_5_hdfs_16HadoopFileSystem___init__
> (__pyx_v_self=0x7f758871a970, __pyx_v_host=0x7f758871cc00, __pyx_v_port=8020,
> __pyx_v_user=0x5562e3af6d30 <_Py_NoneStruct>, __pyx_v_replication=3,
> __pyx_v_buffer_size=0, __pyx_v_default_block_size=0x5562e3af6d30
> <_Py_NoneStruct>,
> __pyx_v_kerb_ticket=0x5562e3af6d30 <_Py_NoneStruct>,
> __pyx_v_extra_conf=0x5562e3af6d30 <_Py_NoneStruct>) at _hdfs.cpp:4759
> #9 0x7f75885d4c88 in
> __pyx_pw_7pyarrow_5_hdfs_16HadoopFileSystem_1__init__
> (__pyx_v_self=0x7f758871a970, __pyx_args=0x7f75900bb048,
> __pyx_kwds=0x7f7590033a68)
> at _hdfs.cpp:4343
> #10 0x5562e38ca747 in type_call () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/typeobject.c:915
> #11 0x5562e39117a3 in _PyObject_FastCallDict (kwargs=,
> nargs=, args=,
> func=0x7f75885f1420 <__pyx_type_7pyarrow_5_hdfs_HadoopFileSystem>) at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/tupleobject.c:76
> #12 _PyObject_FastCallKeywords () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/abstract.c:2496
> #13 0x5562e39121d5 in call_function () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4875
> #14 0x5562e3973d68 in _PyEval_EvalFrameDefault () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:3351
> #15 0x5562e38b98f5 in PyEval_EvalFrameEx (throwflag=0, f=0x7f74c0664768)
> at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4166
> #16 _PyEval_EvalCodeWithName () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4166
> #17 0x5562e38bad79 in PyEval_EvalCodeEx (_co=,
> glo
[jira] [Commented] (HDFS-17567) Return value of method RouterRpcClient#invokeSequential is not accurate
[
https://issues.apache.org/jira/browse/HDFS-17567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17880052#comment-17880052
]
ASF GitHub Bot commented on HDFS-17567:
---
KeeProMise commented on code in PR #6923:
URL: https://github.com/apache/hadoop/pull/6923#discussion_r1748371835
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterRpcClient.java:
##
@@ -1175,6 +1177,7 @@ public RemoteResult
invokeSequential(
}
// Return the first result, whether it is the value or not
@SuppressWarnings("unchecked") T ret = (T) firstResult;
+@SuppressWarnings("unchecked") R loc = (R) firstLocation;
return new RemoteResult<>(locations.get(0), ret);
Review Comment:
Hi, @hfutatzhanghb According to your explanation, maybe this should be
changed to:
```java
@SuppressWarnings("unchecked") R loc = (R) firstLocation;
return new RemoteResult<>(loc, ret);
```
> Return value of method RouterRpcClient#invokeSequential is not accurate
> ---
>
> Key: HDFS-17567
> URL: https://issues.apache.org/jira/browse/HDFS-17567
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Affects Versions: 3.4.0
>Reporter: farmmamba
>Assignee: farmmamba
>Priority: Major
> Labels: pull-request-available
>
> Below code is the return value in method RouterRpcClient#invokeSequential.
>
> {code:java}
> // Return the first result, whether it is the value or not
> @SuppressWarnings("unchecked") T ret = (T) firstResult;
> return new RemoteResult<>(locations.get(0), ret); {code}
>
>
> `locations.get(0)` is not accurate, because it may not be the remote location
> where ret was returned.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17611) Move all DistCp execution logic to execute()
[
https://issues.apache.org/jira/browse/HDFS-17611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879237#comment-17879237
]
ASF GitHub Bot commented on HDFS-17611:
---
hadoop-yetus commented on PR #7025:
URL: https://github.com/apache/hadoop/pull/7025#issuecomment-2329085207
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 31s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 47m 1s | | trunk passed |
| +1 :green_heart: | compile | 0m 32s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 35s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 29s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 52s | | trunk passed |
| +1 :green_heart: | shadedclient | 34m 33s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| -1 :x: | mvninstall | 0m 21s |
[/patch-mvninstall-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7025/3/artifact/out/patch-mvninstall-hadoop-tools_hadoop-distcp.txt)
| hadoop-distcp in the patch failed. |
| -1 :x: | compile | 0m 23s |
[/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7025/3/artifact/out/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
| hadoop-distcp in the patch failed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | javac | 0m 23s |
[/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7025/3/artifact/out/patch-compile-hadoop-tools_hadoop-distcp-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
| hadoop-distcp in the patch failed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | compile | 0m 21s |
[/patch-compile-hadoop-tools_hadoop-distcp-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7025/3/artifact/out/patch-compile-hadoop-tools_hadoop-distcp-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt)
| hadoop-distcp in the patch failed with JDK Private
Build-1.8.0_422-8u422-b05-1~20.04-b05. |
| -1 :x: | javac | 0m 21s |
[/patch-compile-hadoop-tools_hadoop-distcp-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7025/3/artifact/out/patch-compile-hadoop-tools_hadoop-distcp-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt)
| hadoop-distcp in the patch failed with JDK Private
Build-1.8.0_422-8u422-b05-1~20.04-b05. |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 17s | | the patch passed |
| -1 :x: | mvnsite | 0m 22s |
[/patch-mvnsite-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7025/3/artifact/out/patch-mvnsite-hadoop-tools_hadoop-distcp.txt)
| hadoop-distcp in the patch failed. |
| -1 :x: | javadoc | 0m 22s |
[/patch-javadoc-hadoop-tools_hadoop-distcp-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7025/3/artifact/out/patch-javadoc-hadoop-tools_hadoop-distcp-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
| hadoop-distcp in the patch failed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| -1 :x: | javadoc | 0m 21s |
[/patch-javadoc-hadoop-tools_hadoop-distcp-jdkPrivateBuild-1.8.0_422-8u422-b05-1~20.04-b05.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7025/3/artifact/out/patch-javadoc-hadoop-tools_hadoop-distcp-jdkPrivateBuild-1.8.0_42
[jira] [Commented] (HDFS-16084) getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv() failed
[
https://issues.apache.org/jira/browse/HDFS-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879207#comment-17879207
]
ASF GitHub Bot commented on HDFS-16084:
---
steveloughran commented on PR #6969:
URL: https://github.com/apache/hadoop/pull/6969#issuecomment-2328828757
> Not see any place marked 3.4.2, would you mind to confirm? Thanks.
branch-3.4 is going to be where 3.4.2 is going to be forked from -if you do
a PR for that branch it'll be out on the relevant release
> getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv()
> failed
>
>
> Key: HDFS-16084
> URL: https://issues.apache.org/jira/browse/HDFS-16084
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.2.1
>Reporter: Antoine Pitrou
>Assignee: kevin cai
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> First reported in ARROW-13011: when a libhdfs API call fails because
> CLASSPATH isn't set, calling the API a second time leads to a crash.
> *Backtrace*
> This was obtained from the ARROW-13011 reproducer:
> {code:java}
> #0 globalClassReference (className=className@entry=0x7f75883c13b0
> "org/apache/hadoop/conf/Configuration", env=env@entry=0x6c2f2f3a73666468,
> out=out@entry=0x7fffd86e3020) at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:279
> #1 0x7f75883b9511 in constructNewObjectOfClass
> (env=env@entry=0x6c2f2f3a73666468, out=out@entry=0x7fffd86e3148,
> className=className@entry=0x7f75883c13b0
> "org/apache/hadoop/conf/Configuration",
> ctorSignature=ctorSignature@entry=0x7f75883c1180 "()V")
> at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:212
> #2 0x7f75883bb6d0 in hdfsBuilderConnect (bld=0x5562e4bbb3e0) at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:700
> #3 0x7f758de31ef3 in arrow::io::internal::LibHdfsShim::BuilderConnect
> (this=0x7f758e768240 namespace)::libhdfs_shim>,
> bld=0x5562e4bbb3e0) at /arrow/cpp/src/arrow/io/hdfs_internal.cc:366
> #4 0x7f758de2d098 in
> arrow::io::HadoopFileSystem::HadoopFileSystemImpl::Connect
> (this=0x5562e4a9f750, config=0x5562e46edc30)
> at /arrow/cpp/src/arrow/io/hdfs.cc:372
> #5 0x7f758de2e646 in arrow::io::HadoopFileSystem::Connect
> (config=0x5562e46edc30, fs=0x5562e46edd08) at
> /arrow/cpp/src/arrow/io/hdfs.cc:590
> #6 0x7f758d532d2a in arrow::fs::HadoopFileSystem::Impl::Init
> (this=0x5562e46edc30) at /arrow/cpp/src/arrow/filesystem/hdfs.cc:59
> #7 0x7f758d536931 in arrow::fs::HadoopFileSystem::Make (options=...,
> io_context=...) at /arrow/cpp/src/arrow/filesystem/hdfs.cc:409
> #8 0x7f75885d7445 in
> __pyx_pf_7pyarrow_5_hdfs_16HadoopFileSystem___init__
> (__pyx_v_self=0x7f758871a970, __pyx_v_host=0x7f758871cc00, __pyx_v_port=8020,
> __pyx_v_user=0x5562e3af6d30 <_Py_NoneStruct>, __pyx_v_replication=3,
> __pyx_v_buffer_size=0, __pyx_v_default_block_size=0x5562e3af6d30
> <_Py_NoneStruct>,
> __pyx_v_kerb_ticket=0x5562e3af6d30 <_Py_NoneStruct>,
> __pyx_v_extra_conf=0x5562e3af6d30 <_Py_NoneStruct>) at _hdfs.cpp:4759
> #9 0x7f75885d4c88 in
> __pyx_pw_7pyarrow_5_hdfs_16HadoopFileSystem_1__init__
> (__pyx_v_self=0x7f758871a970, __pyx_args=0x7f75900bb048,
> __pyx_kwds=0x7f7590033a68)
> at _hdfs.cpp:4343
> #10 0x5562e38ca747 in type_call () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/typeobject.c:915
> #11 0x5562e39117a3 in _PyObject_FastCallDict (kwargs=,
> nargs=, args=,
> func=0x7f75885f1420 <__pyx_type_7pyarrow_5_hdfs_HadoopFileSystem>) at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/tupleobject.c:76
> #12 _PyObject_FastCallKeywords () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/abstract.c:2496
> #13 0x5562e39121d5 in call_function () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4875
> #14 0x5562e3973d68 in _PyEval_EvalFrameDefault () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:3351
> #15 0x5562e38b98f5 in PyEval_EvalFrameEx (throwflag=0, f=0x7f74c0664768)
> at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4166
> #16 _PyEval_EvalCodeWithName () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4166
> #17 0x5562e38bad79 in PyEval_EvalCodeEx (_co=,
> globals=, locals=, args=,
> argcount=,
> kws=, kwcount=0, defs=0x0, defcount=0, kwdefs
[jira] [Commented] (HDFS-17611) Move all DistCp execution logic to execute()
[
https://issues.apache.org/jira/browse/HDFS-17611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879185#comment-17879185
]
ASF GitHub Bot commented on HDFS-17611:
---
steveloughran commented on code in PR #7025:
URL: https://github.com/apache/hadoop/pull/7025#discussion_r1743537144
##
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java:
##
@@ -438,6 +446,15 @@ protected DistCpContext getContext() {
return context;
}
+ /**
+ * Sets the current context.
+ *
+ * @param context context to be set to.
+ */
+ public void setContext(DistCpContext context) {
Review Comment:
is this for testing or part of the public API?
##
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java:
##
@@ -181,15 +180,24 @@ public int run(String[] argv) {
return DistCpConstants.SUCCESS;
}
+ public Job execute() throws Exception {
Review Comment:
add javadoc, emphasise it is the original entry point
##
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/DistCp.java:
##
@@ -139,10 +139,9 @@ public int run(String[] argv) {
}
try {
- context = new DistCpContext(OptionsParser.parse(argv));
- checkSplitLargeFile();
- setTargetPathExists();
- LOG.info("Input Options: " + context);
+ DistCpContext ctx = new DistCpContext(OptionsParser.parse(argv));
+ LOG.info("Input Options: " + ctx);
Review Comment:
move to using {} in log...the + is only there because it was originally
commons-logging
> Move all DistCp execution logic to execute()
>
>
> Key: HDFS-17611
> URL: https://issues.apache.org/jira/browse/HDFS-17611
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Felix N
>Assignee: Felix N
>Priority: Minor
> Labels: pull-request-available
>
> Many code flows create a DistCp instance and call the public method execute()
> to get the Job object for better control over the distcp job but some logics
> are only called by the run() method. Should move these lines over to
> execute().
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17611) Move all DistCp execution logic to execute()
[ https://issues.apache.org/jira/browse/HDFS-17611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879183#comment-17879183 ] ASF GitHub Bot commented on HDFS-17611: --- steveloughran commented on PR #7025: URL: https://github.com/apache/hadoop/pull/7025#issuecomment-2328568006 I was worried about the "Trivial code change" comment. never is, sadly. > Move all DistCp execution logic to execute() > > > Key: HDFS-17611 > URL: https://issues.apache.org/jira/browse/HDFS-17611 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Felix N >Assignee: Felix N >Priority: Minor > Labels: pull-request-available > > Many code flows create a DistCp instance and call the public method execute() > to get the Job object for better control over the distcp job but some logics > are only called by the run() method. Should move these lines over to > execute(). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16084) getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv() failed
[
https://issues.apache.org/jira/browse/HDFS-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17879065#comment-17879065
]
ASF GitHub Bot commented on HDFS-16084:
---
kevincai commented on PR #6969:
URL: https://github.com/apache/hadoop/pull/6969#issuecomment-2327683039
> > > Committed to trunk. Thanks @kevincai for your contributions!
> > > BTW, what's your JIRA account? When I want to assign
https://issues.apache.org/jira/browse/HDFS-16084 to you but I am not sure if
the reporter and you is the same one. Thanks!
> >
> >
> > I didn't apply one.
>
> What about apply one at JIRA side? @kevincai
Got the jira account created, please assign to me as `kevin.cai`
> getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv()
> failed
>
>
> Key: HDFS-16084
> URL: https://issues.apache.org/jira/browse/HDFS-16084
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.2.1
>Reporter: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> First reported in ARROW-13011: when a libhdfs API call fails because
> CLASSPATH isn't set, calling the API a second time leads to a crash.
> *Backtrace*
> This was obtained from the ARROW-13011 reproducer:
> {code:java}
> #0 globalClassReference (className=className@entry=0x7f75883c13b0
> "org/apache/hadoop/conf/Configuration", env=env@entry=0x6c2f2f3a73666468,
> out=out@entry=0x7fffd86e3020) at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:279
> #1 0x7f75883b9511 in constructNewObjectOfClass
> (env=env@entry=0x6c2f2f3a73666468, out=out@entry=0x7fffd86e3148,
> className=className@entry=0x7f75883c13b0
> "org/apache/hadoop/conf/Configuration",
> ctorSignature=ctorSignature@entry=0x7f75883c1180 "()V")
> at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:212
> #2 0x7f75883bb6d0 in hdfsBuilderConnect (bld=0x5562e4bbb3e0) at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:700
> #3 0x7f758de31ef3 in arrow::io::internal::LibHdfsShim::BuilderConnect
> (this=0x7f758e768240 namespace)::libhdfs_shim>,
> bld=0x5562e4bbb3e0) at /arrow/cpp/src/arrow/io/hdfs_internal.cc:366
> #4 0x7f758de2d098 in
> arrow::io::HadoopFileSystem::HadoopFileSystemImpl::Connect
> (this=0x5562e4a9f750, config=0x5562e46edc30)
> at /arrow/cpp/src/arrow/io/hdfs.cc:372
> #5 0x7f758de2e646 in arrow::io::HadoopFileSystem::Connect
> (config=0x5562e46edc30, fs=0x5562e46edd08) at
> /arrow/cpp/src/arrow/io/hdfs.cc:590
> #6 0x7f758d532d2a in arrow::fs::HadoopFileSystem::Impl::Init
> (this=0x5562e46edc30) at /arrow/cpp/src/arrow/filesystem/hdfs.cc:59
> #7 0x7f758d536931 in arrow::fs::HadoopFileSystem::Make (options=...,
> io_context=...) at /arrow/cpp/src/arrow/filesystem/hdfs.cc:409
> #8 0x7f75885d7445 in
> __pyx_pf_7pyarrow_5_hdfs_16HadoopFileSystem___init__
> (__pyx_v_self=0x7f758871a970, __pyx_v_host=0x7f758871cc00, __pyx_v_port=8020,
> __pyx_v_user=0x5562e3af6d30 <_Py_NoneStruct>, __pyx_v_replication=3,
> __pyx_v_buffer_size=0, __pyx_v_default_block_size=0x5562e3af6d30
> <_Py_NoneStruct>,
> __pyx_v_kerb_ticket=0x5562e3af6d30 <_Py_NoneStruct>,
> __pyx_v_extra_conf=0x5562e3af6d30 <_Py_NoneStruct>) at _hdfs.cpp:4759
> #9 0x7f75885d4c88 in
> __pyx_pw_7pyarrow_5_hdfs_16HadoopFileSystem_1__init__
> (__pyx_v_self=0x7f758871a970, __pyx_args=0x7f75900bb048,
> __pyx_kwds=0x7f7590033a68)
> at _hdfs.cpp:4343
> #10 0x5562e38ca747 in type_call () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/typeobject.c:915
> #11 0x5562e39117a3 in _PyObject_FastCallDict (kwargs=,
> nargs=, args=,
> func=0x7f75885f1420 <__pyx_type_7pyarrow_5_hdfs_HadoopFileSystem>) at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/tupleobject.c:76
> #12 _PyObject_FastCallKeywords () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/abstract.c:2496
> #13 0x5562e39121d5 in call_function () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4875
> #14 0x5562e3973d68 in _PyEval_EvalFrameDefault () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:3351
> #15 0x5562e38b98f5 in PyEval_EvalFrameEx (throwflag=0, f=0x7f74c0664768)
> at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4166
> #16 _PyEval_EvalCodeWithName () at
> /home/conda/feedstock_root/build_artifacts/python_16137113610
[jira] [Commented] (HDFS-17611) Move all DistCp execution logic to execute()
[
https://issues.apache.org/jira/browse/HDFS-17611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878792#comment-17878792
]
ASF GitHub Bot commented on HDFS-17611:
---
hadoop-yetus commented on PR #7025:
URL: https://github.com/apache/hadoop/pull/7025#issuecomment-2326075208
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 35s | | trunk passed |
| +1 :green_heart: | compile | 0m 33s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 30s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 35s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 34s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 30s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 51s | | trunk passed |
| +1 :green_heart: | shadedclient | 33m 49s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 21s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 0m 21s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 18s |
[/results-checkstyle-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7025/1/artifact/out/results-checkstyle-hadoop-tools_hadoop-distcp.txt)
| hadoop-tools/hadoop-distcp: The patch generated 1 new + 13 unchanged - 0
fixed = 14 total (was 13) |
| +1 :green_heart: | mvnsite | 0m 24s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 22s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 51s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 25m 59s |
[/patch-unit-hadoop-tools_hadoop-distcp.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7025/1/artifact/out/patch-unit-hadoop-tools_hadoop-distcp.txt)
| hadoop-distcp in the patch passed. |
| +1 :green_heart: | asflicense | 0m 39s | | The patch does not
generate ASF License warnings. |
| | | 148m 51s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.tools.contract.TestHDFSContractDistCp |
| | hadoop.tools.TestExternalCall |
| | hadoop.tools.contract.TestLocalContractDistCp |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7025/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/7025 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 95f198158801 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d6f8168717ddf5d7ce3b71dc290721449b58ed49 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
[jira] [Commented] (HDFS-17602) RBF mount point with SPACE order can not find the most available namespace, it will choose an irrelevant namespace
[ https://issues.apache.org/jira/browse/HDFS-17602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878788#comment-17878788 ] ASF GitHub Bot commented on HDFS-17602: --- KeeProMise commented on PR #6991: URL: https://github.com/apache/hadoop/pull/6991#issuecomment-2326053676 Hi, @wzk784533 For the failure of test4tests, you need to add a UT > RBF mount point with SPACE order can not find the most available namespace, > it will choose an irrelevant namespace > -- > > Key: HDFS-17602 > URL: https://issues.apache.org/jira/browse/HDFS-17602 > Project: Hadoop HDFS > Issue Type: Bug > Components: router >Reporter: Zhongkun Wu >Priority: Critical > Labels: pull-request-available > Attachments: image-2024-08-12-10-08-54-031.png, > image-2024-08-12-10-08-58-271.png, image-2024-08-12-10-12-48-428.png, > image-2024-08-12-10-14-20-580.png, image-2024-08-12-10-25-26-003.png, > image-2024-08-12-10-25-42-863.png > > > !image-2024-08-12-10-08-58-271.png! > In the > MultipleDestinationMountTableResolver.java we invoke > orderedResolver.getFirstNamespace(path, mountTableResult); > It will then invoke this function in RouterResolver.java > !image-2024-08-12-10-12-48-428.png! > and now we are in > chooseFirstNamespace function in AvailableSpaceResolver.java > !image-2024-08-12-10-14-20-580.png! > > The path parameter is the destination where we want to create a file > the loc parameter is the mount point we set > > this function will choose the most available namespace in all the namespace > we have in StateStore, which is not the same as the mount point we set for > our destination. > > As a result we will get a namespace irrelevant to the namespaces we set for > the destination path > > !image-2024-08-12-10-25-42-863.png! > in the log above: > we get the namespace we don't set with our destination path, So the it will > choose the first namespace it sees and it's not really the most available > namespace among the namespaces we set for our destination > > > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17602) RBF mount point with SPACE order can not find the most available namespace, it will choose an irrelevant namespace
[
https://issues.apache.org/jira/browse/HDFS-17602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878787#comment-17878787
]
ASF GitHub Bot commented on HDFS-17602:
---
KeeProMise commented on code in PR #6991:
URL: https://github.com/apache/hadoop/pull/6991#discussion_r1741742608
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/order/AvailableSpaceResolver.java:
##
@@ -118,6 +121,20 @@ protected String chooseFirstNamespace(String path,
PathLocation loc) {
getSubclusterMapping();
List subclusterList = new LinkedList<>(
subclusterInfo.values());
+
+if (loc != null && loc.getDestinations() != null) {
+Set locSet = new HashSet();
+for (RemoteLocation dest : loc.getDestinations()) {
+locSet.add(dest.getNameserviceId());
+}
+List filteredSubclusterList = new
LinkedList<>();
+for (SubclusterAvailableSpace cluster : subclusterList) {
+if (locSet.contains(cluster.getNameserviceId())) {
+filteredSubclusterList.add(cluster);
+}
+}
+subclusterList = filteredSubclusterList;
Review Comment:
You can change the code indentation.
> RBF mount point with SPACE order can not find the most available namespace,
> it will choose an irrelevant namespace
> --
>
> Key: HDFS-17602
> URL: https://issues.apache.org/jira/browse/HDFS-17602
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: router
>Reporter: Zhongkun Wu
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2024-08-12-10-08-54-031.png,
> image-2024-08-12-10-08-58-271.png, image-2024-08-12-10-12-48-428.png,
> image-2024-08-12-10-14-20-580.png, image-2024-08-12-10-25-26-003.png,
> image-2024-08-12-10-25-42-863.png
>
>
> !image-2024-08-12-10-08-58-271.png!
> In the
> MultipleDestinationMountTableResolver.java we invoke
> orderedResolver.getFirstNamespace(path, mountTableResult);
> It will then invoke this function in RouterResolver.java
> !image-2024-08-12-10-12-48-428.png!
> and now we are in
> chooseFirstNamespace function in AvailableSpaceResolver.java
> !image-2024-08-12-10-14-20-580.png!
>
> The path parameter is the destination where we want to create a file
> the loc parameter is the mount point we set
>
> this function will choose the most available namespace in all the namespace
> we have in StateStore, which is not the same as the mount point we set for
> our destination.
>
> As a result we will get a namespace irrelevant to the namespaces we set for
> the destination path
>
> !image-2024-08-12-10-25-42-863.png!
> in the log above:
> we get the namespace we don't set with our destination path, So the it will
> choose the first namespace it sees and it's not really the most available
> namespace among the namespaces we set for our destination
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17602) RBF mount point with SPACE order can not find the most available namespace, it will choose an irrelevant namespace
[
https://issues.apache.org/jira/browse/HDFS-17602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878783#comment-17878783
]
ASF GitHub Bot commented on HDFS-17602:
---
KeeProMise commented on code in PR #6991:
URL: https://github.com/apache/hadoop/pull/6991#discussion_r1741714989
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/resolver/order/AvailableSpaceResolver.java:
##
@@ -118,6 +121,20 @@ protected String chooseFirstNamespace(String path,
PathLocation loc) {
getSubclusterMapping();
List subclusterList = new LinkedList<>(
subclusterInfo.values());
+
+if (loc != null && loc.getDestinations() != null) {
Review Comment:
Hi, @wzk784533 I think this modification is good, but I want to consult
under what circumstances this loc or loc.getDestinations() would be null.
> RBF mount point with SPACE order can not find the most available namespace,
> it will choose an irrelevant namespace
> --
>
> Key: HDFS-17602
> URL: https://issues.apache.org/jira/browse/HDFS-17602
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: router
>Reporter: Zhongkun Wu
>Priority: Critical
> Labels: pull-request-available
> Attachments: image-2024-08-12-10-08-54-031.png,
> image-2024-08-12-10-08-58-271.png, image-2024-08-12-10-12-48-428.png,
> image-2024-08-12-10-14-20-580.png, image-2024-08-12-10-25-26-003.png,
> image-2024-08-12-10-25-42-863.png
>
>
> !image-2024-08-12-10-08-58-271.png!
> In the
> MultipleDestinationMountTableResolver.java we invoke
> orderedResolver.getFirstNamespace(path, mountTableResult);
> It will then invoke this function in RouterResolver.java
> !image-2024-08-12-10-12-48-428.png!
> and now we are in
> chooseFirstNamespace function in AvailableSpaceResolver.java
> !image-2024-08-12-10-14-20-580.png!
>
> The path parameter is the destination where we want to create a file
> the loc parameter is the mount point we set
>
> this function will choose the most available namespace in all the namespace
> we have in StateStore, which is not the same as the mount point we set for
> our destination.
>
> As a result we will get a namespace irrelevant to the namespaces we set for
> the destination path
>
> !image-2024-08-12-10-25-42-863.png!
> in the log above:
> we get the namespace we don't set with our destination path, So the it will
> choose the first namespace it sees and it's not really the most available
> namespace among the namespaces we set for our destination
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17611) Move all DistCp execution logic to execute()
[ https://issues.apache.org/jira/browse/HDFS-17611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878742#comment-17878742 ] ASF GitHub Bot commented on HDFS-17611: --- kokon191 opened a new pull request, #7025: URL: https://github.com/apache/hadoop/pull/7025 ### Description of PR Many code flows create a DistCp instance and call the public method execute() to get the Job object for better control over the distcp job but some logics are only called by the run() method. Should move these lines over to execute(). ### How was this patch tested? Trivial code change > Move all DistCp execution logic to execute() > > > Key: HDFS-17611 > URL: https://issues.apache.org/jira/browse/HDFS-17611 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Felix N >Assignee: Felix N >Priority: Minor > > Many code flows create a DistCp instance and call the public method execute() > to get the Job object for better control over the distcp job but some logics > are only called by the run() method. Should move these lines over to > execute(). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17611) Move all DistCp execution logic to execute()
[ https://issues.apache.org/jira/browse/HDFS-17611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17611: -- Labels: pull-request-available (was: ) > Move all DistCp execution logic to execute() > > > Key: HDFS-17611 > URL: https://issues.apache.org/jira/browse/HDFS-17611 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Felix N >Assignee: Felix N >Priority: Minor > Labels: pull-request-available > > Many code flows create a DistCp instance and call the public method execute() > to get the Job object for better control over the distcp job but some logics > are only called by the run() method. Should move these lines over to > execute(). -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16084) getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv() failed
[
https://issues.apache.org/jira/browse/HDFS-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878690#comment-17878690
]
ASF GitHub Bot commented on HDFS-16084:
---
Hexiaoqiao commented on PR #6969:
URL: https://github.com/apache/hadoop/pull/6969#issuecomment-2325527015
> > Committed to trunk. Thanks @kevincai for your contributions!
> > BTW, what's your JIRA account? When I want to assign
https://issues.apache.org/jira/browse/HDFS-16084 to you but I am not sure if
the reporter and you is the same one. Thanks!
>
> I didn't apply one.
What about apply one at JIRA side? @kevincai
> getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv()
> failed
>
>
> Key: HDFS-16084
> URL: https://issues.apache.org/jira/browse/HDFS-16084
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.2.1
>Reporter: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> First reported in ARROW-13011: when a libhdfs API call fails because
> CLASSPATH isn't set, calling the API a second time leads to a crash.
> *Backtrace*
> This was obtained from the ARROW-13011 reproducer:
> {code:java}
> #0 globalClassReference (className=className@entry=0x7f75883c13b0
> "org/apache/hadoop/conf/Configuration", env=env@entry=0x6c2f2f3a73666468,
> out=out@entry=0x7fffd86e3020) at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:279
> #1 0x7f75883b9511 in constructNewObjectOfClass
> (env=env@entry=0x6c2f2f3a73666468, out=out@entry=0x7fffd86e3148,
> className=className@entry=0x7f75883c13b0
> "org/apache/hadoop/conf/Configuration",
> ctorSignature=ctorSignature@entry=0x7f75883c1180 "()V")
> at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:212
> #2 0x7f75883bb6d0 in hdfsBuilderConnect (bld=0x5562e4bbb3e0) at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:700
> #3 0x7f758de31ef3 in arrow::io::internal::LibHdfsShim::BuilderConnect
> (this=0x7f758e768240 namespace)::libhdfs_shim>,
> bld=0x5562e4bbb3e0) at /arrow/cpp/src/arrow/io/hdfs_internal.cc:366
> #4 0x7f758de2d098 in
> arrow::io::HadoopFileSystem::HadoopFileSystemImpl::Connect
> (this=0x5562e4a9f750, config=0x5562e46edc30)
> at /arrow/cpp/src/arrow/io/hdfs.cc:372
> #5 0x7f758de2e646 in arrow::io::HadoopFileSystem::Connect
> (config=0x5562e46edc30, fs=0x5562e46edd08) at
> /arrow/cpp/src/arrow/io/hdfs.cc:590
> #6 0x7f758d532d2a in arrow::fs::HadoopFileSystem::Impl::Init
> (this=0x5562e46edc30) at /arrow/cpp/src/arrow/filesystem/hdfs.cc:59
> #7 0x7f758d536931 in arrow::fs::HadoopFileSystem::Make (options=...,
> io_context=...) at /arrow/cpp/src/arrow/filesystem/hdfs.cc:409
> #8 0x7f75885d7445 in
> __pyx_pf_7pyarrow_5_hdfs_16HadoopFileSystem___init__
> (__pyx_v_self=0x7f758871a970, __pyx_v_host=0x7f758871cc00, __pyx_v_port=8020,
> __pyx_v_user=0x5562e3af6d30 <_Py_NoneStruct>, __pyx_v_replication=3,
> __pyx_v_buffer_size=0, __pyx_v_default_block_size=0x5562e3af6d30
> <_Py_NoneStruct>,
> __pyx_v_kerb_ticket=0x5562e3af6d30 <_Py_NoneStruct>,
> __pyx_v_extra_conf=0x5562e3af6d30 <_Py_NoneStruct>) at _hdfs.cpp:4759
> #9 0x7f75885d4c88 in
> __pyx_pw_7pyarrow_5_hdfs_16HadoopFileSystem_1__init__
> (__pyx_v_self=0x7f758871a970, __pyx_args=0x7f75900bb048,
> __pyx_kwds=0x7f7590033a68)
> at _hdfs.cpp:4343
> #10 0x5562e38ca747 in type_call () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/typeobject.c:915
> #11 0x5562e39117a3 in _PyObject_FastCallDict (kwargs=,
> nargs=, args=,
> func=0x7f75885f1420 <__pyx_type_7pyarrow_5_hdfs_HadoopFileSystem>) at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/tupleobject.c:76
> #12 _PyObject_FastCallKeywords () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/abstract.c:2496
> #13 0x5562e39121d5 in call_function () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4875
> #14 0x5562e3973d68 in _PyEval_EvalFrameDefault () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:3351
> #15 0x5562e38b98f5 in PyEval_EvalFrameEx (throwflag=0, f=0x7f74c0664768)
> at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4166
> #16 _PyEval_EvalCodeWithName () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4166
> #17 0x5562e38bad79 in PyEval_EvalCodeEx (_co=,
> globa
[jira] [Commented] (HDFS-16084) getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv() failed
[
https://issues.apache.org/jira/browse/HDFS-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878689#comment-17878689
]
ASF GitHub Bot commented on HDFS-16084:
---
Hexiaoqiao commented on PR #6969:
URL: https://github.com/apache/hadoop/pull/6969#issuecomment-2325525755
@steveloughran No plan to backport to other active branches if no more
requirement.
> so target 3.4.2?
Not see any place marked 3.4.2, would you mind to confirm? Thanks.
> getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv()
> failed
>
>
> Key: HDFS-16084
> URL: https://issues.apache.org/jira/browse/HDFS-16084
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.2.1
>Reporter: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> First reported in ARROW-13011: when a libhdfs API call fails because
> CLASSPATH isn't set, calling the API a second time leads to a crash.
> *Backtrace*
> This was obtained from the ARROW-13011 reproducer:
> {code:java}
> #0 globalClassReference (className=className@entry=0x7f75883c13b0
> "org/apache/hadoop/conf/Configuration", env=env@entry=0x6c2f2f3a73666468,
> out=out@entry=0x7fffd86e3020) at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:279
> #1 0x7f75883b9511 in constructNewObjectOfClass
> (env=env@entry=0x6c2f2f3a73666468, out=out@entry=0x7fffd86e3148,
> className=className@entry=0x7f75883c13b0
> "org/apache/hadoop/conf/Configuration",
> ctorSignature=ctorSignature@entry=0x7f75883c1180 "()V")
> at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:212
> #2 0x7f75883bb6d0 in hdfsBuilderConnect (bld=0x5562e4bbb3e0) at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:700
> #3 0x7f758de31ef3 in arrow::io::internal::LibHdfsShim::BuilderConnect
> (this=0x7f758e768240 namespace)::libhdfs_shim>,
> bld=0x5562e4bbb3e0) at /arrow/cpp/src/arrow/io/hdfs_internal.cc:366
> #4 0x7f758de2d098 in
> arrow::io::HadoopFileSystem::HadoopFileSystemImpl::Connect
> (this=0x5562e4a9f750, config=0x5562e46edc30)
> at /arrow/cpp/src/arrow/io/hdfs.cc:372
> #5 0x7f758de2e646 in arrow::io::HadoopFileSystem::Connect
> (config=0x5562e46edc30, fs=0x5562e46edd08) at
> /arrow/cpp/src/arrow/io/hdfs.cc:590
> #6 0x7f758d532d2a in arrow::fs::HadoopFileSystem::Impl::Init
> (this=0x5562e46edc30) at /arrow/cpp/src/arrow/filesystem/hdfs.cc:59
> #7 0x7f758d536931 in arrow::fs::HadoopFileSystem::Make (options=...,
> io_context=...) at /arrow/cpp/src/arrow/filesystem/hdfs.cc:409
> #8 0x7f75885d7445 in
> __pyx_pf_7pyarrow_5_hdfs_16HadoopFileSystem___init__
> (__pyx_v_self=0x7f758871a970, __pyx_v_host=0x7f758871cc00, __pyx_v_port=8020,
> __pyx_v_user=0x5562e3af6d30 <_Py_NoneStruct>, __pyx_v_replication=3,
> __pyx_v_buffer_size=0, __pyx_v_default_block_size=0x5562e3af6d30
> <_Py_NoneStruct>,
> __pyx_v_kerb_ticket=0x5562e3af6d30 <_Py_NoneStruct>,
> __pyx_v_extra_conf=0x5562e3af6d30 <_Py_NoneStruct>) at _hdfs.cpp:4759
> #9 0x7f75885d4c88 in
> __pyx_pw_7pyarrow_5_hdfs_16HadoopFileSystem_1__init__
> (__pyx_v_self=0x7f758871a970, __pyx_args=0x7f75900bb048,
> __pyx_kwds=0x7f7590033a68)
> at _hdfs.cpp:4343
> #10 0x5562e38ca747 in type_call () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/typeobject.c:915
> #11 0x5562e39117a3 in _PyObject_FastCallDict (kwargs=,
> nargs=, args=,
> func=0x7f75885f1420 <__pyx_type_7pyarrow_5_hdfs_HadoopFileSystem>) at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/tupleobject.c:76
> #12 _PyObject_FastCallKeywords () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/abstract.c:2496
> #13 0x5562e39121d5 in call_function () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4875
> #14 0x5562e3973d68 in _PyEval_EvalFrameDefault () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:3351
> #15 0x5562e38b98f5 in PyEval_EvalFrameEx (throwflag=0, f=0x7f74c0664768)
> at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4166
> #16 _PyEval_EvalCodeWithName () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4166
> #17 0x5562e38bad79 in PyEval_EvalCodeEx (_co=,
> globals=, locals=, args=,
> argcount=,
> kws=, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0,
> closure=0x0)
> at
> /home/conda/feedstock_root/b
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[
https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878446#comment-17878446
]
ASF GitHub Bot commented on HDFS-17381:
---
hadoop-yetus commented on PR #6551:
URL: https://github.com/apache/hadoop/pull/6551#issuecomment-2323453513
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 8s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 28s | | trunk passed |
| +1 :green_heart: | compile | 9m 2s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 8m 26s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 2m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 12s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 55s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 38s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 52s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 23s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 17s | | the patch passed |
| +1 :green_heart: | compile | 8m 36s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 8m 36s | | the patch passed |
| +1 :green_heart: | compile | 8m 30s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 8m 30s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 6s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/8/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 7 new + 106 unchanged - 0 fixed = 113 total (was
106) |
| +1 :green_heart: | mvnsite | 2m 10s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 48s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 44s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 4m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 58s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 11s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 1s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 24m 44s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 43s | | The patch does not
generate ASF License warnings. |
| | | 183m 9s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/8/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6551 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 892917e01b88 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / f22360af04c4df7182297f472be768e5c6ae3be5 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-mul
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[
https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878441#comment-17878441
]
ASF GitHub Bot commented on HDFS-17381:
---
sadanand48 commented on code in PR #6551:
URL: https://github.com/apache/hadoop/pull/6551#discussion_r1740148045
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/WithErasureCoding.java:
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs;
+
+import java.io.IOException;
+
+/**
+ * Filesystems that support EC can implement this interface.
+ */
+public interface WithErasureCoding {
Review Comment:
Done. (Elaborated these in the javadoc)
> Distcp of EC files should not be limited to DFS.
>
>
> Key: HDFS-17381
> URL: https://issues.apache.org/jira/browse/HDFS-17381
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Reporter: Sadanand Shenoy
>Assignee: Sadanand Shenoy
>Priority: Major
> Labels: pull-request-available
>
> Currently EC file support in distcp is limited to DFS and the code checks if
> the given FS instance is DistributedFileSystem, In Ozone, EC is supported
> now, and this limitation can be removed and a general contract for any
> filesystem that supports EC files should be allowed by implementing few
> interfaces/methods.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[
https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878440#comment-17878440
]
ASF GitHub Bot commented on HDFS-17381:
---
sadanand48 commented on code in PR #6551:
URL: https://github.com/apache/hadoop/pull/6551#discussion_r1740147794
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/WithErasureCoding.java:
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs;
+
+import java.io.IOException;
+
+/**
+ * Filesystems that support EC can implement this interface.
+ */
+public interface WithErasureCoding {
+
+ /**
+ * Get the EC Policy name of the given file
+ * @param fileStatus object of the file whose ecPolicy needs to be obtained
+ * @return the ec Policy name
+ */
+ String getErasureCodingPolicyName(FileStatus fileStatus);
+
+ /**
+ * Set the given ecPolicy on the path
+ * @param path on which the EC policy needs to be set
+ * @throws IOException if the set is not successful
+ */
+ void setErasureCodingPolicy(Path path, String ecPolicyName) throws
Review Comment:
Done. (Elaborated these in the javadoc)
> Distcp of EC files should not be limited to DFS.
>
>
> Key: HDFS-17381
> URL: https://issues.apache.org/jira/browse/HDFS-17381
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Reporter: Sadanand Shenoy
>Assignee: Sadanand Shenoy
>Priority: Major
> Labels: pull-request-available
>
> Currently EC file support in distcp is limited to DFS and the code checks if
> the given FS instance is DistributedFileSystem, In Ozone, EC is supported
> now, and this limitation can be removed and a general contract for any
> filesystem that supports EC files should be allowed by implementing few
> interfaces/methods.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[
https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878439#comment-17878439
]
ASF GitHub Bot commented on HDFS-17381:
---
sadanand48 commented on code in PR #6551:
URL: https://github.com/apache/hadoop/pull/6551#discussion_r1740147516
##
hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java:
##
@@ -3862,6 +3869,11 @@ protected EnumSet getFlags() {
*/
@Override
public FSDataOutputStream build() throws IOException {
+ if (getOptionalKeys().contains(
+ Options.OpenFileOptions.FS_OPTION_OPENFILE_EC_POLICY)) {
+ecPolicyName(getOptions().get(
Review Comment:
Done.
##
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableDirectoryCreateCommand.java:
##
@@ -66,11 +68,12 @@ protected Object doExecute(Object... arguments) throws
Exception {
boolean preserveEC = getFileAttributeSettings(context)
.contains(DistCpOptions.FileAttribute.ERASURECODINGPOLICY);
if (preserveEC && sourceStatus.isErasureCoded()
-&& targetFS instanceof DistributedFileSystem) {
- ErasureCodingPolicy ecPolicy =
- ((HdfsFileStatus) sourceStatus).getErasureCodingPolicy();
- DistributedFileSystem dfs = (DistributedFileSystem) targetFS;
- dfs.setErasureCodingPolicy(target, ecPolicy.getName());
+&& targetFS instanceof WithErasureCoding) {
+ ErasureCodingPolicy ecPolicy = SystemErasureCodingPolicies.getByName(
Review Comment:
Done.
> Distcp of EC files should not be limited to DFS.
>
>
> Key: HDFS-17381
> URL: https://issues.apache.org/jira/browse/HDFS-17381
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Reporter: Sadanand Shenoy
>Assignee: Sadanand Shenoy
>Priority: Major
> Labels: pull-request-available
>
> Currently EC file support in distcp is limited to DFS and the code checks if
> the given FS instance is DistributedFileSystem, In Ozone, EC is supported
> now, and this limitation can be removed and a general contract for any
> filesystem that supports EC files should be allowed by implementing few
> interfaces/methods.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[
https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878438#comment-17878438
]
ASF GitHub Bot commented on HDFS-17381:
---
sadanand48 commented on code in PR #6551:
URL: https://github.com/apache/hadoop/pull/6551#discussion_r1740147441
##
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java:
##
@@ -207,9 +209,13 @@ private long copyToFile(Path targetPath, FileSystem
targetFS,
ErasureCodingPolicy ecPolicy = null;
if (preserveEC && sourceStatus.isErasureCoded()
-&& sourceStatus instanceof HdfsFileStatus
-&& targetFS instanceof DistributedFileSystem) {
- ecPolicy = ((HdfsFileStatus) sourceStatus).getErasureCodingPolicy();
+&& sourceFS instanceof WithErasureCoding
+&& targetFS instanceof WithErasureCoding) {
+ String ecPolicyName =
+ ((WithErasureCoding)
sourceFS).getErasureCodingPolicyName(sourceStatus);
+ if (ecPolicyName != null) {
+ecPolicy = SystemErasureCodingPolicies.getByName(ecPolicyName);
Review Comment:
Done.
##
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java:
##
@@ -24,18 +24,19 @@
import java.util.EnumSet;
import org.apache.hadoop.fs.FileStatus;
-import org.apache.hadoop.hdfs.DistributedFileSystem;
import org.apache.hadoop.hdfs.protocol.ErasureCodingPolicy;
-import org.apache.hadoop.hdfs.protocol.HdfsFileStatus;
+import org.apache.hadoop.hdfs.protocol.SystemErasureCodingPolicies;
import org.apache.hadoop.tools.DistCpOptions;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.CreateFlag;
+import org.apache.hadoop.fs.WithErasureCoding;
Review Comment:
Done.
> Distcp of EC files should not be limited to DFS.
>
>
> Key: HDFS-17381
> URL: https://issues.apache.org/jira/browse/HDFS-17381
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Reporter: Sadanand Shenoy
>Assignee: Sadanand Shenoy
>Priority: Major
> Labels: pull-request-available
>
> Currently EC file support in distcp is limited to DFS and the code checks if
> the given FS instance is DistributedFileSystem, In Ozone, EC is supported
> now, and this limitation can be removed and a general contract for any
> filesystem that supports EC files should be allowed by implementing few
> interfaces/methods.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[
https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878435#comment-17878435
]
ASF GitHub Bot commented on HDFS-17381:
---
hadoop-yetus commented on PR #6551:
URL: https://github.com/apache/hadoop/pull/6551#issuecomment-2323360824
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 18s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 7s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 20m 48s | | trunk passed |
| +1 :green_heart: | compile | 9m 1s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 8m 27s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 2m 9s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 14s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 54s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 46s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 46s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 6s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 21s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 12s | | the patch passed |
| +1 :green_heart: | compile | 8m 46s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 8m 46s | | the patch passed |
| +1 :green_heart: | compile | 8m 28s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 8m 28s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 0s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/7/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 7 new + 88 unchanged - 0 fixed = 95 total (was
88) |
| +1 :green_heart: | mvnsite | 2m 10s | | the patch passed |
| -1 :x: | javadoc | 0m 44s |
[/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/7/artifact/out/patch-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
| hadoop-common in the patch failed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04. |
| +1 :green_heart: | javadoc | 1m 42s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 4m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 21m 14s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 13s | | hadoop-common in the patch
passed. |
| -1 :x: | unit | 2m 3s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/7/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs-client.txt)
| hadoop-hdfs-client in the patch passed. |
| +1 :green_heart: | unit | 24m 15s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 44s | | The patch does not
generate ASF License warnings. |
| | | 183m 45s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.protocol.TestHdfsFileStatusMethods |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/7/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6551 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[
https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878227#comment-17878227
]
ASF GitHub Bot commented on HDFS-17381:
---
hadoop-yetus commented on PR #6551:
URL: https://github.com/apache/hadoop/pull/6551#issuecomment-2322155175
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 5m 10s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 37s | | trunk passed |
| +1 :green_heart: | compile | 10m 23s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 9m 43s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 2m 20s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 2s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 36s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 45s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 47s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 18s | | the patch passed |
| +1 :green_heart: | compile | 9m 56s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 9m 56s | | the patch passed |
| +1 :green_heart: | compile | 9m 54s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 9m 54s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 12s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/6/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 6 new + 45 unchanged - 0 fixed = 51 total (was
45) |
| +1 :green_heart: | mvnsite | 1m 59s | | the patch passed |
| -1 :x: | javadoc | 0m 36s |
[/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/6/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
|
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | javadoc | 1m 23s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 4m 7s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 4s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 16s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 2m 0s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 24m 11s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 194m 54s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6551 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 00ace163478c 5.15.0-117-generic #1
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[
https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878226#comment-17878226
]
ASF GitHub Bot commented on HDFS-17381:
---
hadoop-yetus commented on PR #6551:
URL: https://github.com/apache/hadoop/pull/6551#issuecomment-2322155105
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 46s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 21m 44s | | trunk passed |
| +1 :green_heart: | compile | 10m 30s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 9m 55s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 2m 11s | | trunk passed |
| +1 :green_heart: | mvnsite | 2m 7s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 29s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 19s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 37s | | trunk passed |
| +1 :green_heart: | shadedclient | 21m 53s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 24s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 19s | | the patch passed |
| +1 :green_heart: | compile | 9m 55s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 9m 55s | | the patch passed |
| +1 :green_heart: | compile | 9m 43s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 9m 43s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 2m 12s |
[/results-checkstyle-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/5/artifact/out/results-checkstyle-root.txt)
| root: The patch generated 6 new + 45 unchanged - 0 fixed = 51 total (was
45) |
| +1 :green_heart: | mvnsite | 1m 54s | | the patch passed |
| -1 :x: | javadoc | 0m 36s |
[/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/5/artifact/out/results-javadoc-javadoc-hadoop-common-project_hadoop-common-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
|
hadoop-common-project_hadoop-common-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | javadoc | 1m 22s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 4m 5s | | the patch passed |
| +1 :green_heart: | shadedclient | 22m 1s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 16m 14s | | hadoop-common in the patch
passed. |
| +1 :green_heart: | unit | 1m 56s | | hadoop-hdfs-client in the patch
passed. |
| +1 :green_heart: | unit | 24m 19s | | hadoop-distcp in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 196m 56s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6551/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6551 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux e965f57831f3 5.15.0-117-generic #1
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[
https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878213#comment-17878213
]
ASF GitHub Bot commented on HDFS-17381:
---
steveloughran commented on code in PR #6551:
URL: https://github.com/apache/hadoop/pull/6551#discussion_r1739192871
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/WithErasureCoding.java:
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs;
+
+import java.io.IOException;
+
+/**
+ * Filesystems that support EC can implement this interface.
+ */
+public interface WithErasureCoding {
+
+ /**
+ * Get the EC Policy name of the given file
+ * @param fileStatus object of the file whose ecPolicy needs to be obtained
+ * @return the ec Policy name
+ */
+ String getErasureCodingPolicyName(FileStatus fileStatus);
Review Comment:
you're going to have to define this "fully", specifically
* what if a filestatus for a different FS schema is passed in?
* can a null string be returned
* can an empty string be returned
* if a file has no erasure coding, what is returned?
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/WithErasureCoding.java:
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs;
+
+import java.io.IOException;
+
+/**
+ * Filesystems that support EC can implement this interface.
+ */
+public interface WithErasureCoding {
+
+ /**
+ * Get the EC Policy name of the given file
+ * @param fileStatus object of the file whose ecPolicy needs to be obtained
+ * @return the ec Policy name
+ */
+ String getErasureCodingPolicyName(FileStatus fileStatus);
+
+ /**
+ * Set the given ecPolicy on the path
Review Comment:
nit, add a `.`
##
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/WithErasureCoding.java:
##
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.fs;
+
+import java.io.IOException;
+
+/**
+ * Filesystems that support EC can implement this interface.
+ */
+public interface WithErasureCoding {
+
+ /**
+ * Get the EC Policy name of the given file
+ * @param fileStatus object of the file whose ecPolicy needs to be obtained
+ * @return the ec Policy name
+ */
+ String getErasureCodingPolicyName(FileStatus fileStatus);
+
+ /**
+ * Set the given ecPolicy on the path
+ * @param path on which the EC policy needs to be set
+ * @throws IOException if the set is not suc
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[ https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878171#comment-17878171 ] ASF GitHub Bot commented on HDFS-17381: --- sadanand48 commented on PR #6551: URL: https://github.com/apache/hadoop/pull/6551#issuecomment-2321716400 Sorry for the long silence here, Just picked this again. @steveloughran Thanks for the review comments, I have made the changes you suggested, could you please take a look if it looks ok? I will test it with this then. I don't see many usages of the createFile() api that returns the builder object though and I see this jira (HADOOP-14384) that reduces usage of this API, I hope we can use it now. > Distcp of EC files should not be limited to DFS. > > > Key: HDFS-17381 > URL: https://issues.apache.org/jira/browse/HDFS-17381 > Project: Hadoop HDFS > Issue Type: Bug > Components: distcp >Reporter: Sadanand Shenoy >Assignee: Sadanand Shenoy >Priority: Major > Labels: pull-request-available > > Currently EC file support in distcp is limited to DFS and the code checks if > the given FS instance is DistributedFileSystem, In Ozone, EC is supported > now, and this limitation can be removed and a general contract for any > filesystem that supports EC files should be allowed by implementing few > interfaces/methods. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17381) Distcp of EC files should not be limited to DFS.
[
https://issues.apache.org/jira/browse/HDFS-17381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17878169#comment-17878169
]
ASF GitHub Bot commented on HDFS-17381:
---
sadanand48 commented on code in PR #6551:
URL: https://github.com/apache/hadoop/pull/6551#discussion_r1738970299
##
hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableDirectoryCreateCommand.java:
##
@@ -67,31 +67,13 @@ protected Object doExecute(Object... arguments) throws
Exception {
boolean preserveEC = getFileAttributeSettings(context)
.contains(DistCpOptions.FileAttribute.ERASURECODINGPOLICY);
-if (preserveEC && sourceStatus.isErasureCoded()) {
- ErasureCodingPolicy ecPolicy = null;
- if (sourceFs instanceof DistributedFileSystem) {
-ecPolicy = ((HdfsFileStatus) sourceStatus).getErasureCodingPolicy();
- } else {
-try {
- String ecPolicyName = (String) getErasureCodingPolicyMethod(
- sourceFs).invoke(sourceFs,sourceStatus);
- ecPolicy = SystemErasureCodingPolicies.getByName(ecPolicyName);
-} catch (NoSuchMethodException exception){
- return false;
-}
- }
-
- if (targetFS instanceof DistributedFileSystem) {
-DistributedFileSystem dfs = (DistributedFileSystem) targetFS;
-dfs.setErasureCodingPolicy(target, ecPolicy.getName());
- } else {
-try {
- setErasureCodingPolicyMethod(targetFS).invoke(targetFS,
- ecPolicy.getName());
-} catch (NoSuchMethodException exception) {
- return false;
-}
- }
+if (preserveEC && sourceStatus.isErasureCoded()
+&& targetFS instanceof WithErasureCoding) {
+ ErasureCodingPolicy ecPolicy = SystemErasureCodingPolicies.getByName(
+ ((WithErasureCoding) sourceFs).getErasureCodingPolicyName(
+ sourceStatus));
+ WithErasureCoding ecFs = (DistributedFileSystem) targetFS;
Review Comment:
fixed.
> Distcp of EC files should not be limited to DFS.
>
>
> Key: HDFS-17381
> URL: https://issues.apache.org/jira/browse/HDFS-17381
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: distcp
>Reporter: Sadanand Shenoy
>Assignee: Sadanand Shenoy
>Priority: Major
> Labels: pull-request-available
>
> Currently EC file support in distcp is limited to DFS and the code checks if
> the given FS instance is DistributedFileSystem, In Ozone, EC is supported
> now, and this limitation can be removed and a general contract for any
> filesystem that supports EC files should be allowed by implementing few
> interfaces/methods.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17599) EC: Fix the mismatch between locations and indices for mover
[
https://issues.apache.org/jira/browse/HDFS-17599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877963#comment-17877963
]
ASF GitHub Bot commented on HDFS-17599:
---
haiyang1987 commented on PR #6980:
URL: https://github.com/apache/hadoop/pull/6980#issuecomment-2320052315
Committed to trunk. Thanks @tomscut for your works. And @Hexiaoqiao for
your reviews.
> EC: Fix the mismatch between locations and indices for mover
>
>
> Key: HDFS-17599
> URL: https://issues.apache.org/jira/browse/HDFS-17599
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.0, 3.4.0
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-08-03-17-59-08-059.png,
> image-2024-08-03-18-00-01-950.png
>
>
> We set the EC policy to (6+3) and also have nodes that were in state
> ENTERING_MAINTENANCE.
>
> When we move the data of some directories from SSD to HDD, some blocks move
> fail due to disk full, as shown in the figure below
> (blk_-9223372033441574269).
> We tried to move again and found the following error "{color:#ff}Replica
> does not exist{color}".
> Observing the information of fsck, it can be found that the wrong
> blockid(blk_-9223372033441574270) was found when moving block.
>
> {*}Mover Logs{*}:
> !image-2024-08-03-17-59-08-059.png|width=741,height=85!
>
> {*}FSCK Info{*}:
> !image-2024-08-03-18-00-01-950.png|width=738,height=120!
>
> {*}Root Cause{*}:
> Similar to this HDFS-16333, when mover is initialized, only the `LIVE` node
> is processed. As a result, the datanode in the `ENTERING_MAINTENANCE` state
> in the locations is filtered when initializing `DBlockStriped`, but the
> indices are not adapted, resulting in a mismatch between the location and
> indices lengths. Finally, ec block calculates the wrong blockid when getting
> internal block (see `DBlockStriped#getInternalBlock`).
>
> We added debug logs, and a few key messages are shown below.
> {color:#ff}The result is an incorrect correspondence: xx.xx.7.31 ->
> -9223372033441574270{color}.
> {code:java}
> DBlock getInternalBlock(StorageGroup storage) {
> // storage == xx.xx.7.31
> // idxInLocs == 1 (location ([xx.xx.,85.29:DISK, xx.xx.7.31:DISK,
> xx.xx.207.22:DISK, xx.xx.8.25:DISK, xx.xx.79.30:DISK, xx.xx.87.21:DISK,
> xx.xx.8.38:DISK]), xx.xx.179.31 is in the ENTERING_MAINTENANCE state is
> filtered)
> int idxInLocs = locations.indexOf(storage);
> if (idxInLocs == -1) {
> return null;
> }
> // idxInGroup == 2 (indices is [1,2,3,4,5,6,7,8])
> byte idxInGroup = indices[idxInLocs];
> // blkId: -9223372033441574272 + 2 = -9223372033441574270
> long blkId = getBlock().getBlockId() + idxInGroup;
> long numBytes = getInternalBlockLength(getNumBytes(), cellSize,
> dataBlockNum, idxInGroup);
> Block blk = new Block(getBlock());
> blk.setBlockId(blkId);
> blk.setNumBytes(numBytes);
> DBlock dblk = new DBlock(blk);
> dblk.addLocation(storage);
> return dblk;
> } {code}
> {*}Solution{*}:
> When initializing DBlockStriped, if any location is filtered out, we need to
> remove the corresponding element in the indices to do the adaptation.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17599) EC: Fix the mismatch between locations and indices for mover
[
https://issues.apache.org/jira/browse/HDFS-17599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877962#comment-17877962
]
ASF GitHub Bot commented on HDFS-17599:
---
haiyang1987 merged PR #6980:
URL: https://github.com/apache/hadoop/pull/6980
> EC: Fix the mismatch between locations and indices for mover
>
>
> Key: HDFS-17599
> URL: https://issues.apache.org/jira/browse/HDFS-17599
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.0, 3.4.0
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-08-03-17-59-08-059.png,
> image-2024-08-03-18-00-01-950.png
>
>
> We set the EC policy to (6+3) and also have nodes that were in state
> ENTERING_MAINTENANCE.
>
> When we move the data of some directories from SSD to HDD, some blocks move
> fail due to disk full, as shown in the figure below
> (blk_-9223372033441574269).
> We tried to move again and found the following error "{color:#ff}Replica
> does not exist{color}".
> Observing the information of fsck, it can be found that the wrong
> blockid(blk_-9223372033441574270) was found when moving block.
>
> {*}Mover Logs{*}:
> !image-2024-08-03-17-59-08-059.png|width=741,height=85!
>
> {*}FSCK Info{*}:
> !image-2024-08-03-18-00-01-950.png|width=738,height=120!
>
> {*}Root Cause{*}:
> Similar to this HDFS-16333, when mover is initialized, only the `LIVE` node
> is processed. As a result, the datanode in the `ENTERING_MAINTENANCE` state
> in the locations is filtered when initializing `DBlockStriped`, but the
> indices are not adapted, resulting in a mismatch between the location and
> indices lengths. Finally, ec block calculates the wrong blockid when getting
> internal block (see `DBlockStriped#getInternalBlock`).
>
> We added debug logs, and a few key messages are shown below.
> {color:#ff}The result is an incorrect correspondence: xx.xx.7.31 ->
> -9223372033441574270{color}.
> {code:java}
> DBlock getInternalBlock(StorageGroup storage) {
> // storage == xx.xx.7.31
> // idxInLocs == 1 (location ([xx.xx.,85.29:DISK, xx.xx.7.31:DISK,
> xx.xx.207.22:DISK, xx.xx.8.25:DISK, xx.xx.79.30:DISK, xx.xx.87.21:DISK,
> xx.xx.8.38:DISK]), xx.xx.179.31 is in the ENTERING_MAINTENANCE state is
> filtered)
> int idxInLocs = locations.indexOf(storage);
> if (idxInLocs == -1) {
> return null;
> }
> // idxInGroup == 2 (indices is [1,2,3,4,5,6,7,8])
> byte idxInGroup = indices[idxInLocs];
> // blkId: -9223372033441574272 + 2 = -9223372033441574270
> long blkId = getBlock().getBlockId() + idxInGroup;
> long numBytes = getInternalBlockLength(getNumBytes(), cellSize,
> dataBlockNum, idxInGroup);
> Block blk = new Block(getBlock());
> blk.setBlockId(blkId);
> blk.setNumBytes(numBytes);
> DBlock dblk = new DBlock(blk);
> dblk.addLocation(storage);
> return dblk;
> } {code}
> {*}Solution{*}:
> When initializing DBlockStriped, if any location is filtered out, we need to
> remove the corresponding element in the indices to do the adaptation.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17594) [ARR] RouterCacheAdmin supports asynchronous rpc.
[
https://issues.apache.org/jira/browse/HDFS-17594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877762#comment-17877762
]
ASF GitHub Bot commented on HDFS-17594:
---
hadoop-yetus commented on PR #6986:
URL: https://github.com/apache/hadoop/pull/6986#issuecomment-2318280824
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 18m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 4 new or modified test files. |
_ HDFS-17531 Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 58s | | HDFS-17531 passed |
| +1 :green_heart: | compile | 0m 51s | | HDFS-17531 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 0m 39s | | HDFS-17531 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 31s | | HDFS-17531 passed |
| +1 :green_heart: | mvnsite | 0m 44s | | HDFS-17531 passed |
| +1 :green_heart: | javadoc | 0m 44s | | HDFS-17531 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 31s | | HDFS-17531 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 1m 27s | | HDFS-17531 passed |
| +1 :green_heart: | shadedclient | 39m 13s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 0m 43s | | the patch passed |
| +1 :green_heart: | compile | 0m 31s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 0m 31s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 20s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 34s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 30s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 25s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 1m 28s | | the patch passed |
| +1 :green_heart: | shadedclient | 39m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 34m 19s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 37s | | The patch does not
generate ASF License warnings. |
| | | 194m 23s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.47 ServerAPI=1.47 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6986/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6986 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 966aea133556 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HDFS-17531 / aeecad6efb3deb638059b6cf3e7790f9aec009f5 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6986/2/testReport/ |
| Max. process+thread count | 3497 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6986/2/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.
[jira] [Commented] (HDFS-17594) [ARR] RouterCacheAdmin supports asynchronous rpc.
[
https://issues.apache.org/jira/browse/HDFS-17594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877702#comment-17877702
]
ASF GitHub Bot commented on HDFS-17594:
---
Archie-wang commented on code in PR #6986:
URL: https://github.com/apache/hadoop/pull/6986#discussion_r1736172568
##
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterAsyncCacheAdmin.java:
##
@@ -0,0 +1,75 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hdfs.server.federation.router;
+
+import org.apache.hadoop.fs.CacheFlag;
+import org.apache.hadoop.hdfs.protocol.CacheDirectiveEntry;
+import org.apache.hadoop.hdfs.protocol.CacheDirectiveInfo;
+import org.apache.hadoop.hdfs.protocol.CachePoolEntry;
+import org.apache.hadoop.fs.BatchedRemoteIterator.BatchedEntries;
+import
org.apache.hadoop.hdfs.server.federation.resolver.FederationNamespaceInfo;
+import org.apache.hadoop.hdfs.server.federation.resolver.RemoteLocation;
+import org.apache.hadoop.hdfs.server.federation.router.async.ApplyFunction;
+
+import java.io.IOException;
+import java.util.EnumSet;
+import java.util.Map;
+
+import static
org.apache.hadoop.hdfs.server.federation.router.async.AsyncUtil.asyncApply;
+import static
org.apache.hadoop.hdfs.server.federation.router.async.AsyncUtil.asyncReturn;
+
+/**
+ * Module that implements all the asynchronous RPC calls in
+ * {@link org.apache.hadoop.hdfs.protocol.ClientProtocol} related to Cache
Admin
+ * in the {@link RouterRpcServer}.
+ */
+public class RouterAsyncCacheAdmin extends RouterCacheAdmin{
+
+ public RouterAsyncCacheAdmin(RouterRpcServer server) {
+super(server);
+ }
+
+ @Override
+ public long addCacheDirective(
+ CacheDirectiveInfo path, EnumSet flags) throws IOException {
+invokeAddCacheDirective(path, flags);
+asyncApply((ApplyFunction, Long>)
Review Comment:
done
> [ARR] RouterCacheAdmin supports asynchronous rpc.
> -
>
> Key: HDFS-17594
> URL: https://issues.apache.org/jira/browse/HDFS-17594
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Reporter: Jian Zhang
>Assignee: Jian Zhang
>Priority: Major
> Labels: pull-request-available
>
> *Describe*
> The main new addition is RouterAsyncCacheAdmin, which extends
> RouterCacheAdmin so that cache admin supports asynchronous rpc.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16446) Consider ioutils of disk when choosing volume
[ https://issues.apache.org/jira/browse/HDFS-16446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877571#comment-17877571 ] ASF GitHub Bot commented on HDFS-16446: --- uflswe commented on PR #3960: URL: https://github.com/apache/hadoop/pull/3960#issuecomment-2316657301 @whbing > Hi, @tomscut , we also face slow nodes caused by io, but in most cases it is slow to read data because the frequency of reading in the cluster far exceeds that of writing. We have counted many cases, when reading slowly, the **iowait** is basically high, but only one or a few disks are full of **ioutil** (12 disks per dn). Therefore, I have the following points to discuss.嗨， @tomscut ，我们也遇到了 io 导致的慢节点，但大多数情况下读取数据都很慢，因为集群中读取的频率远远超过写入的频率。我们统计过很多情况，慢读的时候，iowait 基本都很高，但只有一个或者几个磁盘装满了 ioutil（每个 dn 12 个磁盘）。因此，我有以下几点要讨论。 > > 1. Is there any further consideration for slow reading?是否有进一步的考虑来考虑慢读？ > 2. Have you considered reporting iowait to nn, like XmitsInProgress, ActiveTransferThreadCount, etc, so that it is taken into account when choosing a DN?您是否考虑过将 iowait 报告给 nn，如 XmitsInProgress、ActiveTransferThreadCount 等，以便在选择 DN 时将其考虑在内？ The first suggestion is good. But in my opinion, VolumeChoosingPolicy is only relevant during the write operation, determining which disk on a DataNode should store the data block. When reading files from HDFS, the focus is on selecting the appropriate DataNode that contains the required data blocks, not on selecting specific disks within a DataNode. > Consider ioutils of disk when choosing volume > - > > Key: HDFS-16446 > URL: https://issues.apache.org/jira/browse/HDFS-16446 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Tao Li >Assignee: Tao Li >Priority: Major > Labels: pull-request-available > Attachments: image-2022-02-05-09-50-12-241.png > > Time Spent: 3h > Remaining Estimate: 0h > > Consider ioutils of disk when choosing volume to avoid busy disks. > Document: > [https://docs.google.com/document/d/1Ko1J7shz8hVLnNACT6PKVQ_leIHf_YaIFA2s3yJMZHQ/edit?usp=sharing] > Principle is as follows: > !https://user-images.githubusercontent.com/55134131/159827737-f4ca4d66-c2f2-4bef-901b-6d2bc7bdda9a.png|width=440,height=192! -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16084) getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv() failed
[
https://issues.apache.org/jira/browse/HDFS-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877526#comment-17877526
]
ASF GitHub Bot commented on HDFS-16084:
---
steveloughran commented on PR #6969:
URL: https://github.com/apache/hadoop/pull/6969#issuecomment-2316113232
@Hexiaoqiao you want to get into hadoop branch 3.4, so target 3.4.2?
> getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv()
> failed
>
>
> Key: HDFS-16084
> URL: https://issues.apache.org/jira/browse/HDFS-16084
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.2.1
>Reporter: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> First reported in ARROW-13011: when a libhdfs API call fails because
> CLASSPATH isn't set, calling the API a second time leads to a crash.
> *Backtrace*
> This was obtained from the ARROW-13011 reproducer:
> {code:java}
> #0 globalClassReference (className=className@entry=0x7f75883c13b0
> "org/apache/hadoop/conf/Configuration", env=env@entry=0x6c2f2f3a73666468,
> out=out@entry=0x7fffd86e3020) at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:279
> #1 0x7f75883b9511 in constructNewObjectOfClass
> (env=env@entry=0x6c2f2f3a73666468, out=out@entry=0x7fffd86e3148,
> className=className@entry=0x7f75883c13b0
> "org/apache/hadoop/conf/Configuration",
> ctorSignature=ctorSignature@entry=0x7f75883c1180 "()V")
> at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:212
> #2 0x7f75883bb6d0 in hdfsBuilderConnect (bld=0x5562e4bbb3e0) at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:700
> #3 0x7f758de31ef3 in arrow::io::internal::LibHdfsShim::BuilderConnect
> (this=0x7f758e768240 namespace)::libhdfs_shim>,
> bld=0x5562e4bbb3e0) at /arrow/cpp/src/arrow/io/hdfs_internal.cc:366
> #4 0x7f758de2d098 in
> arrow::io::HadoopFileSystem::HadoopFileSystemImpl::Connect
> (this=0x5562e4a9f750, config=0x5562e46edc30)
> at /arrow/cpp/src/arrow/io/hdfs.cc:372
> #5 0x7f758de2e646 in arrow::io::HadoopFileSystem::Connect
> (config=0x5562e46edc30, fs=0x5562e46edd08) at
> /arrow/cpp/src/arrow/io/hdfs.cc:590
> #6 0x7f758d532d2a in arrow::fs::HadoopFileSystem::Impl::Init
> (this=0x5562e46edc30) at /arrow/cpp/src/arrow/filesystem/hdfs.cc:59
> #7 0x7f758d536931 in arrow::fs::HadoopFileSystem::Make (options=...,
> io_context=...) at /arrow/cpp/src/arrow/filesystem/hdfs.cc:409
> #8 0x7f75885d7445 in
> __pyx_pf_7pyarrow_5_hdfs_16HadoopFileSystem___init__
> (__pyx_v_self=0x7f758871a970, __pyx_v_host=0x7f758871cc00, __pyx_v_port=8020,
> __pyx_v_user=0x5562e3af6d30 <_Py_NoneStruct>, __pyx_v_replication=3,
> __pyx_v_buffer_size=0, __pyx_v_default_block_size=0x5562e3af6d30
> <_Py_NoneStruct>,
> __pyx_v_kerb_ticket=0x5562e3af6d30 <_Py_NoneStruct>,
> __pyx_v_extra_conf=0x5562e3af6d30 <_Py_NoneStruct>) at _hdfs.cpp:4759
> #9 0x7f75885d4c88 in
> __pyx_pw_7pyarrow_5_hdfs_16HadoopFileSystem_1__init__
> (__pyx_v_self=0x7f758871a970, __pyx_args=0x7f75900bb048,
> __pyx_kwds=0x7f7590033a68)
> at _hdfs.cpp:4343
> #10 0x5562e38ca747 in type_call () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/typeobject.c:915
> #11 0x5562e39117a3 in _PyObject_FastCallDict (kwargs=,
> nargs=, args=,
> func=0x7f75885f1420 <__pyx_type_7pyarrow_5_hdfs_HadoopFileSystem>) at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/tupleobject.c:76
> #12 _PyObject_FastCallKeywords () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/abstract.c:2496
> #13 0x5562e39121d5 in call_function () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4875
> #14 0x5562e3973d68 in _PyEval_EvalFrameDefault () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:3351
> #15 0x5562e38b98f5 in PyEval_EvalFrameEx (throwflag=0, f=0x7f74c0664768)
> at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4166
> #16 _PyEval_EvalCodeWithName () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4166
> #17 0x5562e38bad79 in PyEval_EvalCodeEx (_co=,
> globals=, locals=, args=,
> argcount=,
> kws=, kwcount=0, defs=0x0, defcount=0, kwdefs=0x0,
> closure=0x0)
> at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4187
> #18 0x5562e398b6eb in PyEval_EvalCode (co=,
>
[jira] [Commented] (HDFS-16084) getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv() failed
[
https://issues.apache.org/jira/browse/HDFS-16084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17877246#comment-17877246
]
ASF GitHub Bot commented on HDFS-16084:
---
kevincai commented on PR #6969:
URL: https://github.com/apache/hadoop/pull/6969#issuecomment-2314310318
> Committed to trunk. Thanks @kevincai for your contributions!
>
> BTW, what's your JIRA account? When I want to assign
https://issues.apache.org/jira/browse/HDFS-16084 to you but I am not sure if
the reporter and you is the same one. Thanks!
I didn't apply one.
> getJNIEnv() returns invalid pointer when called twice after getGlobalJNIEnv()
> failed
>
>
> Key: HDFS-16084
> URL: https://issues.apache.org/jira/browse/HDFS-16084
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: libhdfs
>Affects Versions: 3.2.1
>Reporter: Antoine Pitrou
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> First reported in ARROW-13011: when a libhdfs API call fails because
> CLASSPATH isn't set, calling the API a second time leads to a crash.
> *Backtrace*
> This was obtained from the ARROW-13011 reproducer:
> {code:java}
> #0 globalClassReference (className=className@entry=0x7f75883c13b0
> "org/apache/hadoop/conf/Configuration", env=env@entry=0x6c2f2f3a73666468,
> out=out@entry=0x7fffd86e3020) at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:279
> #1 0x7f75883b9511 in constructNewObjectOfClass
> (env=env@entry=0x6c2f2f3a73666468, out=out@entry=0x7fffd86e3148,
> className=className@entry=0x7f75883c13b0
> "org/apache/hadoop/conf/Configuration",
> ctorSignature=ctorSignature@entry=0x7f75883c1180 "()V")
> at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/jni_helper.c:212
> #2 0x7f75883bb6d0 in hdfsBuilderConnect (bld=0x5562e4bbb3e0) at
> /build/source/hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c:700
> #3 0x7f758de31ef3 in arrow::io::internal::LibHdfsShim::BuilderConnect
> (this=0x7f758e768240 namespace)::libhdfs_shim>,
> bld=0x5562e4bbb3e0) at /arrow/cpp/src/arrow/io/hdfs_internal.cc:366
> #4 0x7f758de2d098 in
> arrow::io::HadoopFileSystem::HadoopFileSystemImpl::Connect
> (this=0x5562e4a9f750, config=0x5562e46edc30)
> at /arrow/cpp/src/arrow/io/hdfs.cc:372
> #5 0x7f758de2e646 in arrow::io::HadoopFileSystem::Connect
> (config=0x5562e46edc30, fs=0x5562e46edd08) at
> /arrow/cpp/src/arrow/io/hdfs.cc:590
> #6 0x7f758d532d2a in arrow::fs::HadoopFileSystem::Impl::Init
> (this=0x5562e46edc30) at /arrow/cpp/src/arrow/filesystem/hdfs.cc:59
> #7 0x7f758d536931 in arrow::fs::HadoopFileSystem::Make (options=...,
> io_context=...) at /arrow/cpp/src/arrow/filesystem/hdfs.cc:409
> #8 0x7f75885d7445 in
> __pyx_pf_7pyarrow_5_hdfs_16HadoopFileSystem___init__
> (__pyx_v_self=0x7f758871a970, __pyx_v_host=0x7f758871cc00, __pyx_v_port=8020,
> __pyx_v_user=0x5562e3af6d30 <_Py_NoneStruct>, __pyx_v_replication=3,
> __pyx_v_buffer_size=0, __pyx_v_default_block_size=0x5562e3af6d30
> <_Py_NoneStruct>,
> __pyx_v_kerb_ticket=0x5562e3af6d30 <_Py_NoneStruct>,
> __pyx_v_extra_conf=0x5562e3af6d30 <_Py_NoneStruct>) at _hdfs.cpp:4759
> #9 0x7f75885d4c88 in
> __pyx_pw_7pyarrow_5_hdfs_16HadoopFileSystem_1__init__
> (__pyx_v_self=0x7f758871a970, __pyx_args=0x7f75900bb048,
> __pyx_kwds=0x7f7590033a68)
> at _hdfs.cpp:4343
> #10 0x5562e38ca747 in type_call () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/typeobject.c:915
> #11 0x5562e39117a3 in _PyObject_FastCallDict (kwargs=,
> nargs=, args=,
> func=0x7f75885f1420 <__pyx_type_7pyarrow_5_hdfs_HadoopFileSystem>) at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/tupleobject.c:76
> #12 _PyObject_FastCallKeywords () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Objects/abstract.c:2496
> #13 0x5562e39121d5 in call_function () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4875
> #14 0x5562e3973d68 in _PyEval_EvalFrameDefault () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:3351
> #15 0x5562e38b98f5 in PyEval_EvalFrameEx (throwflag=0, f=0x7f74c0664768)
> at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4166
> #16 _PyEval_EvalCodeWithName () at
> /home/conda/feedstock_root/build_artifacts/python_1613711361059/work/Python/ceval.c:4166
> #17 0x5562e38bad79 in PyEval_EvalCodeEx (_co=,
> globals=, locals=, args=,
> argcount=,
> kws=, kwcount=0
[jira] [Commented] (HDFS-17545) [ARR] router async rpc client.
[
https://issues.apache.org/jira/browse/HDFS-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876978#comment-17876978
]
ASF GitHub Bot commented on HDFS-17545:
---
hadoop-yetus commented on PR #6871:
URL: https://github.com/apache/hadoop/pull/6871#issuecomment-2312205660
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ HDFS-17531 Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 44s | | HDFS-17531 passed |
| +1 :green_heart: | compile | 0m 30s | | HDFS-17531 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 0m 23s | | HDFS-17531 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 23s | | HDFS-17531 passed |
| +1 :green_heart: | mvnsite | 0m 29s | | HDFS-17531 passed |
| +1 :green_heart: | javadoc | 0m 31s | | HDFS-17531 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 23s | | HDFS-17531 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 56s | | HDFS-17531 passed |
| +1 :green_heart: | shadedclient | 20m 26s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 21s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 20s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 0m 20s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 23s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 17s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 52s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 24s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 27m 16s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 109m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6871/17/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6871 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 487d352023e2 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HDFS-17531 / 0521134a1d4b5610313aeb88cd42235c637d20b4 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6871/17/testReport/ |
| Max. process+thread count | 4098 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6871/17/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apac
[jira] [Commented] (HDFS-17545) [ARR] router async rpc client.
[
https://issues.apache.org/jira/browse/HDFS-17545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876938#comment-17876938
]
ASF GitHub Bot commented on HDFS-17545:
---
hadoop-yetus commented on PR #6871:
URL: https://github.com/apache/hadoop/pull/6871#issuecomment-2311789259
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 6m 40s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 3 new or modified test files. |
_ HDFS-17531 Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 57s | | HDFS-17531 passed |
| +1 :green_heart: | compile | 0m 29s | | HDFS-17531 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 0m 26s | | HDFS-17531 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 0m 21s | | HDFS-17531 passed |
| +1 :green_heart: | mvnsite | 0m 29s | | HDFS-17531 passed |
| +1 :green_heart: | javadoc | 0m 30s | | HDFS-17531 passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 0m 23s | | HDFS-17531 passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 54s | | HDFS-17531 passed |
| +1 :green_heart: | shadedclient | 20m 50s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 22s | | the patch passed |
| +1 :green_heart: | compile | 0m 23s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 0m 23s | | the patch passed |
| +1 :green_heart: | compile | 0m 19s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 0m 19s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 13s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 22s | | the patch passed |
| -1 :x: | javadoc | 0m 19s |
[/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6871/16/artifact/out/results-javadoc-javadoc-hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt)
|
hadoop-hdfs-project_hadoop-hdfs-rbf-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 generated 1 new + 0
unchanged - 0 fixed = 1 total (was 0) |
| +1 :green_heart: | javadoc | 0m 18s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 0m 53s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 19s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 27m 0s | | hadoop-hdfs-rbf in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 27s | | The patch does not
generate ASF License warnings. |
| | | 116m 18s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6871/16/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6871 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets xmllint |
| uname | Linux 1e898889bcc8 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | HDFS-17531 / 023cbaacc34d96480b1b56c0bb64294175371d60 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/jo
[jira] [Commented] (HDFS-17556) Avoid adding block to neededReconstruction repeatedly in decommission
[
https://issues.apache.org/jira/browse/HDFS-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876753#comment-17876753
]
ASF GitHub Bot commented on HDFS-17556:
---
lfxy commented on PR #6896:
URL: https://github.com/apache/hadoop/pull/6896#issuecomment-2310458590
@zhengchenyu We should check (1) -> (2), but not (4) -> (5), because
(4)->(5) is not only used in decommission flow.
> Avoid adding block to neededReconstruction repeatedly in decommission
> -
>
> Key: HDFS-17556
> URL: https://issues.apache.org/jira/browse/HDFS-17556
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.5.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Major
> Labels: pull-request-available
>
> In decommission and maintenance process, before added to
> BlockManager::neededReconstruction block will be check if it has been added.
> The check contains if block is in BlockManager::neededReconstruction or in
> PendingReconstructionBlocks::pendingReconstructions as below code.
> But it also need to check if it is in
> PendingReconstructionBlocks::timedOutItems. Or else
> DatanodeAdminDefaultMonitor will add block to
> BlockManager::neededReconstruction repeatedly if block time out in
> PendingReconstructionBlocks::pendingReconstructions.
>
> {code:java}
> if (!blockManager.neededReconstruction.contains(block) &&
> blockManager.pendingReconstruction.getNumReplicas(block) == 0 &&
> blockManager.isPopulatingReplQueues()) {
> // Process these blocks only when active NN is out of safe mode.
> blockManager.neededReconstruction.add(block,
> liveReplicas, num.readOnlyReplicas(),
> num.outOfServiceReplicas(),
> blockManager.getExpectedRedundancyNum(block));
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17556) Avoid adding block to neededReconstruction repeatedly in decommission
[
https://issues.apache.org/jira/browse/HDFS-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876752#comment-17876752
]
ASF GitHub Bot commented on HDFS-17556:
---
zhengchenyu commented on PR #6896:
URL: https://github.com/apache/hadoop/pull/6896#issuecomment-2310436077
@lfxy
you does not catch me. I said the same thing as you said. My (1) is just
your (1) -> (2), My (2) is just your (4) -> (5).
I suggest checking in (4)->(5).
Although we filter out timed out blocks in (1) -> (2), we still need to add
these block in (4) -> (5). We can do the check in (4) -> (5). Generally
speaking, the verification cost in (4) -> (5) is lower, because there are fewer
blocks to check. And there is no need to change timedOutItems from List to Set.
> Avoid adding block to neededReconstruction repeatedly in decommission
> -
>
> Key: HDFS-17556
> URL: https://issues.apache.org/jira/browse/HDFS-17556
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.5.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Major
> Labels: pull-request-available
>
> In decommission and maintenance process, before added to
> BlockManager::neededReconstruction block will be check if it has been added.
> The check contains if block is in BlockManager::neededReconstruction or in
> PendingReconstructionBlocks::pendingReconstructions as below code.
> But it also need to check if it is in
> PendingReconstructionBlocks::timedOutItems. Or else
> DatanodeAdminDefaultMonitor will add block to
> BlockManager::neededReconstruction repeatedly if block time out in
> PendingReconstructionBlocks::pendingReconstructions.
>
> {code:java}
> if (!blockManager.neededReconstruction.contains(block) &&
> blockManager.pendingReconstruction.getNumReplicas(block) == 0 &&
> blockManager.isPopulatingReplQueues()) {
> // Process these blocks only when active NN is out of safe mode.
> blockManager.neededReconstruction.add(block,
> liveReplicas, num.readOnlyReplicas(),
> num.outOfServiceReplicas(),
> blockManager.getExpectedRedundancyNum(block));
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17556) Avoid adding block to neededReconstruction repeatedly in decommission
[
https://issues.apache.org/jira/browse/HDFS-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876743#comment-17876743
]
ASF GitHub Bot commented on HDFS-17556:
---
lfxy commented on PR #6896:
URL: https://github.com/apache/hadoop/pull/6896#issuecomment-2310388780
@zhengchenyu I think it's wrong you described about pendingReconstruction.
The block flow in decommission is:
(1)DatanodeAdminManager::monitor --> (2)neededReconstruction -->
(3)PendingReconstructionBlocks::pendingReconstructions -->
(4)PendingReconstructionBlocks::timedOutItems--> (5)neededReconstruction
If reconstruction task failed in dn, block will flow from (2) to (5)
circularly. When the block is in timedOutItems and we don't check,
DatanodeAdminManager will add block to neededReconstruction repeated.
> Avoid adding block to neededReconstruction repeatedly in decommission
> -
>
> Key: HDFS-17556
> URL: https://issues.apache.org/jira/browse/HDFS-17556
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.5.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Major
> Labels: pull-request-available
>
> In decommission and maintenance process, before added to
> BlockManager::neededReconstruction block will be check if it has been added.
> The check contains if block is in BlockManager::neededReconstruction or in
> PendingReconstructionBlocks::pendingReconstructions as below code.
> But it also need to check if it is in
> PendingReconstructionBlocks::timedOutItems. Or else
> DatanodeAdminDefaultMonitor will add block to
> BlockManager::neededReconstruction repeatedly if block time out in
> PendingReconstructionBlocks::pendingReconstructions.
>
> {code:java}
> if (!blockManager.neededReconstruction.contains(block) &&
> blockManager.pendingReconstruction.getNumReplicas(block) == 0 &&
> blockManager.isPopulatingReplQueues()) {
> // Process these blocks only when active NN is out of safe mode.
> blockManager.neededReconstruction.add(block,
> liveReplicas, num.readOnlyReplicas(),
> num.outOfServiceReplicas(),
> blockManager.getExpectedRedundancyNum(block));
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16757) Add a new method copyBlockCrossNamespace to DataNode
[ https://issues.apache.org/jira/browse/HDFS-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876722#comment-17876722 ] ASF GitHub Bot commented on HDFS-16757: --- LiuGuH commented on PR #6926: URL: https://github.com/apache/hadoop/pull/6926#issuecomment-2310196805 > Another one, now we don't consider capacity used between different BlockPool when hardlink at DataNode side, right? Will it resolve at next PR? Thanks for reveiw. I will see this problem and try to resolve in last PR. > Add a new method copyBlockCrossNamespace to DataNode > > > Key: HDFS-16757 > URL: https://issues.apache.org/jira/browse/HDFS-16757 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: Haiyang Hu >Priority: Minor > Labels: pull-request-available > > Add a new method copyBlockCrossNamespace in DataTransferProtocol at the > DataNode Side. > This method will copy a source block from one namespace to a target block > from a different namespace. If the target DN is the same with the current DN, > this method will copy the block via HardLink. If the target DN is different > with the current DN, this method will copy the block via TransferBlock. > This method will contains some parameters: > * ExtendedBlock sourceBlock > * Token sourceBlockToken > * ExtendedBlock targetBlock > * Token targetBlockToken > * DatanodeInfo targetDN -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16757) Add a new method copyBlockCrossNamespace to DataNode
[
https://issues.apache.org/jira/browse/HDFS-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876721#comment-17876721
]
ASF GitHub Bot commented on HDFS-16757:
---
LiuGuH commented on code in PR #6926:
URL: https://github.com/apache/hadoop/pull/6926#discussion_r1731228580
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java:
##
@@ -136,6 +136,11 @@ public class DFSConfigKeys extends CommonConfigurationKeys
{
"dfs.datanode.ec.reconstruct.write.bandwidthPerSec";
public static final long
DFS_DATANODE_EC_RECONSTRUCT_WRITE_BANDWIDTHPERSEC_DEFAULT =
0; // A value of zero indicates no limit
+ public static final String
DFS_DATANODE_COPY_BLOCK_CROSS_NAMESPACE_SOCKET_TIMEOUT_MS_KEY =
Review Comment:
This is used for copyBlockCrossNamespaceExecutor task timeout.And it
not , a task will block until it sucess or failed. This can be removed or
retain.
> Add a new method copyBlockCrossNamespace to DataNode
>
>
> Key: HDFS-16757
> URL: https://issues.apache.org/jira/browse/HDFS-16757
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: ZanderXu
>Assignee: Haiyang Hu
>Priority: Minor
> Labels: pull-request-available
>
> Add a new method copyBlockCrossNamespace in DataTransferProtocol at the
> DataNode Side.
> This method will copy a source block from one namespace to a target block
> from a different namespace. If the target DN is the same with the current DN,
> this method will copy the block via HardLink. If the target DN is different
> with the current DN, this method will copy the block via TransferBlock.
> This method will contains some parameters:
> * ExtendedBlock sourceBlock
> * Token sourceBlockToken
> * ExtendedBlock targetBlock
> * Token targetBlockToken
> * DatanodeInfo targetDN
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17556) Avoid adding block to neededReconstruction repeatedly in decommission
[
https://issues.apache.org/jira/browse/HDFS-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876720#comment-17876720
]
ASF GitHub Bot commented on HDFS-17556:
---
zhengchenyu commented on PR #6896:
URL: https://github.com/apache/hadoop/pull/6896#issuecomment-2310187881
I think you may mean that add block to pendingReconstruction in two place is
duplicated.
* (1) add decommission block to pendingReconstruction
* (2) add timed out block to pendingReconstruction
I think in (2), when we add timed out block to pendingReconstruction, we
will use isNeededReconstruction to check. Will it really be repeated?
> Avoid adding block to neededReconstruction repeatedly in decommission
> -
>
> Key: HDFS-17556
> URL: https://issues.apache.org/jira/browse/HDFS-17556
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.5.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Major
> Labels: pull-request-available
>
> In decommission and maintenance process, before added to
> BlockManager::neededReconstruction block will be check if it has been added.
> The check contains if block is in BlockManager::neededReconstruction or in
> PendingReconstructionBlocks::pendingReconstructions as below code.
> But it also need to check if it is in
> PendingReconstructionBlocks::timedOutItems. Or else
> DatanodeAdminDefaultMonitor will add block to
> BlockManager::neededReconstruction repeatedly if block time out in
> PendingReconstructionBlocks::pendingReconstructions.
>
> {code:java}
> if (!blockManager.neededReconstruction.contains(block) &&
> blockManager.pendingReconstruction.getNumReplicas(block) == 0 &&
> blockManager.isPopulatingReplQueues()) {
> // Process these blocks only when active NN is out of safe mode.
> blockManager.neededReconstruction.add(block,
> liveReplicas, num.readOnlyReplicas(),
> num.outOfServiceReplicas(),
> blockManager.getExpectedRedundancyNum(block));
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16757) Add a new method copyBlockCrossNamespace to DataNode
[
https://issues.apache.org/jira/browse/HDFS-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876719#comment-17876719
]
ASF GitHub Bot commented on HDFS-16757:
---
LiuGuH commented on code in PR #6926:
URL: https://github.com/apache/hadoop/pull/6926#discussion_r1731224445
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:
##
@@ -3067,69 +3087,70 @@ public void run() {
//
// Header info
//
-Token accessToken = getBlockAccessToken(b,
-EnumSet.of(BlockTokenIdentifier.AccessMode.WRITE),
-targetStorageTypes, targetStorageIds);
+Token accessToken =
+getBlockAccessToken(target,
EnumSet.of(BlockTokenIdentifier.AccessMode.WRITE),
+targetStorageTypes, targetStorageIds);
-long writeTimeout = dnConf.socketWriteTimeout +
-HdfsConstants.WRITE_TIMEOUT_EXTENSION *
(targets.length-1);
+long writeTimeout =
+dnConf.socketWriteTimeout + HdfsConstants.WRITE_TIMEOUT_EXTENSION
* (targets.length
+- 1);
OutputStream unbufOut = NetUtils.getOutputStream(sock, writeTimeout);
InputStream unbufIn = NetUtils.getInputStream(sock);
-DataEncryptionKeyFactory keyFactory =
- getDataEncryptionKeyFactoryForBlock(b);
-IOStreamPair saslStreams = saslClient.socketSend(sock, unbufOut,
- unbufIn, keyFactory, accessToken, bpReg);
+DataEncryptionKeyFactory keyFactory =
getDataEncryptionKeyFactoryForBlock(source);
+IOStreamPair saslStreams =
+saslClient.socketSend(sock, unbufOut, unbufIn, keyFactory,
accessToken, bpReg);
unbufOut = saslStreams.out;
unbufIn = saslStreams.in;
-
-out = new DataOutputStream(new BufferedOutputStream(unbufOut,
-DFSUtilClient.getSmallBufferSize(getConf(;
+
+out = new DataOutputStream(
+new BufferedOutputStream(unbufOut,
DFSUtilClient.getSmallBufferSize(getConf(;
in = new DataInputStream(unbufIn);
-blockSender = new BlockSender(b, 0, b.getNumBytes(),
-false, false, true, DataNode.this, null, cachingStrategy);
-DatanodeInfo srcNode = new DatanodeInfoBuilder().setNodeID(bpReg)
-.build();
-
-String storageId = targetStorageIds.length > 0 ?
-targetStorageIds[0] : null;
-new Sender(out).writeBlock(b, targetStorageTypes[0], accessToken,
-clientname, targets, targetStorageTypes, srcNode,
-stage, 0, 0, 0, 0, blockSender.getChecksum(), cachingStrategy,
-false, false, null, storageId,
-targetStorageIds);
+blockSender =
+new BlockSender(source, 0, source.getNumBytes(), false, false,
true, DataNode.this,
+null, cachingStrategy);
+DatanodeInfo srcNode = new
DatanodeInfoBuilder().setNodeID(bpReg).build();
+
+String storageId = targetStorageIds.length > 0 ? targetStorageIds[0] :
null;
+new Sender(out).writeBlock(target, targetStorageTypes[0], accessToken,
clientname, targets,
+targetStorageTypes, srcNode, stage, 0, 0, 0, 0,
blockSender.getChecksum(),
+cachingStrategy, false, false, null, storageId, targetStorageIds);
// send data & checksum
blockSender.sendBlock(out, unbufOut, throttler);
// no response necessary
-LOG.info("{}, at {}: Transmitted {} (numBytes={}) to {}",
-getClass().getSimpleName(), DataNode.this.getDisplayName(),
-b, b.getNumBytes(), curTarget);
+LOG.info("{}, at {}: Transmitted {} (numBytes={}) to {}",
getClass().getSimpleName(),
+DataNode.this.getDisplayName(), source, source.getNumBytes(),
curTarget);
// read ack
if (isClient) {
- DNTransferAckProto closeAck = DNTransferAckProto.parseFrom(
- PBHelperClient.vintPrefixed(in));
+ DNTransferAckProto closeAck =
+ DNTransferAckProto.parseFrom(PBHelperClient.vintPrefixed(in));
LOG.debug("{}: close-ack={}", getClass().getSimpleName(), closeAck);
if (closeAck.getStatus() != Status.SUCCESS) {
if (closeAck.getStatus() == Status.ERROR_ACCESS_TOKEN) {
throw new InvalidBlockTokenException(
- "Got access token error for connect ack, targets="
- + Arrays.asList(targets));
+ "Got access token error for connect ack, targets=" +
Arrays.asList(targets));
} else {
- throw new IOException("Bad connect ack, targets="
- + Arrays.asList(targets) + " status=" +
closeAck.getStatus());
+ throw new IOException(
+ "Bad connect ack, targets=" + Arrays.asList(targets) + "
status="
+
[jira] [Commented] (HDFS-16757) Add a new method copyBlockCrossNamespace to DataNode
[
https://issues.apache.org/jira/browse/HDFS-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876716#comment-17876716
]
ASF GitHub Bot commented on HDFS-16757:
---
LiuGuH commented on code in PR #6926:
URL: https://github.com/apache/hadoop/pull/6926#discussion_r1731221302
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:
##
@@ -3003,46 +3010,58 @@ private class DataTransfer implements Runnable {
final DatanodeInfo[] targets;
final StorageType[] targetStorageTypes;
final private String[] targetStorageIds;
-final ExtendedBlock b;
+final private ExtendedBlock source;
+private ExtendedBlock target;
final BlockConstructionStage stage;
final private DatanodeRegistration bpReg;
final String clientname;
final CachingStrategy cachingStrategy;
-/** Throttle to block replication when data transfers or writes. */
+/**
+ * Throttle to block replication when data transfers or writes.
+ */
private DataTransferThrottler throttler;
+private boolean copyBlockCrossNamespace;
Review Comment:
This is used for distinguishing that a DataTransfer task is form NN command
or from fastcopy task.
> Add a new method copyBlockCrossNamespace to DataNode
>
>
> Key: HDFS-16757
> URL: https://issues.apache.org/jira/browse/HDFS-16757
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: ZanderXu
>Assignee: Haiyang Hu
>Priority: Minor
> Labels: pull-request-available
>
> Add a new method copyBlockCrossNamespace in DataTransferProtocol at the
> DataNode Side.
> This method will copy a source block from one namespace to a target block
> from a different namespace. If the target DN is the same with the current DN,
> this method will copy the block via HardLink. If the target DN is different
> with the current DN, this method will copy the block via TransferBlock.
> This method will contains some parameters:
> * ExtendedBlock sourceBlock
> * Token sourceBlockToken
> * ExtendedBlock targetBlock
> * Token targetBlockToken
> * DatanodeInfo targetDN
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16757) Add a new method copyBlockCrossNamespace to DataNode
[
https://issues.apache.org/jira/browse/HDFS-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876715#comment-17876715
]
ASF GitHub Bot commented on HDFS-16757:
---
LiuGuH commented on code in PR #6926:
URL: https://github.com/apache/hadoop/pull/6926#discussion_r1731217325
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/FsDatasetImpl.java:
##
@@ -3863,5 +3864,27 @@ public void setLastDirScannerFinishTime(long time) {
public long getPendingAsyncDeletions() {
return asyncDiskService.countPendingDeletions();
}
+
+ @Override
+ public void hardLinkOneBlock(ExtendedBlock srcBlock, ExtendedBlock dstBlock)
throws IOException {
+BlockLocalPathInfo blpi = getBlockLocalPathInfo(srcBlock);
+FsVolumeImpl v = getVolume(srcBlock);
+
+try (AutoCloseableLock lock = lockManager.writeLock(LockLevel.VOLUME,
dstBlock.getBlockPoolId(),
Review Comment:
Strictly speaking , here we should use src read lock and dst write lock.
And must make sure the order of accquire locks of pools to avoid deadlock.
> Add a new method copyBlockCrossNamespace to DataNode
>
>
> Key: HDFS-16757
> URL: https://issues.apache.org/jira/browse/HDFS-16757
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: ZanderXu
>Assignee: Haiyang Hu
>Priority: Minor
> Labels: pull-request-available
>
> Add a new method copyBlockCrossNamespace in DataTransferProtocol at the
> DataNode Side.
> This method will copy a source block from one namespace to a target block
> from a different namespace. If the target DN is the same with the current DN,
> this method will copy the block via HardLink. If the target DN is different
> with the current DN, this method will copy the block via TransferBlock.
> This method will contains some parameters:
> * ExtendedBlock sourceBlock
> * Token sourceBlockToken
> * ExtendedBlock targetBlock
> * Token targetBlockToken
> * DatanodeInfo targetDN
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-16757) Add a new method copyBlockCrossNamespace to DataNode
[ https://issues.apache.org/jira/browse/HDFS-16757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876713#comment-17876713 ] ASF GitHub Bot commented on HDFS-16757: --- LiuGuH commented on code in PR #6926: URL: https://github.com/apache/hadoop/pull/6926#discussion_r1731210555 ## hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSOutputStream.java: ## @@ -131,6 +131,7 @@ public class DFSOutputStream extends FSOutputSummer private FileEncryptionInfo fileEncryptionInfo; private int writePacketSize; private boolean leaseRecovered = false; + private ExtendedBlock userAssignmentLastBlock; Review Comment: This is used for DFSOutputStream.completeFile() . When fastcop a file from src to dst , and dst is control by outputStream for file lease ,and use addblock() for create block with dst . When completeFile() is invoked, the last block should exits. So userAssignmentLastBlock is called when outputstrem closed. > Add a new method copyBlockCrossNamespace to DataNode > > > Key: HDFS-16757 > URL: https://issues.apache.org/jira/browse/HDFS-16757 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: ZanderXu >Assignee: Haiyang Hu >Priority: Minor > Labels: pull-request-available > > Add a new method copyBlockCrossNamespace in DataTransferProtocol at the > DataNode Side. > This method will copy a source block from one namespace to a target block > from a different namespace. If the target DN is the same with the current DN, > this method will copy the block via HardLink. If the target DN is different > with the current DN, this method will copy the block via TransferBlock. > This method will contains some parameters: > * ExtendedBlock sourceBlock > * Token sourceBlockToken > * ExtendedBlock targetBlock > * Token targetBlockToken > * DatanodeInfo targetDN -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17556) Avoid adding block to neededReconstruction repeatedly in decommission
[
https://issues.apache.org/jira/browse/HDFS-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876649#comment-17876649
]
ASF GitHub Bot commented on HDFS-17556:
---
hadoop-yetus commented on PR #6896:
URL: https://github.com/apache/hadoop/pull/6896#issuecomment-2309791380
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 29s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 46m 10s | | trunk passed |
| +1 :green_heart: | compile | 1m 22s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 1m 16s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 24s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 44s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 16s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 16s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 1m 16s | | the patch passed |
| +1 :green_heart: | compile | 1m 11s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 1m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 58s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 38s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 55s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 223m 4s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 363m 30s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6896 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 1a43d5f27b1f 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / d9edab0f319d8cb0ad01f0746ac905d270bd231c |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/6/testReport/ |
| Max. process+thread count | 3973 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/6/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was automatically generated.
> Avoid adding block to neededReconstruction repeatedly in decommission
[jira] [Commented] (HDFS-17556) Avoid adding block to neededReconstruction repeatedly in decommission
[
https://issues.apache.org/jira/browse/HDFS-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876546#comment-17876546
]
ASF GitHub Bot commented on HDFS-17556:
---
hadoop-yetus commented on PR #6896:
URL: https://github.com/apache/hadoop/pull/6896#issuecomment-2309014541
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 54s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 2 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 44m 53s | | trunk passed |
| +1 :green_heart: | compile | 1m 25s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 1m 10s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 25s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 7s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 43s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 19s | | trunk passed |
| +1 :green_heart: | shadedclient | 36m 3s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 11s | | the patch passed |
| +1 :green_heart: | compile | 1m 15s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 1m 15s | | the patch passed |
| +1 :green_heart: | compile | 1m 8s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 1m 8s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 58s |
[/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/5/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs-project/hadoop-hdfs: The patch generated 1 new + 51 unchanged -
0 fixed = 52 total (was 51) |
| +1 :green_heart: | mvnsite | 1m 13s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 40s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 15s | | the patch passed |
| +1 :green_heart: | shadedclient | 35m 59s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 272m 18s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 47s | | The patch does not
generate ASF License warnings. |
| | | 412m 7s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/5/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6896 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 87a6c1ccca12 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 01c9c841fe2173c2981c1de332b06cc5618a44d8 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6896/5/testReport/ |
| Max. process+thread count | 3103 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job
[jira] [Commented] (HDFS-17607) Reduce the number of times conf is loaded when DataNode startUp
[
https://issues.apache.org/jira/browse/HDFS-17607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876543#comment-17876543
]
ASF GitHub Bot commented on HDFS-17607:
---
hadoop-yetus commented on PR #7012:
URL: https://github.com/apache/hadoop/pull/7012#issuecomment-2309003600
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 18m 27s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 49m 38s | | trunk passed |
| +1 :green_heart: | compile | 1m 28s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | compile | 1m 17s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | checkstyle | 1m 13s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 28s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 11s | | trunk passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 49s | | trunk passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 24s | | trunk passed |
| +1 :green_heart: | shadedclient | 41m 30s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 1m 13s | | the patch passed |
| +1 :green_heart: | compile | 1m 20s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javac | 1m 20s | | the patch passed |
| +1 :green_heart: | compile | 1m 10s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | javac | 1m 10s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 1m 2s | | the patch passed |
| +1 :green_heart: | mvnsite | 1m 15s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 |
| +1 :green_heart: | javadoc | 1m 41s | | the patch passed with JDK
Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| +1 :green_heart: | spotbugs | 3m 23s | | the patch passed |
| +1 :green_heart: | shadedclient | 42m 9s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 262m 6s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 51s | | The patch does not
generate ASF License warnings. |
| | | 436m 36s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.46 ServerAPI=1.46 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7012/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/7012 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 4ae56daf2ddb 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5
20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 4e22cd2bf78f0df0222ab5fff9fe26d58e5f44c5 |
| Default Java | Private Build-1.8.0_422-8u422-b05-1~20.04-b05 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
|
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7012/1/testReport/ |
| Max. process+thread count | 2557 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7012/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
[jira] [Commented] (HDFS-17556) Avoid adding block to neededReconstruction repeatedly in decommission
[
https://issues.apache.org/jira/browse/HDFS-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876520#comment-17876520
]
ASF GitHub Bot commented on HDFS-17556:
---
lfxy commented on PR #6896:
URL: https://github.com/apache/hadoop/pull/6896#issuecomment-2308894828
@haiyang1987 Thank you for your suggestion. I have update the UT.
> Avoid adding block to neededReconstruction repeatedly in decommission
> -
>
> Key: HDFS-17556
> URL: https://issues.apache.org/jira/browse/HDFS-17556
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.5.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Major
> Labels: pull-request-available
>
> In decommission and maintenance process, before added to
> BlockManager::neededReconstruction block will be check if it has been added.
> The check contains if block is in BlockManager::neededReconstruction or in
> PendingReconstructionBlocks::pendingReconstructions as below code.
> But it also need to check if it is in
> PendingReconstructionBlocks::timedOutItems. Or else
> DatanodeAdminDefaultMonitor will add block to
> BlockManager::neededReconstruction repeatedly if block time out in
> PendingReconstructionBlocks::pendingReconstructions.
>
> {code:java}
> if (!blockManager.neededReconstruction.contains(block) &&
> blockManager.pendingReconstruction.getNumReplicas(block) == 0 &&
> blockManager.isPopulatingReplQueues()) {
> // Process these blocks only when active NN is out of safe mode.
> blockManager.neededReconstruction.add(block,
> liveReplicas, num.readOnlyReplicas(),
> num.outOfServiceReplicas(),
> blockManager.getExpectedRedundancyNum(block));
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17556) Avoid adding block to neededReconstruction repeatedly in decommission
[
https://issues.apache.org/jira/browse/HDFS-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876518#comment-17876518
]
ASF GitHub Bot commented on HDFS-17556:
---
haiyang1987 commented on code in PR #6896:
URL: https://github.com/apache/hadoop/pull/6896#discussion_r1730368502
##
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java:
##
@@ -2329,4 +2331,81 @@ public void delayDeleteReplica() {
DataNodeFaultInjector.set(oldInjector);
}
}
+
+ private void waitForDecommissionedNodes(final DatanodeAdminManager
dnAdminMgr,
+ final int trackedNumber)
+ throws TimeoutException, InterruptedException {
+GenericTestUtils
+.waitFor(() -> dnAdminMgr.getNumTrackedNodes() == trackedNumber,
+100, 2000);
+ }
+
+ @Test(timeout = 36)
+ public void testDecommissionWithPendingTimeout() throws Exception {
+Configuration conf = new HdfsConfiguration();
+conf.setBoolean(DFSConfigKeys.DFS_NAMENODE_REDUNDANCY_CONSIDERLOAD_KEY,
false);
+
+HostsFileWriter hostsFileWriter = new HostsFileWriter();
+hostsFileWriter.initialize(conf, "work-dir/decommission");
+conf.setInt(DFSConfigKeys.DFS_NAMENODE_HEARTBEAT_RECHECK_INTERVAL_KEY,
1000);
+conf.setInt(DFSConfigKeys.DFS_HEARTBEAT_INTERVAL_KEY, 1);
+
conf.setInt(DFSConfigKeys.DFS_NAMENODE_RECONSTRUCTION_PENDING_TIMEOUT_SEC_KEY,
2);
+conf.setInt(DFSConfigKeys.DFS_NAMENODE_REDUNDANCY_INTERVAL_SECONDS_KEY,
10);
+conf.setInt(DFSConfigKeys.DFS_NAMENODE_DECOMMISSION_INTERVAL_KEY, 1);
+conf.setLong(DFSConfigKeys.DFS_DATANODE_BALANCE_BANDWIDTHPERSEC_KEY, 1);
+
+MiniDFSCluster cluster = new
MiniDFSCluster.Builder(conf).numDataNodes(2).build();
Review Comment:
From [2357-2380], maybe can update logic
```
MiniDFSCluster cluster = new
MiniDFSCluster.Builder(conf).numDataNodes(2).build();
try {
cluster.waitActive();
BlockManager blockManager = cluster.getNamesystem().getBlockManager();
DatanodeManager dm = blockManager.getDatanodeManager();
dm.setHeartbeatExpireInterval(3000);
DistributedFileSystem fs = cluster.getFileSystem();
// create a file
Path filePath = new Path("/tmp.txt");
DFSTestUtil.createFile(fs, filePath, 1024, (short) 1, 0L);
LocatedBlocks blocks =
NameNodeAdapter.getBlockLocations(cluster.getNameNode(),
filePath.toString(), 0, 1);
LocatedBlock locatedBlock = blocks.get(0);
BlockInfo bi =
blockManager.getStoredBlock(locatedBlock.getBlock().getLocalBlock());
String dnName = locatedBlock.getLocations()[0].getXferAddr();
ArrayList dns = new ArrayList(1);
```
> Avoid adding block to neededReconstruction repeatedly in decommission
> -
>
> Key: HDFS-17556
> URL: https://issues.apache.org/jira/browse/HDFS-17556
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.5.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Major
> Labels: pull-request-available
>
> In decommission and maintenance process, before added to
> BlockManager::neededReconstruction block will be check if it has been added.
> The check contains if block is in BlockManager::neededReconstruction or in
> PendingReconstructionBlocks::pendingReconstructions as below code.
> But it also need to check if it is in
> PendingReconstructionBlocks::timedOutItems. Or else
> DatanodeAdminDefaultMonitor will add block to
> BlockManager::neededReconstruction repeatedly if block time out in
> PendingReconstructionBlocks::pendingReconstructions.
>
> {code:java}
> if (!blockManager.neededReconstruction.contains(block) &&
> blockManager.pendingReconstruction.getNumReplicas(block) == 0 &&
> blockManager.isPopulatingReplQueues()) {
> // Process these blocks only when active NN is out of safe mode.
> blockManager.neededReconstruction.add(block,
> liveReplicas, num.readOnlyReplicas(),
> num.outOfServiceReplicas(),
> blockManager.getExpectedRedundancyNum(block));
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-17607) Reduce the number of times conf is loaded when DataNode startUp
[ https://issues.apache.org/jira/browse/HDFS-17607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDFS-17607: -- Labels: pull-request-available (was: ) > Reduce the number of times conf is loaded when DataNode startUp > --- > > Key: HDFS-17607 > URL: https://issues.apache.org/jira/browse/HDFS-17607 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: lei w >Priority: Major > Labels: pull-request-available > > The value of the conf parameter in the current access method > StorageLocation#makeBlockPoolDir is null, which leads to the problem of > loading conf multiple times when DataNode startUp -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17607) Reduce the number of times conf is loaded when DataNode startUp
[ https://issues.apache.org/jira/browse/HDFS-17607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876517#comment-17876517 ] ASF GitHub Bot commented on HDFS-17607: --- ThinkerLei opened a new pull request, #7012: URL: https://github.com/apache/hadoop/pull/7012 Reduce the number of times conf is loaded when DataNode startUp > Reduce the number of times conf is loaded when DataNode startUp > --- > > Key: HDFS-17607 > URL: https://issues.apache.org/jira/browse/HDFS-17607 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: lei w >Priority: Major > > The value of the conf parameter in the current access method > StorageLocation#makeBlockPoolDir is null, which leads to the problem of > loading conf multiple times when DataNode startUp -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17556) Avoid adding block to neededReconstruction repeatedly in decommission
[
https://issues.apache.org/jira/browse/HDFS-17556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876513#comment-17876513
]
ASF GitHub Bot commented on HDFS-17556:
---
haiyang1987 commented on PR #6896:
URL: https://github.com/apache/hadoop/pull/6896#issuecomment-2308868812
Well work, Leave some comments inline, Thanks~
> Avoid adding block to neededReconstruction repeatedly in decommission
> -
>
> Key: HDFS-17556
> URL: https://issues.apache.org/jira/browse/HDFS-17556
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namanode
>Affects Versions: 3.5.0
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Major
> Labels: pull-request-available
>
> In decommission and maintenance process, before added to
> BlockManager::neededReconstruction block will be check if it has been added.
> The check contains if block is in BlockManager::neededReconstruction or in
> PendingReconstructionBlocks::pendingReconstructions as below code.
> But it also need to check if it is in
> PendingReconstructionBlocks::timedOutItems. Or else
> DatanodeAdminDefaultMonitor will add block to
> BlockManager::neededReconstruction repeatedly if block time out in
> PendingReconstructionBlocks::pendingReconstructions.
>
> {code:java}
> if (!blockManager.neededReconstruction.contains(block) &&
> blockManager.pendingReconstruction.getNumReplicas(block) == 0 &&
> blockManager.isPopulatingReplQueues()) {
> // Process these blocks only when active NN is out of safe mode.
> blockManager.neededReconstruction.add(block,
> liveReplicas, num.readOnlyReplicas(),
> num.outOfServiceReplicas(),
> blockManager.getExpectedRedundancyNum(block));
> } {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17523) Add fine-grained locks metrics in DataSetLockManager
[
https://issues.apache.org/jira/browse/HDFS-17523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876511#comment-17876511
]
ASF GitHub Bot commented on HDFS-17523:
---
ThinkerLei commented on code in PR #6890:
URL: https://github.com/apache/hadoop/pull/6890#discussion_r1730364747
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/AutoCloseDataSetLock.java:
##
@@ -30,17 +33,39 @@
* use a try-with-resource syntax.
*/
public class AutoCloseDataSetLock extends AutoCloseableLock {
- private Lock lock;
+
+ private final Lock lock;
+
+ private final DataNodeLockManager dataNodeLockManager;
+
+ private final boolean isReadLock;
+
+ private static ThreadLocal heldInfoThreadLocal = new
Review Comment:
> Sorry for the delayed response. I've been quite busy recently. Here's the
detailed situation:
>
> In certain versions of JDK 17, there is an issue where weak references to
ThreadLocal cannot be reclaimed(There is a problem with G1, but CMS can reclaim
memory normally). This results in ThreadLocal objects accumulating a large
amount of data, causing significant CPU overhead during traversal and element
retrieval. We initially noticed this issue during latency profiling related to
Ozone OM locks, where CPU usage for our OM spiked over a period of time. Hence,
I suggested that enabling lock profiling should ideally be an optional feature.
>
> OpenJDK 17 lower version bug: https://bugs.openjdk.org/browse/JDK-8188055.
>
> We need to make sure that the JDK17 version we choose has the above patch.
First of all, I'm very sorry that I haven't been able to reply to you
recently because I've been busy with work. Secondly, thank you very much for
your suggestion. I will modify this function to be optional later. @slfan1989
> Add fine-grained locks metrics in DataSetLockManager
> -
>
> Key: HDFS-17523
> URL: https://issues.apache.org/jira/browse/HDFS-17523
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: lei w
>Priority: Major
> Labels: pull-request-available
>
> Currently we use fine-grained locks to manage FsDataSetImpl. But we did not
> add lock-related metrics. In some cases, we actually need lock-holding
> information to understand the time-consuming lock-holding of a certain
> operation. Using this information, we can also optimize some long-term lock
> operations as early as possible.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-17599) EC: Fix the mismatch between locations and indices for mover
[
https://issues.apache.org/jira/browse/HDFS-17599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17876493#comment-17876493
]
ASF GitHub Bot commented on HDFS-17599:
---
haiyang1987 commented on PR #6980:
URL: https://github.com/apache/hadoop/pull/6980#issuecomment-2308760305
Will commit if no more other comments while wait two workday.
> EC: Fix the mismatch between locations and indices for mover
>
>
> Key: HDFS-17599
> URL: https://issues.apache.org/jira/browse/HDFS-17599
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.0, 3.4.0
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2024-08-03-17-59-08-059.png,
> image-2024-08-03-18-00-01-950.png
>
>
> We set the EC policy to (6+3) and also have nodes that were in state
> ENTERING_MAINTENANCE.
>
> When we move the data of some directories from SSD to HDD, some blocks move
> fail due to disk full, as shown in the figure below
> (blk_-9223372033441574269).
> We tried to move again and found the following error "{color:#ff}Replica
> does not exist{color}".
> Observing the information of fsck, it can be found that the wrong
> blockid(blk_-9223372033441574270) was found when moving block.
>
> {*}Mover Logs{*}:
> !image-2024-08-03-17-59-08-059.png|width=741,height=85!
>
> {*}FSCK Info{*}:
> !image-2024-08-03-18-00-01-950.png|width=738,height=120!
>
> {*}Root Cause{*}:
> Similar to this HDFS-16333, when mover is initialized, only the `LIVE` node
> is processed. As a result, the datanode in the `ENTERING_MAINTENANCE` state
> in the locations is filtered when initializing `DBlockStriped`, but the
> indices are not adapted, resulting in a mismatch between the location and
> indices lengths. Finally, ec block calculates the wrong blockid when getting
> internal block (see `DBlockStriped#getInternalBlock`).
>
> We added debug logs, and a few key messages are shown below.
> {color:#ff}The result is an incorrect correspondence: xx.xx.7.31 ->
> -9223372033441574270{color}.
> {code:java}
> DBlock getInternalBlock(StorageGroup storage) {
> // storage == xx.xx.7.31
> // idxInLocs == 1 (location ([xx.xx.,85.29:DISK, xx.xx.7.31:DISK,
> xx.xx.207.22:DISK, xx.xx.8.25:DISK, xx.xx.79.30:DISK, xx.xx.87.21:DISK,
> xx.xx.8.38:DISK]), xx.xx.179.31 is in the ENTERING_MAINTENANCE state is
> filtered)
> int idxInLocs = locations.indexOf(storage);
> if (idxInLocs == -1) {
> return null;
> }
> // idxInGroup == 2 (indices is [1,2,3,4,5,6,7,8])
> byte idxInGroup = indices[idxInLocs];
> // blkId: -9223372033441574272 + 2 = -9223372033441574270
> long blkId = getBlock().getBlockId() + idxInGroup;
> long numBytes = getInternalBlockLength(getNumBytes(), cellSize,
> dataBlockNum, idxInGroup);
> Block blk = new Block(getBlock());
> blk.setBlockId(blkId);
> blk.setNumBytes(numBytes);
> DBlock dblk = new DBlock(blk);
> dblk.addLocation(storage);
> return dblk;
> } {code}
> {*}Solution{*}:
> When initializing DBlockStriped, if any location is filtered out, we need to
> remove the corresponding element in the indices to do the adaptation.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
