[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17830312#comment-17830312
]
ASF GitHub Bot commented on HDFS-17430:
---
haiyang1987 commented on PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#issuecomment-2017094511
Thanks @ZanderXu @Hexiaoqiao @dineshchitlangia for your review and merge~
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle failures.
> DatanodeInfo[] recoveryInfos;
> if (recoveryLocations.size() > 1) {
>if (recoveryLocations.size() != storages.length) {
> LOG.info("Skipped stale nodes for recovery : "
> + (storages.length - recoveryLocations.size()));
>}
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
> } else {
>// If too many replicas are stale, then choose all replicas to
>// participate in block recovery.
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
> }
> {code}
> {code:java}
> 2024-03-13 21:58:01,425 INFO datanode.DataNode
> (BlockRecoveryWorker.java:logRecoverBlock(563))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> BlockRecoveryWorker: NameNode at xxx:8040 calls
> recoverBlock(BP-xxx:blk_xxx_xxx,
> targets=[DatanodeInfoWithStorage[dn1:50010,null,null],
> DatanodeInfoWithStorage[dn2:50010,null,null]],
> newGenerationStamp=28577373754, newBlock=null, isStriped=false)
> {code}
> *t4.* When dn2 executes BlockRecoveryWorker#recover, it will call
> initReplicaRecovery operation on dn1, however, since the dn1 machine is
> currently down state at this time, it will take a very long time to timeout,
> the default number of retries to establish a server connection is 45 times.
> related logs：
> {code:java}
> 2024-03-13 21:59:31,518 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn1:8010. Already tried 0 time(s); maxRetries=45
> ...
> 2024-03-13 23:05:35,295 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn2:8010. Already tried 44 time(s); maxRetries=45
> 2024-03-13 23:07:05,392 WARN protocol.InterDatanodeProtocol
> (BlockRecoveryWorker.java:recover(170))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Failed to recover block (block=BP-xxx:blk_xxx_xxx,
> datanode=DatanodeInfoWithStorage[dn1:50010,null,null])
> org.apache.hadoop.net.ConnectTimeoutException:
> Call From dn2 to dn1:8010 failed on socket timeout exception:
> org.apache.hadoop.net.ConnectTimeoutException: 9 millis timeout while
> waiting for channel to be ready for connect.ch :
> java.nio.channels.SocketChannel[connection-pending remote=dn:8010]; For more
> details see:
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829884#comment-17829884
]
Dinesh Chitlangia commented on HDFS-17430:
--
Thanks [~haiyang Hu] for the improvement and [~Zander Huang] for the reviews
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.5.0
>
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle failures.
> DatanodeInfo[] recoveryInfos;
> if (recoveryLocations.size() > 1) {
>if (recoveryLocations.size() != storages.length) {
> LOG.info("Skipped stale nodes for recovery : "
> + (storages.length - recoveryLocations.size()));
>}
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
> } else {
>// If too many replicas are stale, then choose all replicas to
>// participate in block recovery.
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
> }
> {code}
> {code:java}
> 2024-03-13 21:58:01,425 INFO datanode.DataNode
> (BlockRecoveryWorker.java:logRecoverBlock(563))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> BlockRecoveryWorker: NameNode at xxx:8040 calls
> recoverBlock(BP-xxx:blk_xxx_xxx,
> targets=[DatanodeInfoWithStorage[dn1:50010,null,null],
> DatanodeInfoWithStorage[dn2:50010,null,null]],
> newGenerationStamp=28577373754, newBlock=null, isStriped=false)
> {code}
> *t4.* When dn2 executes BlockRecoveryWorker#recover, it will call
> initReplicaRecovery operation on dn1, however, since the dn1 machine is
> currently down state at this time, it will take a very long time to timeout,
> the default number of retries to establish a server connection is 45 times.
> related logs：
> {code:java}
> 2024-03-13 21:59:31,518 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn1:8010. Already tried 0 time(s); maxRetries=45
> ...
> 2024-03-13 23:05:35,295 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn2:8010. Already tried 44 time(s); maxRetries=45
> 2024-03-13 23:07:05,392 WARN protocol.InterDatanodeProtocol
> (BlockRecoveryWorker.java:recover(170))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Failed to recover block (block=BP-xxx:blk_xxx_xxx,
> datanode=DatanodeInfoWithStorage[dn1:50010,null,null])
> org.apache.hadoop.net.ConnectTimeoutException:
> Call From dn2 to dn1:8010 failed on socket timeout exception:
> org.apache.hadoop.net.ConnectTimeoutException: 9 millis timeout while
> waiting for channel to be ready for connect.ch :
> java.nio.channels.SocketChannel[connection-pending remote=dn:8010]; For more
> details see: http://wiki.apache.org/hadoop/SocketTimeout
> at
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829883#comment-17829883
]
ASF GitHub Bot commented on HDFS-17430:
---
dineshchitlangia commented on PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#issuecomment-2015146422
I am having trouble with my JIRA account, once that is resolved I will mark
the jira as resolved.
Thank you.
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle failures.
> DatanodeInfo[] recoveryInfos;
> if (recoveryLocations.size() > 1) {
>if (recoveryLocations.size() != storages.length) {
> LOG.info("Skipped stale nodes for recovery : "
> + (storages.length - recoveryLocations.size()));
>}
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
> } else {
>// If too many replicas are stale, then choose all replicas to
>// participate in block recovery.
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
> }
> {code}
> {code:java}
> 2024-03-13 21:58:01,425 INFO datanode.DataNode
> (BlockRecoveryWorker.java:logRecoverBlock(563))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> BlockRecoveryWorker: NameNode at xxx:8040 calls
> recoverBlock(BP-xxx:blk_xxx_xxx,
> targets=[DatanodeInfoWithStorage[dn1:50010,null,null],
> DatanodeInfoWithStorage[dn2:50010,null,null]],
> newGenerationStamp=28577373754, newBlock=null, isStriped=false)
> {code}
> *t4.* When dn2 executes BlockRecoveryWorker#recover, it will call
> initReplicaRecovery operation on dn1, however, since the dn1 machine is
> currently down state at this time, it will take a very long time to timeout,
> the default number of retries to establish a server connection is 45 times.
> related logs：
> {code:java}
> 2024-03-13 21:59:31,518 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn1:8010. Already tried 0 time(s); maxRetries=45
> ...
> 2024-03-13 23:05:35,295 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn2:8010. Already tried 44 time(s); maxRetries=45
> 2024-03-13 23:07:05,392 WARN protocol.InterDatanodeProtocol
> (BlockRecoveryWorker.java:recover(170))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Failed to recover block (block=BP-xxx:blk_xxx_xxx,
> datanode=DatanodeInfoWithStorage[dn1:50010,null,null])
> org.apache.hadoop.net.ConnectTimeoutException:
> Call From dn2 to dn1:8010 failed on socket timeout exception:
> org.apache.hadoop.net.ConnectTimeoutException: 9 millis timeout while
> waiting for channel to be ready for connect.ch :
> java.nio.channels.SocketChannel[connection-pending remote=dn:8010]; For more
>
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829881#comment-17829881
]
ASF GitHub Bot commented on HDFS-17430:
---
dineshchitlangia merged PR #6635:
URL: https://github.com/apache/hadoop/pull/6635
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle failures.
> DatanodeInfo[] recoveryInfos;
> if (recoveryLocations.size() > 1) {
>if (recoveryLocations.size() != storages.length) {
> LOG.info("Skipped stale nodes for recovery : "
> + (storages.length - recoveryLocations.size()));
>}
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
> } else {
>// If too many replicas are stale, then choose all replicas to
>// participate in block recovery.
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
> }
> {code}
> {code:java}
> 2024-03-13 21:58:01,425 INFO datanode.DataNode
> (BlockRecoveryWorker.java:logRecoverBlock(563))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> BlockRecoveryWorker: NameNode at xxx:8040 calls
> recoverBlock(BP-xxx:blk_xxx_xxx,
> targets=[DatanodeInfoWithStorage[dn1:50010,null,null],
> DatanodeInfoWithStorage[dn2:50010,null,null]],
> newGenerationStamp=28577373754, newBlock=null, isStriped=false)
> {code}
> *t4.* When dn2 executes BlockRecoveryWorker#recover, it will call
> initReplicaRecovery operation on dn1, however, since the dn1 machine is
> currently down state at this time, it will take a very long time to timeout,
> the default number of retries to establish a server connection is 45 times.
> related logs：
> {code:java}
> 2024-03-13 21:59:31,518 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn1:8010. Already tried 0 time(s); maxRetries=45
> ...
> 2024-03-13 23:05:35,295 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn2:8010. Already tried 44 time(s); maxRetries=45
> 2024-03-13 23:07:05,392 WARN protocol.InterDatanodeProtocol
> (BlockRecoveryWorker.java:recover(170))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Failed to recover block (block=BP-xxx:blk_xxx_xxx,
> datanode=DatanodeInfoWithStorage[dn1:50010,null,null])
> org.apache.hadoop.net.ConnectTimeoutException:
> Call From dn2 to dn1:8010 failed on socket timeout exception:
> org.apache.hadoop.net.ConnectTimeoutException: 9 millis timeout while
> waiting for channel to be ready for connect.ch :
> java.nio.channels.SocketChannel[connection-pending remote=dn:8010]; For more
> details see: http://wiki.apache.org/hadoop/SocketTimeout
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829788#comment-17829788
]
ASF GitHub Bot commented on HDFS-17430:
---
hadoop-yetus commented on PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#issuecomment-2014620711
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 19s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 32m 40s | | trunk passed |
| +1 :green_heart: | compile | 0m 43s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 42s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 42s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 4s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 57s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 21m 10s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | compile | 0m 33s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 33s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 30s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 39s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 34s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 204m 34s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6635/4/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 30s | | The patch does not
generate ASF License warnings. |
| | | 293m 42s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6635/4/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6635 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 64cc061bf5ee 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 9709f8ab5519875e54c1c7a3f27530cc5bc955be |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829554#comment-17829554
]
ASF GitHub Bot commented on HDFS-17430:
---
hadoop-yetus commented on PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#issuecomment-2012266961
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 20s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 1s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 1s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 33m 0s | | trunk passed |
| +1 :green_heart: | compile | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 37s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | checkstyle | 0m 36s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 42s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 40s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 5s | | trunk passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 49s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 27s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 23m 40s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 42s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 39s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | javac | 0m 39s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 30s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 38s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 32s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 4s | | the patch passed with JDK
Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| +1 :green_heart: | spotbugs | 1m 42s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 53s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 202m 55s | | hadoop-hdfs in the patch
passed. |
| +1 :green_heart: | asflicense | 0m 31s | | The patch does not
generate ASF License warnings. |
| | | 295m 8s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.45 ServerAPI=1.45 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6635/3/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6635 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 10d0020a22a4 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c21a656da06274270a468d51b6320b63144ff9c2 |
| Default Java | Private Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private
Build-1.8.0_402-8u402-ga-2ubuntu1~20.04-b06 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6635/3/testReport/ |
| Max. process+thread count | 4129 (vs. ulimit of 5500) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829422#comment-17829422
]
ASF GitHub Bot commented on HDFS-17430:
---
haiyang1987 commented on PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#issuecomment-2011611948
Update PR.
Hi @ZanderXu @Hexiaoqiao @dineshchitlangia please help me review this PR
again when you have free time, Thank you very much.
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle failures.
> DatanodeInfo[] recoveryInfos;
> if (recoveryLocations.size() > 1) {
>if (recoveryLocations.size() != storages.length) {
> LOG.info("Skipped stale nodes for recovery : "
> + (storages.length - recoveryLocations.size()));
>}
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
> } else {
>// If too many replicas are stale, then choose all replicas to
>// participate in block recovery.
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
> }
> {code}
> {code:java}
> 2024-03-13 21:58:01,425 INFO datanode.DataNode
> (BlockRecoveryWorker.java:logRecoverBlock(563))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> BlockRecoveryWorker: NameNode at xxx:8040 calls
> recoverBlock(BP-xxx:blk_xxx_xxx,
> targets=[DatanodeInfoWithStorage[dn1:50010,null,null],
> DatanodeInfoWithStorage[dn2:50010,null,null]],
> newGenerationStamp=28577373754, newBlock=null, isStriped=false)
> {code}
> *t4.* When dn2 executes BlockRecoveryWorker#recover, it will call
> initReplicaRecovery operation on dn1, however, since the dn1 machine is
> currently down state at this time, it will take a very long time to timeout,
> the default number of retries to establish a server connection is 45 times.
> related logs：
> {code:java}
> 2024-03-13 21:59:31,518 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn1:8010. Already tried 0 time(s); maxRetries=45
> ...
> 2024-03-13 23:05:35,295 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn2:8010. Already tried 44 time(s); maxRetries=45
> 2024-03-13 23:07:05,392 WARN protocol.InterDatanodeProtocol
> (BlockRecoveryWorker.java:recover(170))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Failed to recover block (block=BP-xxx:blk_xxx_xxx,
> datanode=DatanodeInfoWithStorage[dn1:50010,null,null])
> org.apache.hadoop.net.ConnectTimeoutException:
> Call From dn2 to dn1:8010 failed on socket timeout exception:
> org.apache.hadoop.net.ConnectTimeoutException: 9 millis timeout while
> waiting for channel to be ready for connect.ch :
> java.nio.channels.SocketChannel[connection-pending
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829364#comment-17829364
]
ASF GitHub Bot commented on HDFS-17430:
---
haiyang1987 commented on code in PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#discussion_r1533200066
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##
@@ -1755,12 +1755,24 @@ private BlockRecoveryCommand
getBlockRecoveryCommand(String blockPoolId,
LOG.info("Skipped stale nodes for recovery : "
+ (storages.length - recoveryLocations.size()));
}
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
} else {
-// If too many replicas are stale, then choose all replicas to
+// If too many replicas are stale, then choose live replicas to
// participate in block recovery.
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
+recoveryLocations.clear();
+storageIdx.clear();
+for (int i = 0; i < storages.length; ++i) {
+ if (storages[i].getDatanodeDescriptor().isAlive()) {
+recoveryLocations.add(storages[i]);
+storageIdx.add(i);
+ }
+}
+assert recoveryLocations.size() > 0 : "recoveryLocations size should
be > 0";
Review Comment:
Check the code again. when processing handleHeartbeat executes
getBlockRecoveryCommand, the datanode should be in the live state at this
time, so the size of recoveryLocations is at least 1.
so here maybey remove this assert logic.
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle failures.
> DatanodeInfo[] recoveryInfos;
> if (recoveryLocations.size() > 1) {
>if (recoveryLocations.size() != storages.length) {
> LOG.info("Skipped stale nodes for recovery : "
> + (storages.length - recoveryLocations.size()));
>}
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
> } else {
>// If too many replicas are stale, then choose all replicas to
>// participate in block recovery.
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
> }
> {code}
> {code:java}
> 2024-03-13 21:58:01,425 INFO datanode.DataNode
> (BlockRecoveryWorker.java:logRecoverBlock(563))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> BlockRecoveryWorker: NameNode at xxx:8040 calls
> recoverBlock(BP-xxx:blk_xxx_xxx,
> targets=[DatanodeInfoWithStorage[dn1:50010,null,null],
> DatanodeInfoWithStorage[dn2:50010,null,null]],
> newGenerationStamp=28577373754, newBlock=null, isStriped=false)
> {code}
> *t4.* When dn2 executes BlockRecoveryWorker#recover, it will call
> initReplicaRecovery operation on dn1, however, since the dn1 machine is
> currently down state at this time, it will take a very long
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17829363#comment-17829363
]
ASF GitHub Bot commented on HDFS-17430:
---
haiyang1987 commented on code in PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#discussion_r1533195923
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##
@@ -1755,12 +1755,24 @@ private BlockRecoveryCommand
getBlockRecoveryCommand(String blockPoolId,
LOG.info("Skipped stale nodes for recovery : "
+ (storages.length - recoveryLocations.size()));
}
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
} else {
-// If too many replicas are stale, then choose all replicas to
+// If too many replicas are stale, then choose live replicas to
// participate in block recovery.
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
+recoveryLocations.clear();
+storageIdx.clear();
+for (int i = 0; i < storages.length; ++i) {
+ if (storages[i].getDatanodeDescriptor().isAlive()) {
Review Comment:
Thanks @Hexiaoqiao for you comment.
Sir suggestion is here only will choose non stale and is live replicas to
participate in block recovery ?
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle failures.
> DatanodeInfo[] recoveryInfos;
> if (recoveryLocations.size() > 1) {
>if (recoveryLocations.size() != storages.length) {
> LOG.info("Skipped stale nodes for recovery : "
> + (storages.length - recoveryLocations.size()));
>}
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
> } else {
>// If too many replicas are stale, then choose all replicas to
>// participate in block recovery.
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
> }
> {code}
> {code:java}
> 2024-03-13 21:58:01,425 INFO datanode.DataNode
> (BlockRecoveryWorker.java:logRecoverBlock(563))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> BlockRecoveryWorker: NameNode at xxx:8040 calls
> recoverBlock(BP-xxx:blk_xxx_xxx,
> targets=[DatanodeInfoWithStorage[dn1:50010,null,null],
> DatanodeInfoWithStorage[dn2:50010,null,null]],
> newGenerationStamp=28577373754, newBlock=null, isStriped=false)
> {code}
> *t4.* When dn2 executes BlockRecoveryWorker#recover, it will call
> initReplicaRecovery operation on dn1, however, since the dn1 machine is
> currently down state at this time, it will take a very long time to timeout,
> the default number of retries to establish a server connection is 45 times.
> related logs：
> {code:java}
> 2024-03-13 21:59:31,518 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
>
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828912#comment-17828912
]
ASF GitHub Bot commented on HDFS-17430:
---
Hexiaoqiao commented on code in PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#discussion_r1531824148
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##
@@ -1755,12 +1755,24 @@ private BlockRecoveryCommand
getBlockRecoveryCommand(String blockPoolId,
LOG.info("Skipped stale nodes for recovery : "
+ (storages.length - recoveryLocations.size()));
}
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
} else {
-// If too many replicas are stale, then choose all replicas to
+// If too many replicas are stale, then choose live replicas to
// participate in block recovery.
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
+recoveryLocations.clear();
+storageIdx.clear();
+for (int i = 0; i < storages.length; ++i) {
+ if (storages[i].getDatanodeDescriptor().isAlive()) {
Review Comment:
What about add this condition to L1736~L1740 together?
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##
@@ -1755,12 +1755,24 @@ private BlockRecoveryCommand
getBlockRecoveryCommand(String blockPoolId,
LOG.info("Skipped stale nodes for recovery : "
+ (storages.length - recoveryLocations.size()));
}
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
} else {
-// If too many replicas are stale, then choose all replicas to
+// If too many replicas are stale, then choose live replicas to
// participate in block recovery.
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
+recoveryLocations.clear();
+storageIdx.clear();
+for (int i = 0; i < storages.length; ++i) {
+ if (storages[i].getDatanodeDescriptor().isAlive()) {
+recoveryLocations.add(storages[i]);
+storageIdx.add(i);
+ }
+}
+assert recoveryLocations.size() > 0 : "recoveryLocations size should
be > 0";
Review Comment:
Is this assert necessary here, or `recoveryLocations` size could be 0 if all
DataNodes are not alive?
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle failures.
> DatanodeInfo[] recoveryInfos;
> if (recoveryLocations.size() > 1) {
>if (recoveryLocations.size() != storages.length) {
> LOG.info("Skipped stale nodes for recovery : "
> + (storages.length - recoveryLocations.size()));
>}
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
> } else {
>// If too many
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828589#comment-17828589
]
ASF GitHub Bot commented on HDFS-17430:
---
dineshchitlangia commented on PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#issuecomment-2008619571
@ZanderXu as you had posted the first set of suggestions, could you confirm
if your suggestions are addressed? We can merge once we have your +1
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle failures.
> DatanodeInfo[] recoveryInfos;
> if (recoveryLocations.size() > 1) {
>if (recoveryLocations.size() != storages.length) {
> LOG.info("Skipped stale nodes for recovery : "
> + (storages.length - recoveryLocations.size()));
>}
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
> } else {
>// If too many replicas are stale, then choose all replicas to
>// participate in block recovery.
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
> }
> {code}
> {code:java}
> 2024-03-13 21:58:01,425 INFO datanode.DataNode
> (BlockRecoveryWorker.java:logRecoverBlock(563))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> BlockRecoveryWorker: NameNode at xxx:8040 calls
> recoverBlock(BP-xxx:blk_xxx_xxx,
> targets=[DatanodeInfoWithStorage[dn1:50010,null,null],
> DatanodeInfoWithStorage[dn2:50010,null,null]],
> newGenerationStamp=28577373754, newBlock=null, isStriped=false)
> {code}
> *t4.* When dn2 executes BlockRecoveryWorker#recover, it will call
> initReplicaRecovery operation on dn1, however, since the dn1 machine is
> currently down state at this time, it will take a very long time to timeout,
> the default number of retries to establish a server connection is 45 times.
> related logs：
> {code:java}
> 2024-03-13 21:59:31,518 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn1:8010. Already tried 0 time(s); maxRetries=45
> ...
> 2024-03-13 23:05:35,295 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn2:8010. Already tried 44 time(s); maxRetries=45
> 2024-03-13 23:07:05,392 WARN protocol.InterDatanodeProtocol
> (BlockRecoveryWorker.java:recover(170))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Failed to recover block (block=BP-xxx:blk_xxx_xxx,
> datanode=DatanodeInfoWithStorage[dn1:50010,null,null])
> org.apache.hadoop.net.ConnectTimeoutException:
> Call From dn2 to dn1:8010 failed on socket timeout exception:
> org.apache.hadoop.net.ConnectTimeoutException: 9 millis timeout while
> waiting for channel to be ready for connect.ch :
>
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828586#comment-17828586
]
ASF GitHub Bot commented on HDFS-17430:
---
haiyang1987 commented on PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#issuecomment-2008615230
Hi Sir @dineshchitlangia @ayushtkn Would you mind help review this PR when
you have free time? Thank you so much.
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle failures.
> DatanodeInfo[] recoveryInfos;
> if (recoveryLocations.size() > 1) {
>if (recoveryLocations.size() != storages.length) {
> LOG.info("Skipped stale nodes for recovery : "
> + (storages.length - recoveryLocations.size()));
>}
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
> } else {
>// If too many replicas are stale, then choose all replicas to
>// participate in block recovery.
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
> }
> {code}
> {code:java}
> 2024-03-13 21:58:01,425 INFO datanode.DataNode
> (BlockRecoveryWorker.java:logRecoverBlock(563))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> BlockRecoveryWorker: NameNode at xxx:8040 calls
> recoverBlock(BP-xxx:blk_xxx_xxx,
> targets=[DatanodeInfoWithStorage[dn1:50010,null,null],
> DatanodeInfoWithStorage[dn2:50010,null,null]],
> newGenerationStamp=28577373754, newBlock=null, isStriped=false)
> {code}
> *t4.* When dn2 executes BlockRecoveryWorker#recover, it will call
> initReplicaRecovery operation on dn1, however, since the dn1 machine is
> currently down state at this time, it will take a very long time to timeout,
> the default number of retries to establish a server connection is 45 times.
> related logs：
> {code:java}
> 2024-03-13 21:59:31,518 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn1:8010. Already tried 0 time(s); maxRetries=45
> ...
> 2024-03-13 23:05:35,295 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn2:8010. Already tried 44 time(s); maxRetries=45
> 2024-03-13 23:07:05,392 WARN protocol.InterDatanodeProtocol
> (BlockRecoveryWorker.java:recover(170))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Failed to recover block (block=BP-xxx:blk_xxx_xxx,
> datanode=DatanodeInfoWithStorage[dn1:50010,null,null])
> org.apache.hadoop.net.ConnectTimeoutException:
> Call From dn2 to dn1:8010 failed on socket timeout exception:
> org.apache.hadoop.net.ConnectTimeoutException: 9 millis timeout while
> waiting for channel to be ready for connect.ch :
> java.nio.channels.SocketChannel[connection-pending remote=dn:8010]; For more
>
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828198#comment-17828198
]
ASF GitHub Bot commented on HDFS-17430:
---
haiyang1987 commented on PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#issuecomment-2005993261
The failed unit test seems unrelated to the change.
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle failures.
> DatanodeInfo[] recoveryInfos;
> if (recoveryLocations.size() > 1) {
>if (recoveryLocations.size() != storages.length) {
> LOG.info("Skipped stale nodes for recovery : "
> + (storages.length - recoveryLocations.size()));
>}
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
> } else {
>// If too many replicas are stale, then choose all replicas to
>// participate in block recovery.
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
> }
> {code}
> {code:java}
> 2024-03-13 21:58:01,425 INFO datanode.DataNode
> (BlockRecoveryWorker.java:logRecoverBlock(563))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> BlockRecoveryWorker: NameNode at xxx:8040 calls
> recoverBlock(BP-xxx:blk_xxx_xxx,
> targets=[DatanodeInfoWithStorage[dn1:50010,null,null],
> DatanodeInfoWithStorage[dn2:50010,null,null]],
> newGenerationStamp=28577373754, newBlock=null, isStriped=false)
> {code}
> *t4.* When dn2 executes BlockRecoveryWorker#recover, it will call
> initReplicaRecovery operation on dn1, however, since the dn1 machine is
> currently down state at this time, it will take a very long time to timeout,
> the default number of retries to establish a server connection is 45 times.
> related logs：
> {code:java}
> 2024-03-13 21:59:31,518 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn1:8010. Already tried 0 time(s); maxRetries=45
> ...
> 2024-03-13 23:05:35,295 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn2:8010. Already tried 44 time(s); maxRetries=45
> 2024-03-13 23:07:05,392 WARN protocol.InterDatanodeProtocol
> (BlockRecoveryWorker.java:recover(170))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Failed to recover block (block=BP-xxx:blk_xxx_xxx,
> datanode=DatanodeInfoWithStorage[dn1:50010,null,null])
> org.apache.hadoop.net.ConnectTimeoutException:
> Call From dn2 to dn1:8010 failed on socket timeout exception:
> org.apache.hadoop.net.ConnectTimeoutException: 9 millis timeout while
> waiting for channel to be ready for connect.ch :
> java.nio.channels.SocketChannel[connection-pending remote=dn:8010]; For more
> details see: http://wiki.apache.org/hadoop/SocketTimeout
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828054#comment-17828054
]
ASF GitHub Bot commented on HDFS-17430:
---
hadoop-yetus commented on PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#issuecomment-2004489045
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 24s | | trunk passed |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 48s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 37s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 47s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 42s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 9s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 23m 8s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 38s | | the patch passed |
| +1 :green_heart: | compile | 0m 40s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 40s | | the patch passed |
| +1 :green_heart: | compile | 0m 36s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 36s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 29s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 41s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 34s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 1s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 44s | | the patch passed |
| +1 :green_heart: | shadedclient | 24m 23s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 213m 20s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6635/2/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 27s | | The patch does not
generate ASF License warnings. |
| | | 312m 40s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.tools.TestDFSAdmin |
| | hadoop.metrics2.sink.TestRollingFileSystemSinkWithHdfs |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6635/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6635 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux f3a291500e15 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 6376ed89e382dae4276aeeff3f7dba2def8ead7d |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17828025#comment-17828025
]
ASF GitHub Bot commented on HDFS-17430:
---
hadoop-yetus commented on PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#issuecomment-2004253021
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 22s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ trunk Compile Tests _ |
| -1 :x: | mvninstall | 32m 6s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6635/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 0m 45s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | compile | 0m 41s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 39s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 44s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 44s | | trunk passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 1m 6s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 41s | | trunk passed |
| +1 :green_heart: | shadedclient | 20m 35s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 34s | | the patch passed |
| +1 :green_heart: | compile | 0m 37s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javac | 0m 37s | | the patch passed |
| +1 :green_heart: | compile | 0m 32s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 32s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 27s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 36s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 31s | | the patch passed with JDK
Ubuntu-11.0.22+7-post-Ubuntu-0ubuntu220.04.1 |
| +1 :green_heart: | javadoc | 0m 58s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 1m 43s | | the patch passed |
| +1 :green_heart: | shadedclient | 20m 40s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 217m 23s |
[/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6635/1/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt)
| hadoop-hdfs in the patch passed. |
| +1 :green_heart: | asflicense | 0m 28s | | The patch does not
generate ASF License warnings. |
| | | 304m 59s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests | hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.hdfs.TestReconstructStripedFileWithValidator |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.TestLeaseRecovery2 |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.TestReadStripedFileWithDNFailure |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailureWithRandomECPolicy |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6635/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6635 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 0a365a237f30 5.15.0-94-generic #104-Ubuntu SMP Tue Jan 9
15:25:40 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux |
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827944#comment-17827944
]
ASF GitHub Bot commented on HDFS-17430:
---
haiyang1987 commented on PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#issuecomment-2003727035
Update PR.
Hi @ZanderXu @Hexiaoqiao @tasanuma @zhangshuyan0 please help me review this
PR when you have free time, Thank you very much.
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle failures.
> DatanodeInfo[] recoveryInfos;
> if (recoveryLocations.size() > 1) {
>if (recoveryLocations.size() != storages.length) {
> LOG.info("Skipped stale nodes for recovery : "
> + (storages.length - recoveryLocations.size()));
>}
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
> } else {
>// If too many replicas are stale, then choose all replicas to
>// participate in block recovery.
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
> }
> {code}
> {code:java}
> 2024-03-13 21:58:01,425 INFO datanode.DataNode
> (BlockRecoveryWorker.java:logRecoverBlock(563))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> BlockRecoveryWorker: NameNode at xxx:8040 calls
> recoverBlock(BP-xxx:blk_xxx_xxx,
> targets=[DatanodeInfoWithStorage[dn1:50010,null,null],
> DatanodeInfoWithStorage[dn2:50010,null,null]],
> newGenerationStamp=28577373754, newBlock=null, isStriped=false)
> {code}
> *t4.* When dn2 executes BlockRecoveryWorker#recover, it will call
> initReplicaRecovery operation on dn1, however, since the dn1 machine is
> currently down state at this time, it will take a very long time to timeout,
> the default number of retries to establish a server connection is 45 times.
> related logs：
> {code:java}
> 2024-03-13 21:59:31,518 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn1:8010. Already tried 0 time(s); maxRetries=45
> ...
> 2024-03-13 23:05:35,295 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying connect to server: dn2:8010. Already tried 44 time(s); maxRetries=45
> 2024-03-13 23:07:05,392 WARN protocol.InterDatanodeProtocol
> (BlockRecoveryWorker.java:recover(170))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Failed to recover block (block=BP-xxx:blk_xxx_xxx,
> datanode=DatanodeInfoWithStorage[dn1:50010,null,null])
> org.apache.hadoop.net.ConnectTimeoutException:
> Call From dn2 to dn1:8010 failed on socket timeout exception:
> org.apache.hadoop.net.ConnectTimeoutException: 9 millis timeout while
> waiting for channel to be ready for connect.ch :
> java.nio.channels.SocketChannel[connection-pending
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827941#comment-17827941
]
ASF GitHub Bot commented on HDFS-17430:
---
haiyang1987 commented on code in PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#discussion_r1528394415
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java:
##
@@ -628,7 +628,7 @@ public void run() {
new RecoveryTaskContiguous(b).recover();
}
} catch (IOException e) {
- LOG.warn("recover Block: {} FAILED: {}", b, e);
+ LOG.warn("recover Block: {} FAILED: ", b, e);
Review Comment:
yeah, I will submit a new issue to fix this small problem.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##
@@ -1755,12 +1755,19 @@ private BlockRecoveryCommand
getBlockRecoveryCommand(String blockPoolId,
LOG.info("Skipped stale nodes for recovery : "
+ (storages.length - recoveryLocations.size()));
}
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
} else {
-// If too many replicas are stale, then choose all replicas to
+// If too many replicas are stale, then choose live replicas to
// participate in block recovery.
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
+recoveryLocations.clear();
+storageIdx.clear();
+for (int i = 0; i < storages.length; ++i) {
+ if (storages[i].getDatanodeDescriptor().isAlive()) {
+recoveryLocations.add(storages[i]);
+storageIdx.add(i);
+ }
Review Comment:
get it
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle failures.
> DatanodeInfo[] recoveryInfos;
> if (recoveryLocations.size() > 1) {
>if (recoveryLocations.size() != storages.length) {
> LOG.info("Skipped stale nodes for recovery : "
> + (storages.length - recoveryLocations.size()));
>}
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
> } else {
>// If too many replicas are stale, then choose all replicas to
>// participate in block recovery.
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
> }
> {code}
> {code:java}
> 2024-03-13 21:58:01,425 INFO datanode.DataNode
> (BlockRecoveryWorker.java:logRecoverBlock(563))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> BlockRecoveryWorker: NameNode at xxx:8040 calls
> recoverBlock(BP-xxx:blk_xxx_xxx,
> targets=[DatanodeInfoWithStorage[dn1:50010,null,null],
> DatanodeInfoWithStorage[dn2:50010,null,null]],
> newGenerationStamp=28577373754, newBlock=null, isStriped=false)
> {code}
> *t4.* When dn2 executes
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827939#comment-17827939
]
ASF GitHub Bot commented on HDFS-17430:
---
haiyang1987 commented on code in PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#discussion_r1528393272
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##
@@ -1755,12 +1755,19 @@ private BlockRecoveryCommand
getBlockRecoveryCommand(String blockPoolId,
LOG.info("Skipped stale nodes for recovery : "
+ (storages.length - recoveryLocations.size()));
}
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
} else {
-// If too many replicas are stale, then choose all replicas to
+// If too many replicas are stale, then choose live replicas to
// participate in block recovery.
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
+recoveryLocations.clear();
+storageIdx.clear();
+for (int i = 0; i < storages.length; ++i) {
+ if (storages[i].getDatanodeDescriptor().isAlive()) {
+recoveryLocations.add(storages[i]);
+storageIdx.add(i);
+ }
Review Comment:
yeah, i will add it later.
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle failures.
> DatanodeInfo[] recoveryInfos;
> if (recoveryLocations.size() > 1) {
>if (recoveryLocations.size() != storages.length) {
> LOG.info("Skipped stale nodes for recovery : "
> + (storages.length - recoveryLocations.size()));
>}
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
> } else {
>// If too many replicas are stale, then choose all replicas to
>// participate in block recovery.
>recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
> }
> {code}
> {code:java}
> 2024-03-13 21:58:01,425 INFO datanode.DataNode
> (BlockRecoveryWorker.java:logRecoverBlock(563))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> BlockRecoveryWorker: NameNode at xxx:8040 calls
> recoverBlock(BP-xxx:blk_xxx_xxx,
> targets=[DatanodeInfoWithStorage[dn1:50010,null,null],
> DatanodeInfoWithStorage[dn2:50010,null,null]],
> newGenerationStamp=28577373754, newBlock=null, isStriped=false)
> {code}
> *t4.* When dn2 executes BlockRecoveryWorker#recover, it will call
> initReplicaRecovery operation on dn1, however, since the dn1 machine is
> currently down state at this time, it will take a very long time to timeout,
> the default number of retries to establish a server connection is 45 times.
> related logs：
> {code:java}
> 2024-03-13 21:59:31,518 INFO ipc.Client
> (Client.java:handleConnectionTimeout(904))
> [org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
> Retrying
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827917#comment-17827917
]
ASF GitHub Bot commented on HDFS-17430:
---
ZanderXu commented on code in PR #6635:
URL: https://github.com/apache/hadoop/pull/6635#discussion_r1528310021
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/BlockRecoveryWorker.java:
##
@@ -628,7 +628,7 @@ public void run() {
new RecoveryTaskContiguous(b).recover();
}
} catch (IOException e) {
- LOG.warn("recover Block: {} FAILED: {}", b, e);
+ LOG.warn("recover Block: {} FAILED: ", b, e);
Review Comment:
this modification has nothing to do with this issue, right?
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##
@@ -1755,12 +1755,19 @@ private BlockRecoveryCommand
getBlockRecoveryCommand(String blockPoolId,
LOG.info("Skipped stale nodes for recovery : "
+ (storages.length - recoveryLocations.size()));
}
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
} else {
-// If too many replicas are stale, then choose all replicas to
+// If too many replicas are stale, then choose live replicas to
// participate in block recovery.
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
+recoveryLocations.clear();
+storageIdx.clear();
+for (int i = 0; i < storages.length; ++i) {
+ if (storages[i].getDatanodeDescriptor().isAlive()) {
+recoveryLocations.add(storages[i]);
+storageIdx.add(i);
+ }
Review Comment:
please add some logs for this case.
##
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java:
##
@@ -1755,12 +1755,19 @@ private BlockRecoveryCommand
getBlockRecoveryCommand(String blockPoolId,
LOG.info("Skipped stale nodes for recovery : "
+ (storages.length - recoveryLocations.size()));
}
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
} else {
-// If too many replicas are stale, then choose all replicas to
+// If too many replicas are stale, then choose live replicas to
// participate in block recovery.
-recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
+recoveryLocations.clear();
+storageIdx.clear();
+for (int i = 0; i < storages.length; ++i) {
+ if (storages[i].getDatanodeDescriptor().isAlive()) {
+recoveryLocations.add(storages[i]);
+storageIdx.add(i);
+ }
Review Comment:
please check if all replicas are dead.
> RecoveringBlock will skip no live replicas when get block recovery command.
> ---
>
> Key: HDFS-17430
> URL: https://issues.apache.org/jira/browse/HDFS-17430
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haiyang Hu
>Assignee: Haiyang Hu
>Priority: Major
> Labels: pull-request-available
>
> RecoveringBlock maybe skip no live replicas when get block recovery command.
> *Issue:*
> Currently the following scenarios may lead to failure in the execution of
> BlockRecoveryWorker by the datanode, resulting file being not to be closed
> for a long time.
> *t1.* The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut down
> and will be dead status, the dn2 is live status.
> *t2.* Occurs block recovery.
> related logs：
> {code:java}
> 2024-03-13 21:58:00.651 WARN hdfs.StateChange DIR*
> NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
> recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
> {code}
> *t3.* The dn2 is chosen for block recovery.
> dn1 is marked as stale (is dead state) at this time, here the
> recoveryLocations size is 1, currently according to the following logic, dn1
> and dn2 will be chosen to participate in block recovery.
> DatanodeManager#getBlockRecoveryCommand
> {code:java}
>// Skip stale nodes during recovery
> final List recoveryLocations =
> new ArrayList<>(storages.length);
> final List storageIdx = new ArrayList<>(storages.length);
> for (int i = 0; i < storages.length; ++i) {
>if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
> recoveryLocations.add(storages[i]);
> storageIdx.add(i);
>}
> }
> ...
> // If we only get 1 replica after eliminating stale nodes, choose all
> // replicas for recovery and let the primary data node handle
[jira] [Commented] (HDFS-17430) RecoveringBlock will skip no live replicas when get block recovery command.
[
https://issues.apache.org/jira/browse/HDFS-17430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17827911#comment-17827911
]
ASF GitHub Bot commented on HDFS-17430:
---
haiyang1987 opened a new pull request, #6635:
URL: https://github.com/apache/hadoop/pull/6635
### Description of PR
https://issues.apache.org/jira/browse/HDFS-17430
RecoveringBlock maybe skip no live replicas when get block recovery command.
**Issue:**
Currently the following scenarios may lead to failure in the execution of
BlockRecoveryWorker by the datanode, resulting file being not to be closed for
a long time.
**t1.** The block_xxx_xxx has two replicas[dn1,dn2]; the dn1 machine shut
down and will be dead status, the dn2 is live status.
**t2.** Occurs block recovery.
related logs：
```
2024-03-13 21:58:00.651 WARN hdfs.StateChangeDIR*
NameSystem.internalReleaseLease: File /xxx/file has not been closed. Lease
recovery is in progress. RecoveryId = 28577373754 for block blk_xxx_xxx
```
**t3.** The dn2 is chosen for block recovery.
dn1 is marked as stale (is dead state) at this time, here the
recoveryLocations size is 1, currently according to the following logic, dn1
and dn2 will be chosen to participate in block recovery.
DatanodeManager#getBlockRecoveryCommand
```
// Skip stale nodes during recovery
final List recoveryLocations =
new ArrayList<>(storages.length);
final List storageIdx = new ArrayList<>(storages.length);
for (int i = 0; i < storages.length; ++i) {
if (!storages[i].getDatanodeDescriptor().isStale(staleInterval)) {
recoveryLocations.add(storages[i]);
storageIdx.add(i);
}
}
...
// If we only get 1 replica after eliminating stale nodes, choose all
// replicas for recovery and let the primary data node handle failures.
DatanodeInfo[] recoveryInfos;
if (recoveryLocations.size() > 1) {
if (recoveryLocations.size() != storages.length) {
LOG.info("Skipped stale nodes for recovery : "
+ (storages.length - recoveryLocations.size()));
}
recoveryInfos =
DatanodeStorageInfo.toDatanodeInfos(recoveryLocations);
} else {
// If too many replicas are stale, then choose all replicas to
// participate in block recovery.
recoveryInfos = DatanodeStorageInfo.toDatanodeInfos(storages);
}
```
```
2024-03-13 21:58:01,425 INFO datanode.DataNode
(BlockRecoveryWorker.java:logRecoverBlock(563))
[org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
BlockRecoveryWorker: NameNode at xxx:8040 calls
recoverBlock(BP-xxx:blk_xxx_xxx,
targets=[DatanodeInfoWithStorage[dn1:50010,null,null],
DatanodeInfoWithStorage[dn2:50010,null,null]], newGenerationStamp=28577373754,
newBlock=null, isStriped=false)
```
**t4.** When dn2 executes BlockRecoveryWorker#recover, it will call
initReplicaRecovery operation on dn1, however, since the dn1 machine is
currently down state at this time, it will take a very long time to timeout,
the default number of retries to establish a server connection is 45 times.
related logs：
```
2024-03-13 21:59:31,518 INFO ipc.Client
(Client.java:handleConnectionTimeout(904))
[org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
Retrying connect to server: dn1:8010. Already tried 0 time(s); maxRetries=45
...
2024-03-13 23:05:35,295 INFO ipc.Client
(Client.java:handleConnectionTimeout(904))
[org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
Retrying connect to server: dn2:8010. Already tried 44 time(s); maxRetries=45
2024-03-13 23:07:05,392 WARN protocol.InterDatanodeProtocol
(BlockRecoveryWorker.java:recover(170))
[org.apache.hadoop.hdfs.server.datanode.BlockRecoveryWorker$1@54e291ac] -
Failed to recover block (block=BP-xxx:blk_xxx_xxx,
datanode=DatanodeInfoWithStorage[dn1:50010,null,null])
org.apache.hadoop.net.ConnectTimeoutException:
Call From dn2 to dn1:8010 failed on socket timeout exception:
org.apache.hadoop.net.ConnectTimeoutException: 9 millis timeout while
waiting for channel to be ready for connect.ch :
java.nio.channels.SocketChannel[connection-pending remote=dn:8010]; For more
details see: http://wiki.apache.org/hadoop/SocketTimeout
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:931)
