[
https://issues.apache.org/jira/browse/HDFS-10453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16354711#comment-16354711
]
Erik Krogen edited comment on HDFS-10453 at 2/6/18 11:33 PM:
-------------------------------------------------------------
Hey [~arpitagarwal], IIUC it doesn't need to see the most recent value to fix
this issue. The problem comes when:
1. Block is added to ReplicationWork under lock
2. Block is deleted
3. Block length is read by ReplicationWork as NO_ACK
4. chooseTargets attempts to place a block of size Long.MAX_VALUE; this causes
issue because there is no valid placement, so it takes a long time for the
chooseTargets loop to terminate
If the original block length is read rather than the most recent value, the
issue discussed here does not occur:
1. Block is added to ReplicationWork under lock
2. Block is deleted
3. Block length is read by ReplicationWork as a normal length
4. chooseTargets successfully finds some new locations; then the {{bc == null}}
check properly removes the block from {{neededReplications}}
However, going through this makes me realize, a more simple fix may be to just
fetch and save {{blocksize}} within the constructor for ReplicationWork rather
than calling {{block.getNumBytes()}} within {{chooseTargets()}}. This ensures
consistency, so there should no longer be any ReplicationWorks which consider a
block size of NO_ACK. It avoids the "hacky" nature of checking size equality
with NO_ACK as discussed by Chen. Similar to what was done for the race
condition in HDFS-12832
was (Author: xkrogen):
Hey [~arpitagarwal], IIUC it doesn't need to see the most recent value to fix
this issue. The problem comes when:
1. Block is added to ReplicationWork under lock
2. Block is deleted
3. Block length is read by ReplicationWork as NO_ACK
4. chooseTargets attempts to place a block of size Long.MAX_VALUE; this causes
issue because there is no valid placement, so it takes a long time for the
chooseTargets loop to terminate
If the original block length is read rather than the most recent value, the
issue discussed here does not occur:
1. Block is added to ReplicationWork under lock
2. Block is deleted
3. Block length is read by ReplicationWork as a normal length
4. chooseTargets successfully finds some new locations; then the {{bc == null}}
check properly removes the block from {{neededReplications}}
However, going through this makes me realize, a more simple fix may be to just
fetch and save {{blocksize}} within the constructor for ReplicationWork rather
than calling {{block.getNumBytes()}} within {{chooseTargets()}}. This ensures
consistency, so there should no longer be any ReplicationWorks which consider a
block size of NO_ACK. It avoids the "hacky" nature of checking size equality
with NO_ACK as discussed by Chen.
> ReplicationMonitor thread could stuck for long time due to the race between
> replication and delete of same file in a large cluster.
> -----------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDFS-10453
> URL: https://issues.apache.org/jira/browse/HDFS-10453
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.4.1, 2.5.2, 2.7.1, 2.6.4
> Reporter: He Xiaoqiao
> Assignee: He Xiaoqiao
> Priority: Major
> Fix For: 2.7.6
>
> Attachments: HDFS-10453-branch-2.001.patch,
> HDFS-10453-branch-2.003.patch, HDFS-10453-branch-2.7.004.patch,
> HDFS-10453-branch-2.7.005.patch, HDFS-10453-branch-2.7.006.patch,
> HDFS-10453-branch-2.7.007.patch, HDFS-10453.001.patch
>
>
> ReplicationMonitor thread could stuck for long time and loss data with little
> probability. Consider the typical scenarioļ¼
> (1) create and close a file with the default replicas(3);
> (2) increase replication (to 10) of the file.
> (3) delete the file while ReplicationMonitor is scheduling blocks belong to
> that file for replications.
> if ReplicationMonitor stuck reappeared, NameNode will print log as:
> {code:xml}
> 2016-04-19 10:20:48,083 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 7 to reach 10
> (unavailableStorages=[], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
> newBlock=false) For more information, please enable DEBUG log level on
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
> ......
> 2016-04-19 10:21:17,184 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 7 to reach 10
> (unavailableStorages=[DISK], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
> newBlock=false) For more information, please enable DEBUG log level on
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy
> 2016-04-19 10:21:17,184 WARN
> org.apache.hadoop.hdfs.protocol.BlockStoragePolicy: Failed to place enough
> replicas: expected size is 7 but only 0 storage types can be selected
> (replication=10, selected=[], unavailable=[DISK, ARCHIVE], removed=[DISK,
> DISK, DISK, DISK, DISK, DISK, DISK], policy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]})
> 2016-04-19 10:21:17,184 WARN
> org.apache.hadoop.hdfs.server.blockmanagement.BlockPlacementPolicy: Failed to
> place enough replicas, still in need of 7 to reach 10
> (unavailableStorages=[DISK, ARCHIVE], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]},
> newBlock=false) All required storage types are unavailable:
> unavailableStorages=[DISK, ARCHIVE], storagePolicy=BlockStoragePolicy{HOT:7,
> storageTypes=[DISK], creationFallbacks=[], replicationFallbacks=[ARCHIVE]}
> {code}
> This is because 2 threads (#NameNodeRpcServer and #ReplicationMonitor)
> process same block at the same moment.
> (1) ReplicationMonitor#computeReplicationWorkForBlocks get blocks to
> replicate and leave the global lock.
> (2) FSNamesystem#delete invoked to delete blocks then clear the reference in
> blocksmap, needReplications, etc. the block's NumBytes will set
> NO_ACK(Long.MAX_VALUE) which is used to indicate that the block deletion does
> not need explicit ACK from the node.
> (3) ReplicationMonitor#computeReplicationWorkForBlocks continue to
> chooseTargets for the same blocks and no node will be selected after traverse
> whole cluster because no node choice satisfy the goodness criteria
> (remaining spaces achieve required size Long.MAX_VALUE).
> During of stage#3 ReplicationMonitor stuck for long time, especial in a large
> cluster. invalidateBlocks & neededReplications continues to grow and no
> consumes. it will loss data at the worst.
> This can mostly be avoided by skip chooseTarget for BlockCommand.NO_ACK block
> and remove it from neededReplications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
