zhangshuyan0 opened a new pull request, #6192: URL: https://github.com/apache/hadoop/pull/6192
### Description of PR **Bug description** If, 1. There is a striped block blockinfo1, which has an excess replica on datanodeA. 2. blockinfo1 has an internal block that needs to be reconstruction. 3. The number of racks is less than the number of internal blocks of blockinfo1. Then, NN may choose datanodeA to reconstruct the internal block, resulting in two internal blocks of blockinfo1 on datanodeA, causing confusion. **Root cause and solution** When we use `BlockPlacementPolicyRackFaultTolerant` for choosing targets and the racks is insufficient, `chooseEvenlyFromRemainingRacks` will be called. Currently, `chooseEvenlyFromRemainingRacks` calls `chooseOnce`, `chooseOnce` use `newExcludeNodes` as parameter instead of `excludedNodes`. When we choose targets for reconstructing internal blocks, 'newExcludeNodes' only includes datanodes that contain live replicas, and does not include datanodes that have excess replicas. This may result in datanodes with excess replicas are chosen. https://github.com/apache/hadoop/blob/c8abca30045258668867eeab89fa05fe88fe5be6/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockPlacementPolicyRackFaultTolerant.java#L181-L189 I think we do not need to use `newExcludeNodes`, just pass `excludedNodes` to `chooseOnce`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
