zhengchenyu opened a new pull request, #6915: URL: https://github.com/apache/hadoop/pull/6915
### Description of PR https://issues.apache.org/jira/browse/HDFS-17542 The current reconstruction process of EC blocks is based on the original contiguous blocks. It is mainly implemented through the work constructed by computeReconstructionWorkForBlocks. It can be roughly divided into three processes: * scheduleReconstruction * chooseTargets * validateReconstructionWork For ordinary contiguous blocks: * (1) scheduleReconstruction Select srcNodes as the source of the copy block according to the status of each replica of the block. * (2) chooseTargets Select the target of the copy. * (3) validateReconstructionWork Add the copy command to srcNode, srcNode receives the command through heartbeat, and executes the block copy from src to target. For EC blocks: (1) and (2) seems nearly same. However, whether to perform simple block copy or block reconstruction for EC blocks is determined in (3). And when some storage is busy, may result no work, it will lead to the problem described in [HDFS-17516](https://issues.apache.org/jira/browse/HDFS-17516). Even if no block copying or block reconstruction is generated, pendingReconstruction and neededReconstruction will still be updated until the block times out, which wastes the scheduling opportunity. Because the decision of whether to perform block copy or block reconstruction is made in (3), unnecessary liveBusyBlockIndices, and excludeReconstructedIndices are introduced. We know many bugs are related here. These should be avoided. ### How was this patch tested? unit test and test in cluster ### For code changes: - Move the work of deciding whether to copy or reconstruct blocks from validateReconstructionWork to scheduleReconstruction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
