[ 
https://issues.apache.org/jira/browse/HDFS-14849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16933506#comment-16933506
 ] 

Fei Hui commented on HDFS-14849:
--------------------------------

[~marvelrock] Thanks
I deep into the code and find the workflow
# RedundancyMonitor -> computeDatanodeWork -> computeBlockReconstructionWork -> 
get blocks from *neededReconstruction* -> 
computeReconstructionWorkForBlocks(put block to pendingReconstruction in 
validateReconstructionWork) -> scheduleReconstruction -> ErasureCodingWork
# RedundancyMonitor -> processPendingReconstructions -> get timedOutItems from 
pendingReconstruction ->  add it to neededReconstruction 
I think it will replicate more blocks without your fix. After all of blocks 
from decommissioning nodes replicating successfully, the block will not add to 
neededReconstruction because it will recompute live replicas in 
processPendingReconstructions function.  Replicating block infinitely  does not 
happen, is it right ? Just some blocks are more than one?

> Erasure Coding: replicate block infinitely when datanode being decommissioning
> ------------------------------------------------------------------------------
>
>                 Key: HDFS-14849
>                 URL: https://issues.apache.org/jira/browse/HDFS-14849
>             Project: Hadoop HDFS
>          Issue Type: Bug
>    Affects Versions: 3.3.0
>            Reporter: HuangTao
>            Assignee: HuangTao
>            Priority: Major
>              Labels: EC, HDFS, NameNode
>         Attachments: HDFS-14849.001.patch, HDFS-14849.002.patch, 
> fsck-file.png, liveBlockIndices.png, scheduleReconstruction.png
>
>
> When the datanode keeping in DECOMMISSION_INPROGRESS status, the EC block in 
> that datanode will be replicated infinitely.
> // added 2019/09/19
> I reproduced this scenario in a 163 nodes cluster with decommission 100 nodes 
> simultaneously. 
>  !scheduleReconstruction.png! 
>  !fsck-file.png! 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org

Reply via email to