KevinWikant commented on pull request #3675: URL: https://github.com/apache/hadoop/pull/3675#issuecomment-980107774
> If a DN goes dead and then re-registers, it will be added back into the pending nodes, so I don't think we need to continue to track it in the decommission monitor when it goes dead. We can just handle the dead event in the decommission monitor and stop tracking it, clearing a slot for another node. Then it will be re-added if it comes back by existing logic above. After going through this re-implementation of the logic, I think I found a caveat worth mentioning Note the implementation of "isNodeHealthyForDecommissionOrMaintenance": - https://github.com/apache/hadoop/blob/dc751df63b4ab2c9c26a1efe7479c31fd1de80d5/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L4546 - if HDFS has no lowRedundancy or pendingReconstruction blocks then all datanodes (including dead datanodes) are considered "healthy" for the purpose of being DECOMMISSIONED Here's the happy case: - a dead datanode is removed from DatanodeAdminManager (i.e. removed from both outOfServiceNodeBlocks & pendingNodes) with AdminState=DECOMMISSION_INPROGRESS - a dead datanode comes back alive, gets re-added to DatanodeAdminManager, & transitions to AdminState=DECOMMISSIONED Here's a less happy case: - a dead datanode is removed from DatanodeAdminManager with AdminState=DECOMMISSION_INPROGRESS - the dead datanode never comes back alive & therefore remains with AdminState=DECOMMISSION_INPROGRESS forever The problem is that dead datanodes can be eventually DECOMMISSIONED if the namenode eliminates all lowRedundancy blocks (because of "isNodeHealthyForDecommissionOrMaintenance" logic), but by removing the datanode from the DatanodeAdminManager entirely (until it comes back alive) we prevent the dead datanode from transitioning to DECOMMISSIONED even though it could have So the impact is that a dead datanode which never comes back alive again will never transition from DECOMMISSION_INPROGRESS to DECOMMISSIONED even though it could have -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
