[ https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16842303#comment-16842303 ]
Kihwal Lee commented on HDFS-14186: ----------------------------------- We probably want an ability to tell whether a block report has been received for each and every known storage. If the safe block count is reached AND the nn received all block reports that are supposed to come, it will do only the necessary amount of replication when coming out of safe mode. This is different from waiting for all replica locations for all blocks, which will not happen if any node is down. A similar concept is there as "staleness" check after a HA transition. Standby safe mode can optionally get the list of nodes from the active, if running, and do this check to determine whether it is truly ready to take over. > blockreport storm slow down namenode restart seriously in large cluster > ----------------------------------------------------------------------- > > Key: HDFS-14186 > URL: https://issues.apache.org/jira/browse/HDFS-14186 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode > Affects Versions: 2.7.1 > Reporter: He Xiaoqiao > Assignee: He Xiaoqiao > Priority: Major > Attachments: HDFS-14186.001.patch > > > In the current implementation, the datanode sends blockreport immediately > after register to namenode successfully when restart, and the blockreport > storm will make namenode high load to process them. One result is some > received RPC have to skip because queue time is timeout. If some datanodes' > heartbeat RPC are continually skipped for long times (default is > heartbeatExpireInterval=630s) it will be set DEAD, then datanode has to > re-register and send blockreport again, aggravate blockreport storm and trap > in a vicious circle, and slow down (more than one hour and even more) > namenode startup seriously in a large (several thousands of datanodes) and > busy cluster especially. Although there are many work to optimize namenode > startup, the issue still exists. > I propose to postpone dead datanode check when namenode have finished startup. > Any comments and suggestions are welcome. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org