[
https://issues.apache.org/jira/browse/HDFS-14186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16743281#comment-16743281
]
Íñigo Goiri commented on HDFS-14186:
------------------------------------
{quote}
As description above, if service port is overrun and some DataNode does not
register successfully, lifeline could not fix it, FYI. Please correct me if
there are something wrong.
{quote}
In tried to go through the registration part but it's not that easy to follow;
I would need a proper pass.
In any case, I think that if the DN is not registered, it cannot be marked as
DEAD.
I think, for a DN to show up in the NN, it needs to be fully registered.
If that's the case, the lifeline will keep it as ALIVE.
The issue might be the lock in {{HeartbeatManager#updateLifeline()}}.
Could you post where the NN is spending most of the time?
Ideally a stack trace of the thread that is holding the other requests.
> blockreport storm slow down namenode restart seriously in large cluster
> -----------------------------------------------------------------------
>
> Key: HDFS-14186
> URL: https://issues.apache.org/jira/browse/HDFS-14186
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
> Reporter: He Xiaoqiao
> Assignee: He Xiaoqiao
> Priority: Major
> Attachments: HDFS-14186.001.patch
>
>
> In the current implementation, the datanode sends blockreport immediately
> after register to namenode successfully when restart, and the blockreport
> storm will make namenode high load to process them. One result is some
> received RPC have to skip because queue time is timeout. If some datanodes'
> heartbeat RPC are continually skipped for long times (default is
> heartbeatExpireInterval=630s) it will be set DEAD, then datanode has to
> re-register and send blockreport again, aggravate blockreport storm and trap
> in a vicious circle, and slow down (more than one hour and even more)
> namenode startup seriously in a large (several thousands of datanodes) and
> busy cluster especially. Although there are many work to optimize namenode
> startup, the issue still exists.
> I propose to postpone dead datanode check when namenode have finished startup.
> Any comments and suggestions are welcome.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
