[
https://issues.apache.org/jira/browse/HDFS-9107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14900335#comment-14900335
]
Yi Liu commented on HDFS-9107:
------------------------------
Thanks [~daryn], the issue seems critical.
I have few comments:
*1.* the default heartbeat recheck interval is 5 minutes if not configured, is
it possible a full GC longer than 5 minutes? I see some full gc lasts tens of
seconds, but not saw so long, of course, it depends on the heap size (old
generation). Actually the data node dead (heartbeat expire) interval is 2x than
heartbeat recheck interval, so the full gc should last 10 minutes.
*2.* The patch assumes the full gc happens during the {{sleep}}, it's most
possible, but if it happens after {{long now = ..}} or setting
{{lastHeatbeatCheck}} to {{now}}, the issue still exists, even though small
probability.
But I would like to give +1 for the patch, since it solves the issue if really
happen, and doesn't affect existing logic.
> Prevent NN's unrecoverable death spiral after full GC
> -----------------------------------------------------
>
> Key: HDFS-9107
> URL: https://issues.apache.org/jira/browse/HDFS-9107
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 2.0.0-alpha
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Priority: Critical
> Attachments: HDFS-9107.patch
>
>
> A full GC pause in the NN that exceeds the dead node interval can lead to an
> infinite cycle of full GCs. The most common situation that precipitates an
> unrecoverable state is a network issue that temporarily cuts off multiple
> racks.
> The NN wakes up and falsely starts marking nodes dead. This bloats the
> replication queues which increases memory pressure. The replications create a
> flurry of incremental block reports and a glut of over-replicated blocks.
> The "dead" nodes heartbeat within seconds. The NN forces a re-registration
> which requires a full block report - more memory pressure. The NN now has to
> invalidate all the over-replicated blocks. The extra blocks are added to
> invalidation queues, tracked in an excess blocks map, etc - much more memory
> pressure.
> All the memory pressure can push the NN into another full GC which repeats
> the entire cycle.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
