[
https://issues.apache.org/jira/browse/HDFS-6841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14365300#comment-14365300
]
Kihwal Lee edited comment on HDFS-6841 at 3/17/15 3:13 PM:
-----------------------------------------------------------
We have seen multiple occasions of dead datanode coming back and trying to
resend an IBR every 1 second. Normally the offerService loop makes it hreatbeat
and re-register, but these nodes were skipping heartbeat and kept re-sending
IBR. I got a hold of one such node and got a heap dump to see what's going on.
{noformat}
lastBlockReport (J) : 1426164231598
lastCacheReport (J) : 0
lastDeletedReport (J) : 1426206179858
lastHeartbeat (J) : 2693080084078
{noformat}
Since {{lastHeartbeat}} was huge, it wasn't sending heartbeat. We should move
forward with this.
was (Author: kihwal):
We have seen multiple occasions of dead datanode coming back and trying to
resend an IBR every 1 second. Normally the offerService loop makes it hreatbeat
and re-register, but these nodes were skipping heartbeat and kept re-sending
IBR. I got a hold of one such node and got a heap dump to see what's going on.
{noformat}
lastBlockReport (J) : 1426164231598
lastCacheReport (J) : 0
lastDeletedReport (J) : 1426206179858
lastHeartbeat (J) : 2693080084078
{noformat}
Since {{lastHeartbeat}} is huge, it wasn't sending heartbeat. We should move
forward with this.
> Use Time.monotonicNow() wherever applicable instead of Time.now()
> -----------------------------------------------------------------
>
> Key: HDFS-6841
> URL: https://issues.apache.org/jira/browse/HDFS-6841
> Project: Hadoop HDFS
> Issue Type: Bug
> Reporter: Vinayakumar B
> Assignee: Vinayakumar B
> Attachments: HDFS-6841-001.patch, HDFS-6841-002.patch,
> HDFS-6841-003.patch, HDFS-6841-004.patch
>
>
> {{Time.now()}} used in many places to calculate elapsed time.
> This should be replaced with {{Time.monotonicNow()}} to avoid effect of
> System time changes on elapsed time calculations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
