[
https://issues.apache.org/jira/browse/HADOOP-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676384#action_12676384
]
Raghu Angadi commented on HADOOP-4584:
--------------------------------------
I don't think read/write helps much (since we can't make writes wait for that
long) or necessary for this.
It is not very complicated to scan without locking.. DN just needs to keep
track of changes (additions and deletions) that happen from the time scan
starts to the time it ends and appy those changes at the end. That way scan can
proceed without locking the storage.
Another way to achieve the same to keep a modification timestamp with each
block and at the end of the scan, if a in memory block has a timestamp older
than start of the scan but not found in scan, then it should be deleted.. etc.
None the less it not as easy as not scanning :).
I think Suresh had more elaborate description of the above with examples.
> Slow generation of blockReport at DataNode causes delay of sending heartbeat
> to NameNode
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-4584
> URL: https://issues.apache.org/jira/browse/HADOOP-4584
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Hairong Kuang
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: 4584.patch, 4584.patch, 4584.patch, 4584.patch,
> 4584.patch, 4584.patch
>
>
> sometimes due to disk or some other problems, datanode takes minutes or tens
> of minutes to generate a block report. It causes the datanode not able to
> send heartbeat to NameNode every 3 seconds. In the worst case, it makes
> NameNode to detect a lost heartbeat and wrongly decide that the datanode is
> dead.
> It would be nice to have two threads instead. One thread is for scanning data
> directories and generating block report, and executes the requests sent by
> NameNode; Another thread is for sending heartbeats, block reports, and
> picking up the requests from NameNode. By having these two threads, the
> sending of heartbeats will not get delayed by any slow block report or slow
> execution of NameNode requests.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
