[
https://issues.apache.org/jira/browse/HADOOP-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas updated HADOOP-4584:
------------------------------------
Attachment: 4584.brthread.4.patch
Previous patch results in a bug where blocks are marked corrupt. In the patch,
block report is built by listing blocks from {{FSDataset.volumeMap}}. However
the map has both finalized and non-finalized blocks. The non-finalized blocks
are reported with size set to the default block size in the block report. When
a last block is finalized, block received notification to the name node reports
the new size smaller than previously reported. Namenode marks such blocks as
corrupt.
The new patch adds only finalized blocks to the block report.
> Slow generation of blockReport at DataNode causes delay of sending heartbeat
> to NameNode
> ----------------------------------------------------------------------------------------
>
> Key: HADOOP-4584
> URL: https://issues.apache.org/jira/browse/HADOOP-4584
> Project: Hadoop Core
> Issue Type: Bug
> Components: dfs
> Reporter: Hairong Kuang
> Assignee: Suresh Srinivas
> Fix For: 0.20.0
>
> Attachments: 4584.brthread.2.patch, 4584.brthread.3.patch,
> 4584.brthread.3.patch, 4584.brthread.3.patch, 4584.brthread.3.patch,
> 4584.brthread.3.patch, 4584.brthread.4.patch, 4584.brthread.4.patch,
> 4584.brthread.4.patch, 4584.hbthread.patch, 4584.patch, 4584.patch,
> 4584.patch, 4584.patch, 4584.patch, 4584.patch, Design.pdf, Design.pdf
>
>
> sometimes due to disk or some other problems, datanode takes minutes or tens
> of minutes to generate a block report. It causes the datanode not able to
> send heartbeat to NameNode every 3 seconds. In the worst case, it makes
> NameNode to detect a lost heartbeat and wrongly decide that the datanode is
> dead.
> It would be nice to have two threads instead. One thread is for scanning data
> directories and generating block report, and executes the requests sent by
> NameNode; Another thread is for sending heartbeats, block reports, and
> picking up the requests from NameNode. By having these two threads, the
> sending of heartbeats will not get delayed by any slow block report or slow
> execution of NameNode requests.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
