[
https://issues.apache.org/jira/browse/HADOOP-4584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676344#action_12676344]
Raghu Angadi commented on HADOOP-4584:
--------------------------------------
Ideally there is no requirement for block reports. It is essentially to
used as 'catch all' for various bugs and errors. (of course, it is now
overloaded with job of informing about deletions to NameNode, this should
removed).
Yes, it specifically removes disk scan without fundamentally changing
meaning of block reports. Now DN informs NameNode about the the block that
it thinks it had. because :
* 'rm -r' by admin is just one form of many many things that can go wrong
with blocks on datanode. There is no perticular reason we should have this
very costly disk scan (with a global lock held) just for this.
** In fact 'rm -r' is probably the least probable error (haven't seen
even once in practice).
* We have periodic block verification that does handle various things that
can go wrong with a block (it can improve further).
** So 'rm -r' will be handled, just at the rate of rest of the block
problems.
* on the other hand many users have complained about datanode scans taking
10s of minutes and making datanodes lose heartbeats.
** This makes the system pretty unusable and a major obstruction for
graceful degradation under load and for scalability.
** One can argue that those users should not have so many blocks. But I
think DN should still handle it to the best of it abilities and not die on
them.
** Disks might be slow for many other reasons (other tasks on the
machine, etc).
* I think this is orthogonal to HADOOP-1079 since it addresses RPC and
NameNode overhead of block reports. This jira is only about DataNode side.
Yes, this is a bigger change in semantics than what we proposed earlier :
to scan the directories slowly, without holding the global lock... but
offline scan looks like a workaround for a problem that does not need to be
solved. Not scanning is much simpler than handling offline scan.
Eventually we need to reduce the frequency of block reports.. this can be
done as soon as we add acks for block deletions. This JIRA is major step in
that direction.
Slow generation of blockReport at DataNode causes delay of sending
heartbeat to NameNode
----------------------------------------------------------------------------------------
Key: HADOOP-4584
URL: https://issues.apache.org/jira/browse/HADOOP-4584
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Reporter: Hairong Kuang
Assignee: Suresh Srinivas
Fix For: 0.20.0
Attachments: 4584.patch, 4584.patch, 4584.patch, 4584.patch,
4584.patch, 4584.patch
sometimes due to disk or some other problems, datanode takes minutes or
tens of minutes to generate a block report. It causes the datanode not able
to send heartbeat to NameNode every 3 seconds. In the worst case, it makes
NameNode to detect a lost heartbeat and wrongly decide that the datanode is
dead.
It would be nice to have two threads instead. One thread is for scanning
data directories and generating block report, and executes the requests sent
by NameNode; Another thread is for sending heartbeats, block reports, and
picking up the requests from NameNode. By having these two threads, the
sending of heartbeats will not get delayed by any slow block report or slow
execution of NameNode requests.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
