[
https://issues.apache.org/jira/browse/HADOOP-1220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12491727
]
dhruba borthakur commented on HADOOP-1220:
------------------------------------------
This problem occured again. This time the block in question does not appear in
the dfs data directory. The namenode is requesting a block to be deleted. The
datanode tries this operation and encounters an error because the block is not
in the blockMap. The processCommand() method raises an exception. The code is
such that the variable lastBlockReport is not set if processCommand() raises an
exception. This means that the datanode immediately send another block report
to the namenode.
In short, the above condition causes the datanode to send blockReports almost
once every second!
I propose that we do the following:
1. in Datanode.offerService, replace the following piece of code
DatanodeCommand cmd = namenode.blockReport(dnRegistration,
data.getBlockReport());
processCommand(cmd);
lastBlockReport = now;
with
DatanodeCommand cmd = namenode.blockReport(dnRegistration,
data.getBlockReport());
lastBlockReport = now;
processCommand(cmd);
2. In FSDataSet.invalidate:
a) continue to process all blocks in invalidBlks[] even if one in the
middle encounters a problem.
b) if getFile() returns null, still invoke volumeMap.get() and print
whether we found the block in
volumes or not. The volumeMap is used to generate the blockReport and
this might help in debugging.
> block not found in blockMap
> ---------------------------
>
> Key: HADOOP-1220
> URL: https://issues.apache.org/jira/browse/HADOOP-1220
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: Koji Noguchi
>
> From Hadoop-973 debug message, we had datanode constantly printing out the
> following message.
> ...
> 2007-04-02 23:59:50,122 WARN org.apache.hadoop.dfs.DataNode:
> java.io.IOException: Unexpected error trying to delete block
> blk_-3400783150525166031. Block not found in blockMap.
> at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
> at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
> at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
> at java.lang.Thread.run(Thread.java:619)
> 2007-04-02 23:59:50,433 INFO org.apache.hadoop.dfs.DataNode: Served block
> blk_-8672111663356339464 to /72.30.127.164
> 2007-04-02 23:59:52,993 WARN org.apache.hadoop.dfs.DataNode:
> java.io.IOException: Unexpected error trying to delete block
> blk_-3400783150525166031. Block not found in blockMap.
> at org.apache.hadoop.dfs.FSDataset.invalidate(FSDataset.java:596)
> at org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:460)
> at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1053)
> at java.lang.Thread.run(Thread.java:619)
> ....
> Ten minutes before these logs, there was
> 2007-04-02 20:42:49,648 INFO org.apache.hadoop.dfs.DataNode: Deleting block
> blk_-3400783150525166031
> There was a file for that block in the directory.
> /../../hadoop/dfs/data/data/subdir17/blk_-3400783150525166031
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
