We are seeing a lot of warnings and errors in the HDFS logs (examples
below). I am looking for any help or recommendations anyone can provide. It
almost looks like compaction/splits occur and errors occur while looking for
the old data. Could this be true? Are these warnings and errors "normal"?
2011-01-10 16:55:48,920 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Error processing
datanode Command java.io.IOException: Error in deleting blocks.
2011-01-10 05:33:25,353 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected error
trying to delete block blk_6074091327320053721_1406212. BlockInfo not
found in volumeMap.
We also see a lot of messages like the one below.
2011-01-10 17:04:51,593 WARN
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(x.x.x.x1:50010,
storageID=DS-1353352923-127.0.0.1-50010-1293566862209, infoPort=50075,
ipcPort=50020):Got exception while serving
blk_-6095972096875566659_1323729 to /x.x.x.x8:
java.io.IOException: Block blk_-6095972096875566659_1323729 is not valid.
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:931)
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:894)
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:904)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:172)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:95)
at java.lang.Thread.run(Unknown Source)
2011-01-10 17:04:51,593 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
DatanodeRegistration(x.x.x.x1:50010,
storageID=DS-1353352923-127.0.0.1-50010-1293566862209, infoPort=50075,
ipcPort=50020):DataXceiver
java.io.IOException: Block blk_-6095972096875566659_1323729 is not valid.
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.getBlockFile(FSDataset.java:931)
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.getLength(FSDataset.java:894)
at
org.apache.hadoop.hdfs.server.datanode.FSDataset.getVisibleLength(FSDataset.java:904)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.<init>(BlockSender.java:94)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:172)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:95)
at java.lang.Thread.run(Unknown Source)
I see that the block in question was deleted several hours earlier in the
logs on node 8.
2011-01-10 05:32:04,500 INFO
org.apache.hadoop.hdfs.server.datanode.DataNode: Deleting block
blk_-6095972096875566659_1323729 file
/mainraid/hadoop/hbase/data/current/subdir2/subdir15/blk_-6095972096875566659
Thanks.
