Namenode blockMap not updated when datanode invalidates a block on heart beat
-----------------------------------------------------------------------------
Key: HADOOP-3885
URL: https://issues.apache.org/jira/browse/HADOOP-3885
Project: Hadoop Core
Issue Type: Bug
Components: dfs
Affects Versions: 0.18.0
Reporter: Wei Xu
Here are related logs. dn_10_251_107_213 deleted the block immediately after
it received it (because of over-replication). However, the information is not
reported to namenode and name node ask this DN to invalidate the block again,
causing the error "Unexpected error trying to delete block
blk_-1201258788540100829. BlockInfo not found in volumeMap.".
dn_10_251_107_213 | 080722 145554 2497 INFO dfs.DataNode$DataXceiver: Receiving
block blk_-1201258788540100829 src: /10.251.67.214:54770 dest:
/10.251.67.214:50010
dn_10_251_193_205 | 080722 145554 2468 INFO dfs.DataNode$DataXceiver: Receiving
block blk_-1201258788540100829 src: /10.251.107.213:36718 dest:
/10.251.107.213:50010
dn_10_251_67_214 | 080722 145554 2306 INFO dfs.DataNode$DataXceiver: Receiving
block blk_-1201258788540100829 src: /10.251.67.214:60599 dest:
/10.251.67.214:50010
nn_10_251_210_161 | 080722 145554 30 INFO dfs.FSNamesystem: BLOCK*
NameSystem.allocateBlock:
/user/root/rand/_temporary/_task_200807221347_0001_m_000092_1/part-00092.
blk_-1201258788540100829
dn_10_251_107_213 | 080722 145656 2498 INFO dfs.DataNode$PacketResponder:
Received block blk_-1201258788540100829 of size 29767698 from /10.251.67.214
dn_10_251_193_205 | 080722 145656 2469 INFO dfs.DataNode$PacketResponder:
Received block blk_-1201258788540100829 of size 29767698 from /10.251.107.213
dn_10_251_67_214 | 080722 145656 2307 INFO dfs.DataNode$PacketResponder:
Received block blk_-1201258788540100829 of size 29767698 from /10.251.67.214
dn_10_251_107_213 | 080722 145656 2498 INFO dfs.DataNode$PacketResponder:
PacketResponder 1 for block blk_-1201258788540100829 terminating
dn_10_251_193_205 | 080722 145656 2469 INFO dfs.DataNode$PacketResponder:
PacketResponder 0 for block blk_-1201258788540100829 terminating
dn_10_251_67_214 | 080722 145656 2307 INFO dfs.DataNode$PacketResponder:
PacketResponder 2 for block blk_-1201258788540100829 terminating
nn_10_251_210_161 | 080722 145656 19 INFO dfs.FSNamesystem: BLOCK* ask
10.251.67.214:50010 to replicate blk_-1201258788540100829 to datanode(s)
10.251.107.49:50010
nn_10_251_210_161 | 080722 145656 31 INFO dfs.FSNamesystem: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.251.67.214:50010 is added to
blk_-1201258788540100829 size 29767698
nn_10_251_210_161 | 080722 145656 31 INFO dfs.FSNamesystem: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.251.107.213:50010 is added to
blk_-1201258788540100829 size 29767698
nn_10_251_210_161 | 080722 145656 26 INFO dfs.FSNamesystem: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.251.193.205:50010 is added to
blk_-1201258788540100829 size 29767698
dn_10_251_107_49 | 080722 145658 2484 INFO dfs.DataNode$DataXceiver: Receiving
block blk_-1201258788540100829 src: /10.251.67.214:42237 dest:
/10.251.67.214:50010
dn_10_251_67_214 | 080722 145658 18 INFO dfs.DataNode: 10.251.67.214:50010
Starting thread to transfer block blk_-1201258788540100829 to
10.251.107.49:50010
dn_10_251_107_49 | 080722 145705 2484 INFO dfs.DataNode$DataXceiver: Received
block blk_-1201258788540100829 src: /10.251.67.214:42237 dest:
/10.251.67.214:50010 of size 29767698
dn_10_251_67_214 | 080722 145705 2342 INFO dfs.DataNode$DataTransfer:
10.251.67.214:50010:Transmitted block blk_-1201258788540100829 to
/10.251.107.49:50010
nn_10_251_210_161 | 080722 145705 34 INFO dfs.FSNamesystem: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.251.107.49:50010 is added to
blk_-1201258788540100829 size 29767698
dn_10_251_107_213 | 080722 145710 19 INFO dfs.FSDataset: Deleting block
blk_-1201258788540100829 file
/mnt/hadoop/dfs/data/current/subdir11/blk_-1201258788540100829
nn_10_251_210_161 | 080722 150220 34 INFO dfs.FSNamesystem: BLOCK*
NameSystem.delete: blk_-1201258788540100829 is added to invalidSet of
10.251.67.214:50010
nn_10_251_210_161 | 080722 150220 34 INFO dfs.FSNamesystem: BLOCK*
NameSystem.delete: blk_-1201258788540100829 is added to invalidSet of
10.251.107.213:50010
nn_10_251_210_161 | 080722 150220 34 INFO dfs.FSNamesystem: BLOCK*
NameSystem.delete: blk_-1201258788540100829 is added to invalidSet of
10.251.193.205:50010
nn_10_251_210_161 | 080722 150220 34 INFO dfs.FSNamesystem: BLOCK*
NameSystem.delete: blk_-1201258788540100829 is added to invalidSet of
10.251.107.49:50010
dn_10_251_193_205 | 080722 150224 19 INFO dfs.FSDataset: Deleting block
blk_-1201258788540100829 file
/mnt/hadoop/dfs/data/current/subdir51/blk_-1201258788540100829
dn_10_251_107_49 | 080722 150226 19 INFO dfs.FSDataset: Deleting block
blk_-1201258788540100829 file
/mnt/hadoop/dfs/data/current/subdir24/blk_-1201258788540100829
dn_10_251_107_213 | 080722 150228 19 WARN dfs.FSDataset: Unexpected error
trying to delete block blk_-1201258788540100829. BlockInfo not found in
volumeMap.
dn_10_251_67_214 | 080722 150253 18 INFO dfs.FSDataset: Deleting block
blk_-1201258788540100829 file
/mnt/hadoop/dfs/data/current/subdir43/blk_-1201258788540100829
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
