Ah, yes, very interesting. I get a "Read timed out" on 10.100.11.31,
followed by a bunch of "Served block" messages... then finally a
Transmitted block once HDFS was restarted.
hadoop-rapleaf-datanode-tf4.rapleaf.com.log.2008-01-02:2008-01-02
17:08:17,057 INFO org.apache.hadoop.dfs.DataNode: Received block
blk_4522585614366970680 from /10.100.11.31 and Read timed out
hadoop-rapleaf-datanode-tf4.rapleaf.com.log.2008-01-02:2008-01-02
17:13:17,281 INFO org.apache.hadoop.dfs.DataNode: Served block
blk_4522585614366970680 to /10.100.11.31
------ a bunch of the above message
hadoop-rapleaf-datanode-tf4.rapleaf.com.log.2008-01-02:2008-01-02
18:53:18,737 INFO org.apache.hadoop.dfs.DataNode: Served block
blk_4522585614366970680 to /10.100.11.31
------ HDFS restarted
hadoop-rapleaf-datanode-tf4.rapleaf.com.log.2008-01-03:2008-01-03
16:17:59,637 INFO org.apache.hadoop.dfs.DataNode: Starting thread to
transfer block blk_4522585614366970680 to
[Lorg.apache.hadoop.dfs.DatanodeInfo;@7e9ffe3f
hadoop-rapleaf-datanode-tf4.rapleaf.com.log.2008-01-03:2008-01-03
16:17:59,718 INFO org.apache.hadoop.dfs.DataNode: Transmitted block
blk_4522585614366970680 to /10.100.11.59:7277
On Jan 7, 2008, at 11:06 PM, Raghu Angadi wrote:
hmm... one possibility is that rest of the nodes were down. but the
name node showed other nodes were up. If more than one datanodes
were up, this indicates some bug.
One last thing : grep for this block id at 10.100.11.31. You might
see some useful error message when the block was written.
thanks,
Raghu.
Chris Kline wrote:
Sure. Here are 2 blocks.
----- blk_4522585614366970680
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-02:2008-01-02
17:06:17,058 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.allocateBlock: /data/hbase1/hregion_70236052/
compaction.dir/hregion_70236052/info/mapfiles/-1/index.
blk_4522585614366970680
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-02:2008-01-02
17:06:17,062 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.100.11.31:7277 is
added to blk_4522585614366970680
------ Note that the following is AFTER I restarted HDFS -------
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03
16:17:15,882 DEBUG org.apache.hadoop.dfs.StateChange:
UnderReplicationBlocks.update blk_4522585614366970680 curReplicas
1 curExpectedReplicas 3 oldReplicas 0 oldExpectedReplicas 3
curPri 0 oldPri 3
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03
16:17:15,882 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.UnderReplicationBlock.update:blk_4522585614366970680
has only 1 replicas and need 3 replicas so is added to
neededReplications at priority level 0
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03
16:17:58,808 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.UnderReplicationBlock.remove: Removing block
blk_4522585614366970680 from priority queue 0
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03
16:17:58,808 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.pendingTransfer: blk_4522585614366970680 is removed
from neededReplications to pendingReplications
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03
16:17:58,808 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.pendingTransfer: ask 10.100.11.31:7277 to replicate
blk_4522585614366970680 to datanode(s) 10.100.11.59:7277
10.100.11.64:7277
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03
16:17:59,716 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.blockReceived: blk_4522585614366970680 is received from
10.100.11.59:7277
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03
16:17:59,716 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.100.11.59:7277 is
added to blk_4522585614366970680
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03
16:17:59,717 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.blockReceived: blk_4522585614366970680 is received from
10.100.11.64:7277
hadoop-rapleaf-namenode-tf1.rapleaf.com.log.2008-01-03:2008-01-03
16:17:59,717 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.addStoredBlock: blockMap updated: 10.100.11.64:7277 is
added to blk_4522585614366970680
-Chris
We're hiring engineers. $10,007 reward for referrals we hire.
