Hi all, I am not sure whether it is a hbase problem or a hdfs problem.
When have 8 datanodes & regionservers each with 12 disks. One data node is down due to its system disk broken. After replacing the disk and reinstall the OS, we tried to online this DN & region server. The region server is OK. But, the data node seems can only accept block replication from other data nodes, but failed to accept writing from HBase. Even after we removed all content of data dir and restarted the DN, this problem still exists. The log is: 2011-11-01 22:39:44,397 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received block blk_-6413273237261014407_77967 src: /10.1.2.23:56859 dest: /10.1.2.25:50010of size 67108864 2011-11-01 22:39:45,959 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received block blk_-6561707380479467846_78302 src: /10.1.2.20:47578 dest: /10.1.2.25:50010of size 67108864 2011-11-01 22:39:46,314 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-6233096255264073662_235292 src: /10.1.2.26:36311 dest: /10.1.2.25:50010 2011-11-01 22:39:46,829 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.1.2.21:35450, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE, cliID: DFSClient_1189047403, offset: 0, srvID: DS-1739614687-10.1.2.25-50010-1320156634502, blockid: blk_7603159785320952833_235458, duration: 20790059066 2011-11-01 22:39:46,829 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_7603159785320952833_235458 terminating 2011-11-01 22:39:46,850 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_7980572561428708688_235458 src: /10.1.2.19:37630 dest: /10.1.2.25:50010 2011-11-01 22:39:47,093 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.1.2.19:37611, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE, cliID: DFSClient_-81251145, offset: 0, srvID: DS-1739614687-10.1.2.25-50010-1320156634502, blockid: blk_-9067419333696936066_235458, duration: 20478973800 2011-11-01 22:39:47,093 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-9067419333696936066_235458 terminating 2011-11-01 22:39:47,117 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-495608789656244385_235458 src: /10.1.2.21:35468 dest: /10.1.2.25:50010 2011-11-01 22:39:47,185 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.1.2.23:56845, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE, cliID: DFSClient_1647768037, offset: 0, srvID: DS-1739614687-10.1.2.25-50010-1320156634502, blockid: blk_-7935479347615843321_235458, duration: 20595122675 2011-11-01 22:39:47,185 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-7935479347615843321_235458 terminating 2011-11-01 22:39:47,325 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received block blk_-6233096255264073662_235292 src: /10.1.2.26:36311 dest: /10.1.2.25:50010of size 67108864 2011-11-01 22:39:49,981 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.1.2.25:60888, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE, cliID: DFSClient_-1345645880, offset: 0, srvID: DS-1739614687-10.1.2.25-50010-1320156634502, blockid: blk_2614344148027463064_235458, duration: 19952693395 2011-11-01 22:39:49,982 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 2 for block blk_2614344148027463064_235458 terminating 2011-11-01 22:39:49,999 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_5157459941530714338_235458 src: /10.1.2.25:60892 dest: /10.1.2.25:50010 2011-11-01 22:39:51,258 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-6108054911705086653_39962 src: /10.1.2.21:35475 dest: /10.1.2.25:50010 2011-11-01 22:39:51,259 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Received block blk_-6108054911705086653_39962 src: /10.1.2.21:35475 dest: /10.1.2.25:50010of size 124 2011-11-01 22:39:52,357 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-5980257649446708968_78270 src: /10.1.2.23:56867 dest: /10.1.2.25:50010 2011-11-01 22:39:53,336 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_-7734816649100892246_235458 src: /10.1.2.26:36320 dest: /10.1.2.25:50010 2011-11-01 22:39:57,162 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: / 10.1.2.22:37400, dest: /10.1.2.25:50010, bytes: 67108864, op: HDFS_WRITE, cliID: DFSClient_-858857344, offset: 0, srvID: DS-1739614687-10.1.2.25-50010-1320156634502, blockid: blk_-1542408817127580486_235458, duration: 21156605250 2011-11-01 22:39:57,162 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_-1542408817127580486_235458 terminating Thanks Weihua
