region server shutdown when running on 2 clients. From the log, seems region server can't connect to datanode.
Here is the region server log. I didn't find any exception in the datanode log. to hlog; editcount=291 2010-04-02 16:00:22,376 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_5538189142737707248_12736java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at java.io.DataInputStream.readLong(DataInputStream.java:399) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2399) 2010-04-02 16:00:22,376 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_-2614684857813896108_12085 from any node: java.io.IOException: No live nodes contain current block 2010-04-02 16:00:22,376 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_5538189142737707248_12736 bad datanode[0] 172.31.13.190:50010 2010-04-02 16:00:22,377 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer Exception: java.io.IOException: Connection reset by peer at sun.nio.ch.FileDispatcher.write0(Native Method) at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29) at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:104) at sun.nio.ch.IOUtil.write(IOUtil.java:75) at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:334) at org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55) at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146) at org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105) at java.io.DataOutputStream.write(DataOutputStream.java:90) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2290) 2010-04-02 16:00:22,377 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream ResponseProcessor exception for block blk_1462025791805252013_12727java.io.EOFException at java.io.DataInputStream.readFully(DataInputStream.java:180) at java.io.DataInputStream.readLong(DataInputStream.java:399) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2399) 2010-04-02 16:00:22,378 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_1462025791805252013_12727 bad datanode[0] 172.31.13.190:50010 2010-04-02 16:00:22,379 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_-5108099522205897784_11310 from any node: java.io.IOException: No live nodes contain current block 2010-04-02 16:00:22,378 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery for block blk_-7928413171578270686_12744 bad datanode[0] 172.31.13.190:50010 2010-04-02 16:00:22,381 FATAL org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Replay of hlog required. Forcing server shutdown org.apache.hadoop.hbase.DroppedSnapshotException: region: TestTable,0000073491,1270194953322 at org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:977) at org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:846) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:241) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushSomeRegions(MemStoreFlusher.java:352) at org.apache.hadoop.hbase.regionserver.MemStoreFlusher.reclaimMemStoreMemory(MemStoreFlusher.java:321) at org.apache.hadoop.hbase.regionserver.HRegionServer.put(HRegionServer.java:1775) at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:657) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:915) Caused by: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,384 WARN org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush failed 2010-04-02 16:00:22,385 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,386 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,387 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,387 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,388 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,388 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,388 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,388 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,389 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,389 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60020, call put([...@688c3d16, [Lorg.apache.hadoop.hbase.client.Put;@4efea4f9) from 172.31.13.232:60424: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,389 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,390 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,390 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,390 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,390 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,391 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60020, call put([...@42d1e0a3, [Lorg.apache.hadoop.hbase.client.Put;@60f30f9a) from 172.31.13.232:60424: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,389 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 17 on 60020, call put([...@3000fff3, [Lorg.apache.hadoop.hbase.client.Put;@11a3c324) from 172.31.13.215:36858: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,391 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 13 on 60020, call put([...@3a8edb93, [Lorg.apache.hadoop.hbase.client.Put;@78f1aded) from 172.31.13.232:60424: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,392 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 10 on 60020, call put([...@2cdb1ba9, [Lorg.apache.hadoop.hbase.client.Put;@64f50b40) from 172.31.13.232:60424: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,391 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60020, call put([...@17cadb32, [Lorg.apache.hadoop.hbase.client.Put;@6fcbbd2) from 172.31.13.215:36858: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,390 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,395 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,396 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 12 on 60020, call put([...@13b5ca88, [Lorg.apache.hadoop.hbase.client.Put;@403b0d4) from 172.31.13.215:36858: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,396 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,397 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,397 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,397 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,398 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,398 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,399 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60020, call put([...@2f48adb3, [Lorg.apache.hadoop.hbase.client.Put;@4c83db05) from 172.31.13.232:60424: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,400 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,400 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,400 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 18 on 60020, call put([...@64a04c81, [Lorg.apache.hadoop.hbase.client.Put;@57fedcc3) from 172.31.13.215:36858: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,401 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 22 on 60020, call put([...@1d4fb357, [Lorg.apache.hadoop.hbase.client.Put;@5b8723c9) from 172.31.13.215:36858: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,402 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,402 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,403 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,403 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,403 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 60020, call put([...@4e644ec7, [Lorg.apache.hadoop.hbase.client.Put;@1b701306) from 172.31.13.232:60424: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,404 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,404 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,405 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,405 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,406 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 21 on 60020, call put([...@5af0fcfc, [Lorg.apache.hadoop.hbase.client.Put;@3713309) from 172.31.13.232:60424: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,406 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60020, call put([...@7ccd95a2, [Lorg.apache.hadoop.hbase.client.Put;@1d1afeb1) from 172.31.13.215:36858: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,407 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,407 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,408 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 14 on 60020, call put([...@57472564, [Lorg.apache.hadoop.hbase.client.Put;@7fe7f4ff) from 172.31.13.232:60424: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,408 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 23 on 60020, call put([...@5585d2f3, [Lorg.apache.hadoop.hbase.client.Put;@44dc7b62) from 172.31.13.215:36858: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,409 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 19 on 60020, call put([...@6728823f, [Lorg.apache.hadoop.hbase.client.Put;@147f2b02) from 172.31.13.215:36858: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,409 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could not append. Requesting close of hlog java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,409 ERROR org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,392 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60020, call put([...@7585792f, [Lorg.apache.hadoop.hbase.client.Put;@dd738f0) from 172.31.13.232:60424: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,410 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 20 on 60020, call put([...@31d4244e, [Lorg.apache.hadoop.hbase.client.Put;@5e9b9670) from 172.31.13.215:36858: error: java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... java.io.IOException: All datanodes 172.31.13.190:50010 are bad. Aborting... at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2525) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$1600(DFSClient.java:2078) at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2241) 2010-04-02 16:00:22,987 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60020 2010-04-02 16:00:22,987 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60020: exiting 2010-04-02 16:00:22,988 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60020: exiting 2010-04-02 16:00:22,988 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60020 2010-04-02 16:00:22,989 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder 2010-04-02 16:00:22,990 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_-2614684857813896108_12085 from any node: java.io.IOException: No live nodes contain current block 2010-04-02 16:00:22,987 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60020: exiting 2010-04-02 16:00:22,991 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60020: exiting 2010-04-02 16:00:22,991 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60020: exiting 2010-04-02 16:00:22,991 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60020: exiting 2010-04-02 16:00:22,991 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60020: exiting 2010-04-02 16:00:22,991 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 10 on 60020: exiting 2010-04-02 16:00:22,991 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 11 on 60020: exiting 2010-04-02 16:00:22,991 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 12 on 60020: exiting 2010-04-02 16:00:22,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 13 on 60020: exiting 2010-04-02 16:00:22,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 14 on 60020: exiting 2010-04-02 16:00:22,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 16 on 60020: exiting 2010-04-02 16:00:22,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 17 on 60020: exiting 2010-04-02 16:00:22,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 18 on 60020: exiting 2010-04-02 16:00:22,992 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 19 on 60020: exiting 2010-04-02 16:00:22,993 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 21 on 60020: exiting 2010-04-02 16:00:22,997 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 22 on 60020: exiting 2010-04-02 16:00:22,997 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 23 on 60020: exiting 2010-04-02 16:00:22,998 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 24 on 60020: exiting 2010-04-02 16:00:22,998 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60020: exiting 2010-04-02 16:00:23,000 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60020: exiting 2010-04-02 16:00:23,001 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60020: exiting 2010-04-02 16:00:23,001 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 20 on 60020: exiting 在 2010年4月2日 下午3:46,Chen Bangzhong <bangzh...@gmail.com>写道: > Hi, All > > I am benchmarking hbase. My HDFS clusters includes 4 servers (Dell 860, > with 2 GB RAM). One NameNode, one JobTracker, 2 DataNodes. > > My HBase Cluster also comprise 4 servers too. One Master, 2 region and one > ZooKeeper. (Dell 860, with 2 GB RAM) > > I runned the org.apache.hadoop.PerformanceEvaluation on the ZooKeeper > server. the ROW_LENGTH was changed from 1000 to ROW_LENGTH = 100*1024; > So each value will be 100k in size. > > hadoop version is 0.20.2, hbase version is 0.20.3. dfs.replication set to > 1. > > The following is the command line: > > bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --nomapred > --rows=10000 randomWrite 20. > > It tooks about one hour to complete the test(3468628 ms), about 60 writes > per second. It seems the performance is disappointing. > > Is there anything I can do to make hbase perform better under 100k size ?I > didn't try the method mentioned in the performance wiki yet, because I > thought 60writes/sec is too low. > > If the value size is 1k, hbase performs much better. 200000 sequencewrite > tooks about 16 seconds, about 12500 writes/per second. > > Now I am trying to benchmark using two clients on 2 servers, no result yet. > > > > > >