DFS failures did not shutdown regionserver
------------------------------------------
Key: HBASE-1087
URL: https://issues.apache.org/jira/browse/HBASE-1087
Project: Hadoop HBase
Issue Type: Bug
Affects Versions: 0.19.0
Reporter: Jonathan Gray
Priority: Critical
Fix For: 0.19.0
I lost three Datanodes, reasons of which are still being investigated, but it
has left a number of regions unable to be written to.
Relevant logs:
2008-12-23 02:35:59,591 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)
at sun.nio.ch.IOUtil.write(IOUtil.java:93)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)
at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:140)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
at java.io.DataOutputStream.write(DataOutputStream.java:107)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2209)
2008-12-23 02:35:59,591 WARN org.apache.hadoop.hdfs.DFSClient: DFSOutputStream
ResponseProcessor exception for block
blk_3615512604618056881_86411java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:197)
at java.io.DataInputStream.readLong(DataInputStream.java:416)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2318)
2008-12-23 02:35:59,591 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery
for block blk_3615512604618056881_86411 bad datanode[0] 72.34.249.214:50010
2008-12-23 02:35:59,595 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery
for block blk_3615512604618056881_86411 in pipeline 72.34.249.214:50010,
72.34.249.213:50010, 72.34.249.219:50010: bad datanode 72.34.249.214:50010
2008-12-23 02:38:27,698 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.net.ConnectException: Connection refused
2008-12-23 02:38:27,698 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block
blk_-3678518999439029831_86910
2008-12-23 02:38:27,711 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
5392007859847346106 has been explicitly released by client
2008-12-23 02:38:30,048 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
-5905479324505886709 explicitly acquired by client
2008-12-23 02:38:33,700 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.net.ConnectException: Connection refused
2008-12-23 02:38:33,700 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block
blk_-226119866881174578_86911
2008-12-23 02:38:34,908 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock 346704317670569896
explicitly acquired by client
2008-12-23 02:38:39,702 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.net.ConnectException: Connection refused
2008-12-23 02:38:39,702 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block
blk_1719395740576248920_86913
2008-12-23 02:38:40,945 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
3819942931078736534 explicitly acquired by client
2008-12-23 02:38:45,572 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock 254119037927296402
explicitly acquired by client
2008-12-23 02:38:45,703 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
createBlockOutputStream java.net.ConnectException: Connection refused
2008-12-23 02:38:45,703 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning block
blk_2443399503093377808_86915
2008-12-23 02:38:49,092 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
8573046623144113301 explicitly acquired by client
2008-12-23 02:38:49,385 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
7686739650257547105 explicitly acquired by client
2008-12-23 02:38:49,512 DEBUG
org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
5582966798894532276 explicitly acquired by client
2008-12-23 02:38:51,704 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception: java.io.IOException: Unable to create new block.
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2723)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
2008-12-23 02:38:51,704 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery
for block blk_2443399503093377808_86915 bad datanode[0] nodes == null
2008-12-23 02:38:51,704 WARN org.apache.hadoop.hdfs.DFSClient: Could not get
block locations. Aborting...
2008-12-23 02:38:51,704 FATAL org.apache.hadoop.hbase.regionserver.HLog: Could
not append. Requesting close of log
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2748)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
2008-12-23 02:38:51,706 ERROR org.apache.hadoop.hbase.regionserver.LogRoller:
Log rolling failed with ioe:
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2748)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
at
org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
2008-12-23 02:38:51,706 ERROR
org.apache.hadoop.hbase.regionserver.HRegionServer: java.net.ConnectException:
Connection refused
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
