[
https://issues.apache.org/jira/browse/HBASE-1087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-1087.
--------------------------
Resolution: Fixed
Resolving this issue. Lets reopen or open another if we seen another hang like
this (or add it under hbase-1084 umbrella).
> DFS failures did not shutdown regionserver
> ------------------------------------------
>
> Key: HBASE-1087
> URL: https://issues.apache.org/jira/browse/HBASE-1087
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.19.0
> Reporter: Jonathan Gray
> Priority: Critical
> Fix For: 0.19.0
>
> Attachments: regionserver.threaddump.log
>
>
> I lost three Datanodes, reasons of which are still being investigated, but it
> has left a number of regions unable to be written to.
> Relevant logs:
> 2008-12-23 02:35:59,591 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
> Exception: java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcher.write0(Native Method)
> at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
> at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:122)
> at sun.nio.ch.IOUtil.write(IOUtil.java:93)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:352)
> at
> org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:55)
> at
> org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:140)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:146)
> at
> org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:107)
> at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
> at java.io.DataOutputStream.write(DataOutputStream.java:107)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2209)
> 2008-12-23 02:35:59,591 WARN org.apache.hadoop.hdfs.DFSClient:
> DFSOutputStream ResponseProcessor exception for block
> blk_3615512604618056881_86411java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:197)
> at java.io.DataInputStream.readLong(DataInputStream.java:416)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$ResponseProcessor.run(DFSClient.java:2318)
> 2008-12-23 02:35:59,591 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery
> for block blk_3615512604618056881_86411 bad datanode[0] 72.34.249.214:50010
> 2008-12-23 02:35:59,595 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery
> for block blk_3615512604618056881_86411 in pipeline 72.34.249.214:50010,
> 72.34.249.213:50010, 72.34.249.219:50010: bad datanode 72.34.249.214:50010
> 2008-12-23 02:38:27,698 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.net.ConnectException: Connection refused
> 2008-12-23 02:38:27,698 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_-3678518999439029831_86910
> 2008-12-23 02:38:27,711 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
> 5392007859847346106 has been explicitly released by client
> 2008-12-23 02:38:30,048 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
> -5905479324505886709 explicitly acquired by client
> 2008-12-23 02:38:33,700 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.net.ConnectException: Connection refused
> 2008-12-23 02:38:33,700 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_-226119866881174578_86911
> 2008-12-23 02:38:34,908 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
> 346704317670569896 explicitly acquired by client
> 2008-12-23 02:38:39,702 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.net.ConnectException: Connection refused
> 2008-12-23 02:38:39,702 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_1719395740576248920_86913
> 2008-12-23 02:38:40,945 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
> 3819942931078736534 explicitly acquired by client
> 2008-12-23 02:38:45,572 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
> 254119037927296402 explicitly acquired by client
> 2008-12-23 02:38:45,703 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.net.ConnectException: Connection refused
> 2008-12-23 02:38:45,703 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_2443399503093377808_86915
> 2008-12-23 02:38:49,092 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
> 8573046623144113301 explicitly acquired by client
> 2008-12-23 02:38:49,385 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
> 7686739650257547105 explicitly acquired by client
> 2008-12-23 02:38:49,512 DEBUG
> org.apache.hadoop.hbase.regionserver.HRegionServer: Row lock
> 5582966798894532276 explicitly acquired by client
> 2008-12-23 02:38:51,704 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
> Exception: java.io.IOException: Unable to create new block.
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2723)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2008-12-23 02:38:51,704 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery
> for block blk_2443399503093377808_86915 bad datanode[0] nodes == null
> 2008-12-23 02:38:51,704 WARN org.apache.hadoop.hdfs.DFSClient: Could not get
> block locations. Aborting...
> 2008-12-23 02:38:51,704 FATAL org.apache.hadoop.hbase.regionserver.HLog:
> Could not append. Requesting close of log
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2748)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2008-12-23 02:38:51,706 ERROR org.apache.hadoop.hbase.regionserver.LogRoller:
> Log rolling failed with ioe:
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at sun.nio.ch.SocketAdaptor.connect(SocketAdaptor.java:118)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2748)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2008-12-23 02:38:51,706 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer:
> java.net.ConnectException: Connection refused
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
