[
https://issues.apache.org/jira/browse/HBASE-1132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack resolved HBASE-1132.
--------------------------
Resolution: Fixed
Fix Version/s: 0.19.0
> Can't append to HLog, can't roll log, infinite cycle (another spin on
> HBASE-930)
> --------------------------------------------------------------------------------
>
> Key: HBASE-1132
> URL: https://issues.apache.org/jira/browse/HBASE-1132
> Project: Hadoop HBase
> Issue Type: Bug
> Environment: Ryan Rawson cluster (TRUNK)
> Reporter: stack
> Fix For: 0.19.0
>
>
> Saw below loop in Ryan Rawson logs:
> {code}
> ....
> 2009-01-16 15:32:43,001 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_-2067415907098101353_164148
> 2009-01-16 15:32:45,561 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Could not read from stream
> 2009-01-16 15:32:45,561 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_4699358014912484437_164148
> 2009-01-16 15:32:49,004 INFO org.apache.hadoop.hdfs.DFSClient: Exception in
> createBlockOutputStream java.io.IOException: Bad connect ack with
> firstBadLink 10.10.20.19:50010
> 2009-01-16 15:32:49,004 INFO org.apache.hadoop.hdfs.DFSClient: Abandoning
> block blk_-8649135750875451286_164148
> 2009-01-16 15:32:51,562 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
> Exception: java.io.IOException: Unable to create new block.
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2723)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2009-01-16 15:32:51,562 WARN org.apache.hadoop.hdfs.DFSClient: Error Recovery
> for block blk_4699358014912484437_164148 bad datanode[0] nodes == null
> 2009-01-16 15:32:51,562 WARN org.apache.hadoop.hdfs.DFSClient: Could not get
> block locations. Aborting...
> 2009-01-16 15:32:51,562 FATAL org.apache.hadoop.hbase.regionserver.HLog:
> Could not append. Requesting close of log
> java.io.IOException: Could not read from stream
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
> at java.io.DataInputStream.readByte(DataInputStream.java:265)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2009-01-16 15:32:51,563 ERROR org.apache.hadoop.hbase.regionserver.LogRoller:
> Log rolling failed with ioe:
> java.io.IOException: Could not read from stream
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
> at java.io.DataInputStream.readByte(DataInputStream.java:265)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2009-01-16 15:32:51,564 FATAL org.apache.hadoop.hbase.regionserver.HLog:
> Could not append. Requesting close of log
> java.io.IOException: Could not read from stream
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
> at java.io.DataInputStream.readByte(DataInputStream.java:265)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> 2009-01-16 15:32:51,563 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException:
> Could not read from stream
> 2009-01-16 15:32:51,564 ERROR
> org.apache.hadoop.hbase.regionserver.HRegionServer: java.io.IOException:
> Could not read from stream
> 2009-01-16 15:32:51,564 FATAL org.apache.hadoop.hbase.regionserver.HLog:
> Could not append. Requesting close of log
> java.io.IOException: Could not read from stream
> at
> org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:119)
> at java.io.DataInputStream.readByte(DataInputStream.java:265)
> at org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:325)
> at org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:346)
> at org.apache.hadoop.io.Text.readString(Text.java:400)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream(DFSClient.java:2779)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2704)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.access$2000(DFSClient.java:1997)
> at
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:2183)
> ...
> {code}
> For 930, for different exception type, we triggered abort. Should do same
> here. If IOE and "Can't read from stream", shut down. The filesystem
> check seems to be coming back fine and dandy.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
