One follow up.

I noticed the master was saying this as well:

2010-07-13 17:48:17,595 WARN org.apache.hadoop.hbase.master.HMaster: Processing 
pending operations: ProcessRegionClose of 
filestore,5c4ee356835d82de844600895a7df1595c8b53bd378436f6e8d2a1a99acb550c,1279043794914,
 false, reassign: true
java.lang.RuntimeException: java.lang.NullPointerException
        at 
org.apache.hadoop.hbase.master.RetryableMetaOperation.doWithRetries(RetryableMetaOperation.java:96)
        at 
org.apache.hadoop.hbase.master.ProcessRegionClose.process(ProcessRegionClose.java:63)
        at 
org.apache.hadoop.hbase.master.HMaster.processToDoQueue(HMaster.java:494)
        at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:429)
Caused by: java.lang.NullPointerException

Which lead me to this:

https://issues.apache.org/jira/browse/HBASE-2428

I guess I shouldn't have put off upgrading to 0.20.5 :(
I'm curious if by this comment:

"Since delayqueue is processed first, all we did was try to process the close 
even though in the other more live todo queue, we might have the process of the 
regionserver shutdown waiting, the process of the regionserver that was 
carrying .meta. We'd never recover. "

They mean never as in _never_ or just as in until the next reboot. Or is my 
cluster (and my data) dead and gone now?

Take care,
  -stu

--- On Tue, 7/13/10, Stuart Smith <[email protected]> wrote:

> From: Stuart Smith <[email protected]>
> Subject: Trying to write too much to stdout destabilizes cluster across 
> reboots
> To: [email protected]
> Date: Tuesday, July 13, 2010, 8:58 PM
> Hello,
>  
>    I have a M/R task running over hbase
> tables. I added
>  a one line of output in my reduce tasks running over a
> set
>  of tens of millions of records. This was a little silly,
> but
>  I was kind of curious what would happen. 
>  
>  Well the reduce ran very slowly, and eventually, I had to
>  kill the tasks, and I re-started the whole cluster, as
> other
>  clients were slowing up considerably, and the M/R job
>  started dumping all the stdout directly to the console it
>  was running on. 
>  
>  I've re-started it twice, but I couldn't get hbase to
> come
>  backup (fully). And when I shut down, the region servers
>  will go down when I tell them, but the master needs to be
>  killed with -s 9 (SIG_KILL? I'm a little rusty). It
>  eventually seemed to hobble towards coming back up.. it
> took
>  a looong.., with messages like:
>  
>  
>  2010-07-13 16:32:47,722 INFO
>  org.apache.hadoop.hdfs.DFSClient: Abandoning block
>  blk_-9203944767743886537_123926 
>  2010-07-13 16:32:48,207 DEBUG
>  org.apache.hadoop.hbase.regionserver.HLog: Waiting for
> hlog
>  writers to terminate, iteration #93 
>  2010-07-13 16:32:52,205 INFO
>  org.apache.hadoop.hdfs.DFSClient: Exception in
>  createBlockOutputStream java.io.EOFException 
>  2010-07-13 16:32:52,205 INFO
>  org.apache.hadoop.hdfs.DFSClient: Abandoning block
>  blk_-7742458689391305082_123926 
>  2010-07-13 16:32:53,072 INFO
>  org.apache.hadoop.hdfs.DFSClient: Exception in
>  createBlockOutputStream java.io.EOFException 
>  2010-07-13 16:32:53,072 INFO
>  org.apache.hadoop.hdfs.DFSClient: Abandoning block
>  blk_-281711885322381263_123926 
>  2010-07-13 16:32:53,207 DEBUG
>  org.apache.hadoop.hbase.regionserver.HLog: Waiting for
> hlog
>  writers to terminate, iteration #94 
>  2010-07-13 16:32:53,726 INFO
>  org.apache.hadoop.hdfs.DFSClient: Exception in
>  createBlockOutputStream java.io.IOException: Bad connect
> ack
>  with firstBadLink 192.168.193.155:50010 
>  2010-07-13 16:32:53,726 INFO
>  org.apache.hadoop.hdfs.DFSClient: Abandoning block
>  blk_-2730636850915812524_123926
>  
>  This just seems odd.. dumping that much standard output
> was
>  kind silly, but I wouldn't think it would wedge the
> cluster
>  like this... or did something else happen? Everything has
>  been fine for a while until I did this.
>  
>  I let the cluster sit around for a while, and it did come
>  back for a little while, after printing out this over and
> over (on
>  master):
>  
>  da7738a3ad9,1279039241464 from
>  ubuntu-hadoop-2,60020,1279066792598; 1 of 1
>  2010-07-13 17:28:30,897 INFO
>  org.apache.hadoop.hbase.master.ServerManager: Processing
>  MSG_REPORT_PROCESS_OPEN:
> 
> filemetadata,unknown_23bf6f189f81218a877a41d1ccd82751527abf19beb61281b8a0f6475e682764,1278533813300
>  from ubuntu-hadoop-4,60020,1279066792615; 1 of 3
>  2010-07-13 17:28:30,897 INFO
>  org.apache.hadoop.hbase.master.ServerManager: Processing
>  MSG_REPORT_PROCESS_OPEN:
> 
> filestore,7f45c6a2f5c7d09ae8618bf6d0db5b04f346c583a1dbf24d9db135f2c8da9b57,1279050466715
>  from ubuntu-hadoop-4,60020,1279066792615; 2 of 3
>  2010-07-13 17:28:30,897 INFO
>  org.apache.hadoop.hbase.master.ServerManager: Processing
>  MSG_REPORT_PROCESS_OPEN: .META.,,1 from
>  ubuntu-hadoop-4,60020,1279066792615; 3 of 3
>  
>  But regionservers kept coming down, and bringing down the
>  .META region.. eventually the master seemed to recover,
> and
>  I would bring the regionservers back up individually..
> one
>  of them would soon crash again, repeating the cycle. And
> the
>  master would would die with this on the web page:
>  
>  Trying to contact region server 192.168.193.155:60020 for
>  region .META.,,1, row '', but failed after 3 attempts.
>  Exceptions:
>  java.net.ConnectException: Connection refused
>  java.net.ConnectException: Connection refused
>  java.net.ConnectException: Connection refused
>  
>  RequestURI=/master.jsp
>  
>  The regionservers appear to start dying with this
>  statement:
>  
> 
>  010-07-13 02:12:59,257 DEBUG
>  org.apache.hadoop.hbase.regionserver.CompactSplitThread:
>  Compaction requested for region
> 
> filestore,5a1ec52137fb8bb38affb5ca034ec3817c6611adc2f5a2b399a4ab92effcdedd,1278970890717/1436533481
>  because: regionserver/192.168.193.145:60020.cacheFlusher
>  2010-07-13 02:13:19,336 WARN
>  org.apache.hadoop.hdfs.DFSClient: Failed to connect to
>  /192.168.193.145:50010 for file
>  /hbase/.META./1028785192/info/3836661676936096933 for
> block
>  -649923918996291451:java.io.EOFException
>          at
> 
> java.io.DataInputStream.readShort(DataInputStream.java:298)
>          at
> 
> org.apache.hadoop.hdfs.DFSClient$BlockReader.newBlockReader(DFSClient.java:1384)
>          at
> 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.fetchBlockByteRange(DFSClient.java:1853)
>          at
> 
> org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1922)
>          at
> 
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:46)
>          at
> 
> org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:101)
>  
>  
>  Note that it is failing to contact 192.168.193.145 ...
> this
>  is the ip of the node itself (the dying one).
>  
>  There's a long sequence of exceptions, and the
>  regionservers death ends with this:
>  
>  010-07-13 17:36:57,068 INFO
>  org.apache.zookeeper.ClientCnxn: EventThread shut down
>  2010-07-13 17:36:57,069 ERROR
>  org.apache.hadoop.hdfs.DFSClient: Exception closing file
> 
> /hbase/filestore/compaction.dir/1408434857/4172852130952668434
>  : java.io.EOFException
>  java.io.EOFException
>          at
> 
> java.io.DataInputStream.readByte(DataInputStream.java:250)
>          at
> 
> org.apache.hadoop.io.WritableUtils.readVLong(WritableUtils.java:298)
>          at
> 
> org.apache.hadoop.io.WritableUtils.readVInt(WritableUtils.java:319)
>          at
>  org.apache.hadoop.io.Text.readString(Text.java:400)
>          at
> 
> org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.createBlockOutputStream
>  
>  It just seems like a lot of trouble from one little
>  System.out.println() statement..
> 
>  Restarting hbase just repeats the cycle "Bad connect ack
> with firstBadLink", eventual recovery, regionservers start
> crashing, cluster  becomes unusable.. 
> 
>  In summary, it would seem printing out too much data to
>  stdout can render the whole cluster unstable - across
>  reboots - and I haven't figured out how to stabilize it
>  again. 
>  
>  Take care,
>    -stu
>  
>  PS. The address for the mailing list on the hbase page is
> wrong. It shows:
> 
> [email protected]
> But it should be
> [email protected].
> I hope! (If this mail goes through, I got it right).
> 
> 
>       
> 



Reply via email to