We're running in distributed mode. The rootdir is hdfs://s-hbase-master/hbase. The HDFS files appear to be intact (we'll try to recover the tables tomorrow). We are running java "1.6.0_17-b04". We're attempting to reproduce the problem and will do a thread dump if successful.
Thanks, Joshua On Fri, Oct 7, 2011 at 11:53 PM, Stack <[email protected]> wrote: > Is this standalone? Are you using default /tmp location for your data? > > Please thread dump your hung server the next time. Save a jstack of > the process or send a kill -QUIT PID and send us the .out file. > > What JVM? > > Thanks, > St.Ack > > On Fri, Oct 7, 2011 at 11:58 AM, Joshua Napoli <[email protected]> > wrote: > > I'm using HBase 0.90.4 and encountered a problem where: > > > > 1. HBase became unable to processes requests. REST client requests > > time-out and HBase commands from HBase shell would hang. > > 2. After restarting the cluster, almost all of my tables were missing > > (but the HDFS files remain). > > > > > > While the cluster was hung, the REST logs were repeating (for the same > > tables over and over): > > "2011-10-05 10:09:54,801 DEBUG org.apache.hadoop.hbase.client.HBaseAdmin: > > Sleeping= 32000ms, waiting for all regions to be disabled in ...". > > > > The messages list tables that were meant for deletion. However, many > other > > tables (not listed for disable/delete in the logs) were lost. > > > > What could cause the loss of table information? What is the proper > sequence > > for restarting a cluster that is no longer able to process HBase > requests? > > > > > > Thanks, > > Josh > > >
