Good to hear that... :) I also had a very bad crash of HBase a few days ago... at that time had to run hbck repair to fix some errors and to fix others i had to remove files from /hbase dir. It's a fairly complex system so sometimes it really hard to nail the problem.
~Anil On Fri, Aug 10, 2012 at 8:21 PM, Marco Gallotta <[email protected]>wrote: > w00t! I looked in "/hbase/" on the hdfs and found directories referencing > the stale tables and deleting them worked. It now says 0 inconsistencies. > Maybe worth adding this to the hbck -repair script? Anyway, thanks everyone > for the help! Hopefully this stops the sporadic crashes I was seeing. > > -- > Marco Gallotta | Mountain View, California > Software Engineer, Infrastructure | Loki Studios > fb.me/marco.gallotta | twitter.com/marcog > [email protected] | +1 (650) 417-3313 > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig) > > > On Friday 10 August 2012 at 5:20 PM, Marco Gallotta wrote: > > > I never did that, but I mentioned above that all these come from when I > ^C'ed (i.e. killed) table creation when I was trying to get lzo compression > working and table creation was hanging. Where would I check for references > to these tables in ZK? > > > > -- > > Marco Gallotta | Mountain View, California > > Software Engineer, Infrastructure | Loki Studios > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog( > http://twitter.com/marcog) > > [email protected] (mailto:[email protected]) | +1 (650) 417-3313 > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig) > > > > > > On Friday 10 August 2012 at 5:15 PM, lars hofhansl wrote: > > > > > Did you blow away your cluster once (in the sense of the deleting all > data and starting from scratch)?This could be left-over data in ZK. > > > > > > > > > > > > ----- Original Message ----- > > > From: Marco Gallotta <[email protected] (mailto: > [email protected])> > > > To: [email protected] (mailto:[email protected]) > > > Cc: > > > Sent: Friday, August 10, 2012 2:22 PM > > > Subject: Table listed in "list", but not in .META. > > > > > > Hi there > > > > > > I have a few tables which show up in a "list" in the shell, but > produce "table not found" when performing any operation on them. There is > no reference of them in the .META. table. It seems to be resulting in some > of the hbase services being killed every so often. > > > > > > Here are some logs from master (foo is one of the tables not found): > > > > > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: > Master server abort: loaded coprocessors are: [] > > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: > Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. > state=PENDING_OPEN, ts=1344570044277, > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. > Cannot transit it to OFFLINE. > > > > > > > > > There are also a number of the following types of error logs: > > > > > > 2012-08-09 20:10:04,308 ERROR > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: > Received:OPEN for the > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are > already trying to OPEN. > > > > > > Any ideas how to find and remove any references to these non-existent > tables? > > > > > > -- > > > Marco Gallotta | Mountain View, California > > > Software Engineer, Infrastructure | Loki Studios > > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | > twitter.com/marcog (http://twitter.com/marcog) > > > [email protected] (mailto:[email protected]) | +1 (650) 417-3313 > > > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig) > > > > -- Thanks & Regards, Anil Gupta
