w00t! I looked in "/hbase/" on the hdfs and found directories referencing the stale tables and deleting them worked. It now says 0 inconsistencies. Maybe worth adding this to the hbck -repair script? Anyway, thanks everyone for the help! Hopefully this stops the sporadic crashes I was seeing.
-- Marco Gallotta | Mountain View, California Software Engineer, Infrastructure | Loki Studios fb.me/marco.gallotta | twitter.com/marcog [email protected] | +1 (650) 417-3313 Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Friday 10 August 2012 at 5:20 PM, Marco Gallotta wrote: > I never did that, but I mentioned above that all these come from when I ^C'ed > (i.e. killed) table creation when I was trying to get lzo compression working > and table creation was hanging. Where would I check for references to these > tables in ZK? > > -- > Marco Gallotta | Mountain View, California > Software Engineer, Infrastructure | Loki Studios > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog > (http://twitter.com/marcog) > [email protected] (mailto:[email protected]) | +1 (650) 417-3313 > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig) > > > On Friday 10 August 2012 at 5:15 PM, lars hofhansl wrote: > > > Did you blow away your cluster once (in the sense of the deleting all data > > and starting from scratch)?This could be left-over data in ZK. > > > > > > > > ----- Original Message ----- > > From: Marco Gallotta <[email protected] (mailto:[email protected])> > > To: [email protected] (mailto:[email protected]) > > Cc: > > Sent: Friday, August 10, 2012 2:22 PM > > Subject: Table listed in "list", but not in .META. > > > > Hi there > > > > I have a few tables which show up in a "list" in the shell, but produce > > "table not found" when performing any operation on them. There is no > > reference of them in the .META. table. It seems to be resulting in some of > > the hbase services being killed every so often. > > > > Here are some logs from master (foo is one of the tables not found): > > > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: > > Master server abort: loaded coprocessors are: [] > > 2012-08-09 20:40:44,301 FATAL org.apache.hadoop.hbase.master.HMaster: > > Unexpected state : foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. > > state=PENDING_OPEN, ts=1344570044277, > > server=ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 .. > > Cannot transit it to OFFLINE. > > > > > > There are also a number of the following types of error logs: > > > > 2012-08-09 20:10:04,308 ERROR > > org.apache.hadoop.hbase.master.AssignmentManager: Failed assignment in: > > ip-10-170-150-10.us-west-1.compute.internal,60020,1344559455110 due to > > org.apache.hadoop.hbase.regionserver.RegionAlreadyInTransitionException: > > Received:OPEN for the > > region:foo,,1343175078663.527bb34f4bb5e40dd42e82054d7c5485. ,which we are > > already trying to OPEN. > > > > Any ideas how to find and remove any references to these non-existent > > tables? > > > > -- > > Marco Gallotta | Mountain View, California > > Software Engineer, Infrastructure | Loki Studios > > fb.me/marco.gallotta (http://fb.me/marco.gallotta) | twitter.com/marcog > > (http://twitter.com/marcog) > > [email protected] (mailto:[email protected]) | +1 (650) 417-3313 > > > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig) >
