Table enable/disabled state is kept in zk. If you try to enable/disable root or meta in shell I believe they fail. I'm not sure how you managed it. We could try to do some forensics around time of disk fill.....
St.Ack On Thu, Jun 16, 2011 at 5:16 PM, N.N. Gesli <[email protected]> wrote: > Removing the zookeeper log, data, and snapshot files fixed the problem. > Thank you so much! By the way, I know nothing about disabling root. May be > it happened when namenode crushed?? Would it still keep it as disabled even > after /hbase removed? > > Thanks again. > > -Nicole > > On Thu, Jun 16, 2011 at 3:09 PM, Stack <[email protected]> wrote: > >> Try removing your zk log dir since you are starting over. If I read >> that right, it seems to be saying that the root region is in a >> disabling state. You know anything about that? I didn't think you >> could disable catalog regions but hey, you learn something new every >> day! >> >> St.Ack >> >> On Thu, Jun 16, 2011 at 2:34 PM, N.N. Gesli <[email protected]> wrote: >> > Thank you for your reply. I see some assignment issues in the log as >> > info/debug. Why would it report the log file as "belongs to an existing >> > region server"? I am removing /hbase; I even tried to use a different >> > directory than /hbase. The result is the same. What can I do? >> > >> > Thanks. >> > >> > >> > 2011-06-16 14:06:33,107 INFO >> org.apache.hadoop.hbase.master.ServerManager: >> > Waiting on regionserver(s) to checkin >> > 2011-06-16 14:06:33,894 INFO >> org.apache.hadoop.hbase.master.ServerManager: >> > Registering server=hadoop-hbase02.qa03.sea1.cmates.com >> ,60020,1308258393485, >> > regionCount=0, userLoad=false >> > 2011-06-16 14:06:34,609 INFO >> org.apache.hadoop.hbase.master.ServerManager: >> > Waiting on regionserver(s) count to settle; currently=1 >> > 2011-06-16 14:06:36,111 INFO >> org.apache.hadoop.hbase.master.ServerManager: >> > Finished waiting for regionserver count to settle; count=1, >> sleptFor=12000 >> > 2011-06-16 14:06:36,112 INFO >> org.apache.hadoop.hbase.master.ServerManager: >> > Exiting wait on regionserver(s) to checkin; count=1, stopped=false, count >> of >> > regions out on cluster=0 >> > 2011-06-16 14:06:36,116 INFO >> > org.apache.hadoop.hbase.master.MasterFileSystem: Log folder hdfs:// >> > >> cloud01.qa03.sea1.cmates.com:8020/hbase/.logs/hadoop-hbase02.qa03.sea1.cmates.com,60020,1308258393485belongs >> > to an existing region server >> > 2011-06-16 14:06:37,121 INFO >> > org.apache.hadoop.hbase.catalog.RootLocationEditor: Unsetting ROOT region >> > location in ZooKeeper >> > 2011-06-16 14:06:37,135 INFO >> > org.apache.hadoop.hbase.master.AssignmentManager: Table -ROOT- disabling; >> > skipping assign of -ROOT-,,0.70236052 >> > 2011-06-16 14:06:37,135 DEBUG >> > org.apache.hadoop.hbase.master.AssignmentManager: Table being disabled so >> > deleting ZK node and removing from regions in transition, skipping >> > assignment of region -ROOT-,,0.70236052 >> > 2011-06-16 14:06:37,136 DEBUG org.apache.hadoop.hbase.zookeeper.ZKAssign: >> > master:60000-0x2309a45ba860000 Deleting existing unassigned node for >> > 70236052 that is in expected state RS_ZK_REGION_CLOSED >> > 2011-06-16 14:06:37,142 DEBUG >> > org.apache.hadoop.hbase.master.AssignmentManager: Tried to delete closed >> > node for REGION => {NAME => '-ROOT-,,0', STARTKEY => '', ENDKEY => '', >> > ENCODED => 70236052, TABLE => {{NAME => '-ROOT-', IS_ROOT => 'true', >> IS_META >> > => 'true', FAMILIES => [{NAME => 'info', BLOOMFILTER => 'NONE', >> > REPLICATION_SCOPE => '0', COMPRESSION => 'NONE', VERSIONS => '10', TTL => >> > '2147483647', BLOCKSIZE => '8192', IN_MEMORY => 'true', BLOCKCACHE => >> > 'true'}]}} but it does not exist so just offlining >> > >> >
