Thanks Stack - worked like a charm, and I think all my data is there. So does the content of -ROOT- get overwritten, because I think it was different than it should have been?
On Tue, Sep 21, 2010 at 7:25 PM, Stack <[email protected]> wrote: > Ok. Thanks. I don't know why its not assigning .META. Trying > starting the cluster with a different hbase.rootdir then move the > -ROOT- from the new hbase.rootdir in place of the -ROOT- you currently > have. > > St.Ack > > On Tue, Sep 21, 2010 at 4:17 PM, Stack <[email protected]> wrote: > > Please put the first 1k lines after startup up in pastebin. The below > > is hard to read. Its scanning the -ROOT- but there is no mention of > > .META. being assigned out. > > > > St.Ack > > > > On Tue, Sep 21, 2010 at 3:09 PM, Matt Corgan <[email protected]> > wrote: > >> I think this is the relevant section: > >> > >> 2010-09-21 15:42:11,986 INFO > org.apache.hadoop.hbase.master.ServerManager: > >> Received start message from: HadoopNode41,60020,1285098132069 > >> 2010-09-21 15:42:11,989 INFO > org.apache.hadoop.hbase.master.ServerManager: > >> Received start message from: HadoopNode38,60020,1285098131987 > >> 2010-09-21 15:42:11,989 DEBUG > >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode > >> /hbase/rs/1285098132069 with data 10.214.7.223:60020 > >> 2010-09-21 15:42:11,996 DEBUG > >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode > >> /hbase/rs/1285098131987 with data 10.209.206.15:60020 > >> 2010-09-21 15:42:13,001 DEBUG > org.apache.hadoop.hbase.master.RegionManager: > >> Assigning for address: 10.209.206.15:60020, startcode: 1285098131987, > load: > >> (requests=0, regions=0, usedHeap=32, maxHeap=5195): total nregions to > >> assign=1, regions to give other servers than this=0, isMetaAssign=true > >> 2010-09-21 15:42:13,001 DEBUG > org.apache.hadoop.hbase.master.RegionManager: > >> Assigning address: 10.209.206.15:60020, startcode: 1285098131987, load: > >> (requests=0, regions=0, usedHeap=32, maxHeap=5195) 1 regions > >> 2010-09-21 15:42:13,001 INFO > org.apache.hadoop.hbase.master.RegionManager: > >> Assigning region -ROOT-,,0 to HadoopNode38,60020,1285098131987 > >> 2010-09-21 15:42:13,085 INFO > org.apache.hadoop.hbase.master.ServerManager: > >> Processing MSG_REPORT_OPEN: -ROOT-,,0 from > HadoopNode38,60020,1285098131987; > >> 1 of 1 > >> 2010-09-21 15:42:13,130 WARN > >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create out > of > >> safe mode in ZooKeeper: > >> org.apache.zookeeper.KeeperException$NodeExistsException: > KeeperErrorCode = > >> NodeExists for /hbase/safe-mode > >> 2010-09-21 15:42:13,130 INFO > >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Node exists; just > move > >> on > >> 2010-09-21 15:42:13,130 INFO > org.apache.hadoop.hbase.master.RegionManager: > >> exiting safe mode > >> 2010-09-21 15:42:13,132 DEBUG > >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: SetData of ZNode > >> /hbase/root-region-server with 10.209.206.15:60020 > >> 2010-09-21 15:42:13,133 INFO org.apache.hadoop.hbase.master.BaseScanner: > >> RegionManager.rootScanner scanning meta region {server: > 10.209.206.15:60020, > >> regionname: -ROOT-,,0, startKey: <>} > >> 2010-09-21 15:42:13,223 INFO org.apache.hadoop.hbase.master.BaseScanner: > >> RegionManager.rootScanner scan of 0 row(s) of meta region {server: > >> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete > >> 2010-09-21 15:42:13,466 INFO > org.apache.hadoop.hbase.master.ServerManager: > >> Received start message from: HadoopNode40,60020,1285098133450 > >> 2010-09-21 15:42:13,468 DEBUG > >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode > >> /hbase/rs/1285098133450 with data 10.192.66.240:60020 > >> 2010-09-21 15:42:13,828 INFO > org.apache.hadoop.hbase.master.ServerManager: > >> Received start message from: HadoopNode39,60020,1285098133654 > >> 2010-09-21 15:42:13,830 DEBUG > >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode > >> /hbase/rs/1285098133654 with data 10.206.26.95:60020 > >> 2010-09-21 15:43:10,256 INFO > org.apache.hadoop.hbase.master.ServerManager: 4 > >> region servers, 0 dead, average load 0.25 > >> 2010-09-21 15:43:10,341 INFO org.apache.hadoop.hbase.master.BaseScanner: > >> RegionManager.rootScanner scanning meta region {server: > 10.209.206.15:60020, > >> regionname: -ROOT-,,0, startKey: <>} > >> 2010-09-21 15:43:10,345 INFO org.apache.hadoop.hbase.master.BaseScanner: > >> RegionManager.rootScanner scan of 0 row(s) of meta region {server: > >> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete > >> 2010-09-21 15:43:10,354 INFO org.apache.hadoop.hbase.master.BaseScanner: > All > >> 0 .META. region(s) scanned > >> 2010-09-21 15:43:56,351 INFO org.apache.zookeeper.ZooKeeper: Initiating > >> client connection, connectString=10.249.107.223:2181sessionTimeout=60000 > >> > watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@72898540 > >> 2010-09-21 15:43:56,352 INFO org.apache.zookeeper.ClientCnxn: Attempting > >> connection to server /10.249.107.223:2181 > >> 2010-09-21 15:43:56,362 INFO org.apache.zookeeper.ClientCnxn: Priming > >> connection to java.nio.channels.SocketChannel[connected local=/ > >> 10.249.107.223:56828remote=/10.249.107.223:2181] > >> 2010-09-21 15:43:56,380 INFO org.apache.zookeeper.ClientCnxn: Server > >> connection successful > >> > >> > >> On Tue, Sep 21, 2010 at 6:05 PM, Stack <[email protected]> wrote: > >> > >>> Ok. So .META. is there. What is in your master log on startup? Its > >>> supposed to assign out -ROOT- and .META. (when .META. is assigned > >>> it'll add row to -ROOT-). I'd say hold off on any of the trickery > >>> described in the first mail till we see snippet from your master log. > >>> Lets figure why not assigning .META. before we do any messing. > >>> > >>> Thanks, > >>> St.Ack > >>> > >>> On Tue, Sep 21, 2010 at 2:52 PM, Matt Corgan <[email protected]> > wrote: > >>> > Thanks for responding Stack. In a 4 slave cluster (rf=3) cluster we > >>> > disabled an hdfs/hbase node which also seemed to make a second > >>> regionserver > >>> > crash. I'm not sure exactly what went wrong. > >>> > > >>> > There is data in the .META. directory > >>> > > >>> > [r...@hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs > -lsr > >>> > /hbase/.META. > >>> > drwxr-xr-x - root supergroup 0 2010-09-20 16:44 > >>> > /hbase/.META./1028785192 > >>> > -rw-r--r-- 3 root supergroup 932 2010-09-20 14:25 > >>> > /hbase/.META./1028785192/.regioninfo > >>> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41 > >>> > /hbase/.META./1028785192/info > >>> > -rw-r--r-- 3 root supergroup 5018 2010-09-21 15:41 > >>> > /hbase/.META./1028785192/info/4694688683869779325 > >>> > -rw-r--r-- 3 root supergroup 0 2010-09-21 14:48 > >>> > /hbase/.META./1028785192/info/6832908118284006198 > >>> > -rw-r--r-- 3 root supergroup 228130 2010-09-21 14:44 > >>> > /hbase/.META./1028785192/info/7773537252698623585 > >>> > drwxr-xr-x - root supergroup 0 2010-09-21 14:44 > >>> > /hbase/.META./compaction.dir > >>> > > >>> > > >>> > but it appears the -ROOT- region is empty. If i scan '-ROOT-' in the > >>> hbase > >>> > shell i get: > >>> > > >>> > hbase(main):001:0> scan '-ROOT-' > >>> > ROW COLUMN+CELL > >>> > 0 row(s) in 0.0840 seconds > >>> > > >>> > > >>> > even though it appears -ROOT- exists in hdfs. Any idea why ROOT > would > >>> not > >>> > get loaded from hdfs after a master restart? > >>> > > >>> > [r...@hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs > -lsr > >>> > /hbase/-ROOT- > >>> > drwxr-xr-x - root supergroup 0 2010-09-21 14:50 > >>> > /hbase/-ROOT-/70236052 > >>> > -rw-r--r-- 3 root supergroup 618 2010-09-21 14:50 > >>> > /hbase/-ROOT-/70236052/.regioninfo > >>> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41 > >>> > /hbase/-ROOT-/70236052/info > >>> > -rw-r--r-- 3 root supergroup 489 2010-09-21 15:41 > >>> > /hbase/-ROOT-/70236052/info/308400263036654323 > >>> > -rw-r--r-- 3 root supergroup 462 2010-09-21 13:43 > >>> > /hbase/-ROOT-/70236052/info/8250773031047005400 > >>> > > >>> > > >>> > I'll just follow your suggestion of generating a new .META. and > copying > >>> > over. Thanks, > >>> > > >>> > Matt > >>> > > >>> > > >>> > On Tue, Sep 21, 2010 at 5:20 PM, Stack <[email protected]> wrote: > >>> > > >>> >> On Tue, Sep 21, 2010 at 1:07 PM, Matt Corgan <[email protected]> > >>> wrote: > >>> >> > Is there a way to create a new .META. table if I lost it? For > >>> example, > >>> >> I'm > >>> >> > getting this exception everywhere: > >>> >> > > >>> >> > org.apache.hadoop.hbase.TableNotFoundException: .META. > >>> (NativeException) > >>> >> > > >>> >> > >>> >> How'd you do that? Thats pretty impressive. > >>> >> > >>> >> > > >>> >> > The add_table script doesn't appear to handle .META., giving this > >>> error: > >>> >> > > >>> >> > java.lang.IllegalArgumentException: Illegal first character <46> > at 0. > >>> >> > User-space table names can only start with 'word characters': i.e. > >>> >> > [a-zA-Z_0-9]: .META. > >>> >> > > >>> >> > > >>> >> > And the hbase shell will only create user-space table names. > >>> Restarting > >>> >> the > >>> >> > hbase master also does not appear to re-create it. > >>> >> > > >>> >> > I'm using 0.20.6. > >>> >> > > >>> >> Whats the master log look like when you start up hbase? Whats it > say > >>> >> about missing .META.? > >>> >> > >>> >> If you do a > >>> >> > >>> >> $ bin/hadoop fs -lsr /hbase/.META. > >>> >> > >>> >> ... is there anything in your filesystem? > >>> >> > >>> >> If nothing there, one thing you could do is run an hbase elsewhere. > >>> >> Let it create a new "/hbase" dir and then copy the .META. table from > >>> >> here up to the filesystem with data (Run add_table.rb after .META. > is > >>> >> in place to repopulate its rows). > >>> >> > >>> >> St.Ack > >>> >> > >>> > > >>> > >> > > >
