Ok. Thanks. I don't know why its not assigning .META. Trying starting the cluster with a different hbase.rootdir then move the -ROOT- from the new hbase.rootdir in place of the -ROOT- you currently have.
St.Ack On Tue, Sep 21, 2010 at 4:17 PM, Stack <[email protected]> wrote: > Please put the first 1k lines after startup up in pastebin. The below > is hard to read. Its scanning the -ROOT- but there is no mention of > .META. being assigned out. > > St.Ack > > On Tue, Sep 21, 2010 at 3:09 PM, Matt Corgan <[email protected]> wrote: >> I think this is the relevant section: >> >> 2010-09-21 15:42:11,986 INFO org.apache.hadoop.hbase.master.ServerManager: >> Received start message from: HadoopNode41,60020,1285098132069 >> 2010-09-21 15:42:11,989 INFO org.apache.hadoop.hbase.master.ServerManager: >> Received start message from: HadoopNode38,60020,1285098131987 >> 2010-09-21 15:42:11,989 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode >> /hbase/rs/1285098132069 with data 10.214.7.223:60020 >> 2010-09-21 15:42:11,996 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode >> /hbase/rs/1285098131987 with data 10.209.206.15:60020 >> 2010-09-21 15:42:13,001 DEBUG org.apache.hadoop.hbase.master.RegionManager: >> Assigning for address: 10.209.206.15:60020, startcode: 1285098131987, load: >> (requests=0, regions=0, usedHeap=32, maxHeap=5195): total nregions to >> assign=1, regions to give other servers than this=0, isMetaAssign=true >> 2010-09-21 15:42:13,001 DEBUG org.apache.hadoop.hbase.master.RegionManager: >> Assigning address: 10.209.206.15:60020, startcode: 1285098131987, load: >> (requests=0, regions=0, usedHeap=32, maxHeap=5195) 1 regions >> 2010-09-21 15:42:13,001 INFO org.apache.hadoop.hbase.master.RegionManager: >> Assigning region -ROOT-,,0 to HadoopNode38,60020,1285098131987 >> 2010-09-21 15:42:13,085 INFO org.apache.hadoop.hbase.master.ServerManager: >> Processing MSG_REPORT_OPEN: -ROOT-,,0 from HadoopNode38,60020,1285098131987; >> 1 of 1 >> 2010-09-21 15:42:13,130 WARN >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create out of >> safe mode in ZooKeeper: >> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = >> NodeExists for /hbase/safe-mode >> 2010-09-21 15:42:13,130 INFO >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Node exists; just move >> on >> 2010-09-21 15:42:13,130 INFO org.apache.hadoop.hbase.master.RegionManager: >> exiting safe mode >> 2010-09-21 15:42:13,132 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: SetData of ZNode >> /hbase/root-region-server with 10.209.206.15:60020 >> 2010-09-21 15:42:13,133 INFO org.apache.hadoop.hbase.master.BaseScanner: >> RegionManager.rootScanner scanning meta region {server: 10.209.206.15:60020, >> regionname: -ROOT-,,0, startKey: <>} >> 2010-09-21 15:42:13,223 INFO org.apache.hadoop.hbase.master.BaseScanner: >> RegionManager.rootScanner scan of 0 row(s) of meta region {server: >> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete >> 2010-09-21 15:42:13,466 INFO org.apache.hadoop.hbase.master.ServerManager: >> Received start message from: HadoopNode40,60020,1285098133450 >> 2010-09-21 15:42:13,468 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode >> /hbase/rs/1285098133450 with data 10.192.66.240:60020 >> 2010-09-21 15:42:13,828 INFO org.apache.hadoop.hbase.master.ServerManager: >> Received start message from: HadoopNode39,60020,1285098133654 >> 2010-09-21 15:42:13,830 DEBUG >> org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode >> /hbase/rs/1285098133654 with data 10.206.26.95:60020 >> 2010-09-21 15:43:10,256 INFO org.apache.hadoop.hbase.master.ServerManager: 4 >> region servers, 0 dead, average load 0.25 >> 2010-09-21 15:43:10,341 INFO org.apache.hadoop.hbase.master.BaseScanner: >> RegionManager.rootScanner scanning meta region {server: 10.209.206.15:60020, >> regionname: -ROOT-,,0, startKey: <>} >> 2010-09-21 15:43:10,345 INFO org.apache.hadoop.hbase.master.BaseScanner: >> RegionManager.rootScanner scan of 0 row(s) of meta region {server: >> 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete >> 2010-09-21 15:43:10,354 INFO org.apache.hadoop.hbase.master.BaseScanner: All >> 0 .META. region(s) scanned >> 2010-09-21 15:43:56,351 INFO org.apache.zookeeper.ZooKeeper: Initiating >> client connection, connectString=10.249.107.223:2181 sessionTimeout=60000 >> watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@72898540 >> 2010-09-21 15:43:56,352 INFO org.apache.zookeeper.ClientCnxn: Attempting >> connection to server /10.249.107.223:2181 >> 2010-09-21 15:43:56,362 INFO org.apache.zookeeper.ClientCnxn: Priming >> connection to java.nio.channels.SocketChannel[connected local=/ >> 10.249.107.223:56828remote=/10.249.107.223:2181] >> 2010-09-21 15:43:56,380 INFO org.apache.zookeeper.ClientCnxn: Server >> connection successful >> >> >> On Tue, Sep 21, 2010 at 6:05 PM, Stack <[email protected]> wrote: >> >>> Ok. So .META. is there. What is in your master log on startup? Its >>> supposed to assign out -ROOT- and .META. (when .META. is assigned >>> it'll add row to -ROOT-). I'd say hold off on any of the trickery >>> described in the first mail till we see snippet from your master log. >>> Lets figure why not assigning .META. before we do any messing. >>> >>> Thanks, >>> St.Ack >>> >>> On Tue, Sep 21, 2010 at 2:52 PM, Matt Corgan <[email protected]> wrote: >>> > Thanks for responding Stack. In a 4 slave cluster (rf=3) cluster we >>> > disabled an hdfs/hbase node which also seemed to make a second >>> regionserver >>> > crash. I'm not sure exactly what went wrong. >>> > >>> > There is data in the .META. directory >>> > >>> > [r...@hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs -lsr >>> > /hbase/.META. >>> > drwxr-xr-x - root supergroup 0 2010-09-20 16:44 >>> > /hbase/.META./1028785192 >>> > -rw-r--r-- 3 root supergroup 932 2010-09-20 14:25 >>> > /hbase/.META./1028785192/.regioninfo >>> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41 >>> > /hbase/.META./1028785192/info >>> > -rw-r--r-- 3 root supergroup 5018 2010-09-21 15:41 >>> > /hbase/.META./1028785192/info/4694688683869779325 >>> > -rw-r--r-- 3 root supergroup 0 2010-09-21 14:48 >>> > /hbase/.META./1028785192/info/6832908118284006198 >>> > -rw-r--r-- 3 root supergroup 228130 2010-09-21 14:44 >>> > /hbase/.META./1028785192/info/7773537252698623585 >>> > drwxr-xr-x - root supergroup 0 2010-09-21 14:44 >>> > /hbase/.META./compaction.dir >>> > >>> > >>> > but it appears the -ROOT- region is empty. If i scan '-ROOT-' in the >>> hbase >>> > shell i get: >>> > >>> > hbase(main):001:0> scan '-ROOT-' >>> > ROW COLUMN+CELL >>> > 0 row(s) in 0.0840 seconds >>> > >>> > >>> > even though it appears -ROOT- exists in hdfs. Any idea why ROOT would >>> not >>> > get loaded from hdfs after a master restart? >>> > >>> > [r...@hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs -lsr >>> > /hbase/-ROOT- >>> > drwxr-xr-x - root supergroup 0 2010-09-21 14:50 >>> > /hbase/-ROOT-/70236052 >>> > -rw-r--r-- 3 root supergroup 618 2010-09-21 14:50 >>> > /hbase/-ROOT-/70236052/.regioninfo >>> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41 >>> > /hbase/-ROOT-/70236052/info >>> > -rw-r--r-- 3 root supergroup 489 2010-09-21 15:41 >>> > /hbase/-ROOT-/70236052/info/308400263036654323 >>> > -rw-r--r-- 3 root supergroup 462 2010-09-21 13:43 >>> > /hbase/-ROOT-/70236052/info/8250773031047005400 >>> > >>> > >>> > I'll just follow your suggestion of generating a new .META. and copying >>> > over. Thanks, >>> > >>> > Matt >>> > >>> > >>> > On Tue, Sep 21, 2010 at 5:20 PM, Stack <[email protected]> wrote: >>> > >>> >> On Tue, Sep 21, 2010 at 1:07 PM, Matt Corgan <[email protected]> >>> wrote: >>> >> > Is there a way to create a new .META. table if I lost it? For >>> example, >>> >> I'm >>> >> > getting this exception everywhere: >>> >> > >>> >> > org.apache.hadoop.hbase.TableNotFoundException: .META. >>> (NativeException) >>> >> > >>> >> >>> >> How'd you do that? Thats pretty impressive. >>> >> >>> >> > >>> >> > The add_table script doesn't appear to handle .META., giving this >>> error: >>> >> > >>> >> > java.lang.IllegalArgumentException: Illegal first character <46> at 0. >>> >> > User-space table names can only start with 'word characters': i.e. >>> >> > [a-zA-Z_0-9]: .META. >>> >> > >>> >> > >>> >> > And the hbase shell will only create user-space table names. >>> Restarting >>> >> the >>> >> > hbase master also does not appear to re-create it. >>> >> > >>> >> > I'm using 0.20.6. >>> >> > >>> >> Whats the master log look like when you start up hbase? Whats it say >>> >> about missing .META.? >>> >> >>> >> If you do a >>> >> >>> >> $ bin/hadoop fs -lsr /hbase/.META. >>> >> >>> >> ... is there anything in your filesystem? >>> >> >>> >> If nothing there, one thing you could do is run an hbase elsewhere. >>> >> Let it create a new "/hbase" dir and then copy the .META. table from >>> >> here up to the filesystem with data (Run add_table.rb after .META. is >>> >> in place to repopulate its rows). >>> >> >>> >> St.Ack >>> >> >>> > >>> >> >
