sorry - maybe this is better? http://pastebin.com/gerjBkMT
On Tue, Sep 21, 2010 at 6:09 PM, Matt Corgan <[email protected]> wrote: > I think this is the relevant section: > > 2010-09-21 15:42:11,986 INFO org.apache.hadoop.hbase.master.ServerManager: > Received start message from: HadoopNode41,60020,1285098132069 > 2010-09-21 15:42:11,989 INFO org.apache.hadoop.hbase.master.ServerManager: > Received start message from: HadoopNode38,60020,1285098131987 > 2010-09-21 15:42:11,989 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode > /hbase/rs/1285098132069 with data 10.214.7.223:60020 > 2010-09-21 15:42:11,996 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode > /hbase/rs/1285098131987 with data 10.209.206.15:60020 > 2010-09-21 15:42:13,001 DEBUG org.apache.hadoop.hbase.master.RegionManager: > Assigning for address: 10.209.206.15:60020, startcode: 1285098131987, > load: (requests=0, regions=0, usedHeap=32, maxHeap=5195): total nregions to > assign=1, regions to give other servers than this=0, isMetaAssign=true > 2010-09-21 15:42:13,001 DEBUG org.apache.hadoop.hbase.master.RegionManager: > Assigning address: 10.209.206.15:60020, startcode: 1285098131987, load: > (requests=0, regions=0, usedHeap=32, maxHeap=5195) 1 regions > 2010-09-21 15:42:13,001 INFO org.apache.hadoop.hbase.master.RegionManager: > Assigning region -ROOT-,,0 to HadoopNode38,60020,1285098131987 > 2010-09-21 15:42:13,085 INFO org.apache.hadoop.hbase.master.ServerManager: > Processing MSG_REPORT_OPEN: -ROOT-,,0 from HadoopNode38,60020,1285098131987; > 1 of 1 > 2010-09-21 15:42:13,130 WARN > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create out of > safe mode in ZooKeeper: > org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = > NodeExists for /hbase/safe-mode > 2010-09-21 15:42:13,130 INFO > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Node exists; just move > on > 2010-09-21 15:42:13,130 INFO org.apache.hadoop.hbase.master.RegionManager: > exiting safe mode > 2010-09-21 15:42:13,132 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: SetData of ZNode > /hbase/root-region-server with 10.209.206.15:60020 > 2010-09-21 15:42:13,133 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.rootScanner scanning meta region {server: > 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} > 2010-09-21 15:42:13,223 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.rootScanner scan of 0 row(s) of meta region {server: > 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete > 2010-09-21 15:42:13,466 INFO org.apache.hadoop.hbase.master.ServerManager: > Received start message from: HadoopNode40,60020,1285098133450 > 2010-09-21 15:42:13,468 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode > /hbase/rs/1285098133450 with data 10.192.66.240:60020 > 2010-09-21 15:42:13,828 INFO org.apache.hadoop.hbase.master.ServerManager: > Received start message from: HadoopNode39,60020,1285098133654 > 2010-09-21 15:42:13,830 DEBUG > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Updated ZNode > /hbase/rs/1285098133654 with data 10.206.26.95:60020 > 2010-09-21 15:43:10,256 INFO org.apache.hadoop.hbase.master.ServerManager: > 4 region servers, 0 dead, average load 0.25 > 2010-09-21 15:43:10,341 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.rootScanner scanning meta region {server: > 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} > 2010-09-21 15:43:10,345 INFO org.apache.hadoop.hbase.master.BaseScanner: > RegionManager.rootScanner scan of 0 row(s) of meta region {server: > 10.209.206.15:60020, regionname: -ROOT-,,0, startKey: <>} complete > 2010-09-21 15:43:10,354 INFO org.apache.hadoop.hbase.master.BaseScanner: > All 0 .META. region(s) scanned > 2010-09-21 15:43:56,351 INFO org.apache.zookeeper.ZooKeeper: Initiating > client connection, connectString=10.249.107.223:2181 sessionTimeout=60000 > watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@72898540 > 2010-09-21 15:43:56,352 INFO org.apache.zookeeper.ClientCnxn: Attempting > connection to server /10.249.107.223:2181 > 2010-09-21 15:43:56,362 INFO org.apache.zookeeper.ClientCnxn: Priming > connection to java.nio.channels.SocketChannel[connected local=/ > 10.249.107.223:56828remote=/10.249.107.223:2181] > 2010-09-21 15:43:56,380 INFO org.apache.zookeeper.ClientCnxn: Server > connection successful > > > > On Tue, Sep 21, 2010 at 6:05 PM, Stack <[email protected]> wrote: > >> Ok. So .META. is there. What is in your master log on startup? Its >> supposed to assign out -ROOT- and .META. (when .META. is assigned >> it'll add row to -ROOT-). I'd say hold off on any of the trickery >> described in the first mail till we see snippet from your master log. >> Lets figure why not assigning .META. before we do any messing. >> >> Thanks, >> St.Ack >> >> On Tue, Sep 21, 2010 at 2:52 PM, Matt Corgan <[email protected]> wrote: >> > Thanks for responding Stack. In a 4 slave cluster (rf=3) cluster we >> > disabled an hdfs/hbase node which also seemed to make a second >> regionserver >> > crash. I'm not sure exactly what went wrong. >> > >> > There is data in the .META. directory >> > >> > [r...@hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs >> -lsr >> > /hbase/.META. >> > drwxr-xr-x - root supergroup 0 2010-09-20 16:44 >> > /hbase/.META./1028785192 >> > -rw-r--r-- 3 root supergroup 932 2010-09-20 14:25 >> > /hbase/.META./1028785192/.regioninfo >> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41 >> > /hbase/.META./1028785192/info >> > -rw-r--r-- 3 root supergroup 5018 2010-09-21 15:41 >> > /hbase/.META./1028785192/info/4694688683869779325 >> > -rw-r--r-- 3 root supergroup 0 2010-09-21 14:48 >> > /hbase/.META./1028785192/info/6832908118284006198 >> > -rw-r--r-- 3 root supergroup 228130 2010-09-21 14:44 >> > /hbase/.META./1028785192/info/7773537252698623585 >> > drwxr-xr-x - root supergroup 0 2010-09-21 14:44 >> > /hbase/.META./compaction.dir >> > >> > >> > but it appears the -ROOT- region is empty. If i scan '-ROOT-' in the >> hbase >> > shell i get: >> > >> > hbase(main):001:0> scan '-ROOT-' >> > ROW COLUMN+CELL >> > 0 row(s) in 0.0840 seconds >> > >> > >> > even though it appears -ROOT- exists in hdfs. Any idea why ROOT would >> not >> > get loaded from hdfs after a master restart? >> > >> > [r...@hadoop Master(hadoopmaster):/mnt/hadoop-0.20.2] bin/hadoop fs >> -lsr >> > /hbase/-ROOT- >> > drwxr-xr-x - root supergroup 0 2010-09-21 14:50 >> > /hbase/-ROOT-/70236052 >> > -rw-r--r-- 3 root supergroup 618 2010-09-21 14:50 >> > /hbase/-ROOT-/70236052/.regioninfo >> > drwxr-xr-x - root supergroup 0 2010-09-21 15:41 >> > /hbase/-ROOT-/70236052/info >> > -rw-r--r-- 3 root supergroup 489 2010-09-21 15:41 >> > /hbase/-ROOT-/70236052/info/308400263036654323 >> > -rw-r--r-- 3 root supergroup 462 2010-09-21 13:43 >> > /hbase/-ROOT-/70236052/info/8250773031047005400 >> > >> > >> > I'll just follow your suggestion of generating a new .META. and copying >> > over. Thanks, >> > >> > Matt >> > >> > >> > On Tue, Sep 21, 2010 at 5:20 PM, Stack <[email protected]> wrote: >> > >> >> On Tue, Sep 21, 2010 at 1:07 PM, Matt Corgan <[email protected]> >> wrote: >> >> > Is there a way to create a new .META. table if I lost it? For >> example, >> >> I'm >> >> > getting this exception everywhere: >> >> > >> >> > org.apache.hadoop.hbase.TableNotFoundException: .META. >> (NativeException) >> >> > >> >> >> >> How'd you do that? Thats pretty impressive. >> >> >> >> > >> >> > The add_table script doesn't appear to handle .META., giving this >> error: >> >> > >> >> > java.lang.IllegalArgumentException: Illegal first character <46> at >> 0. >> >> > User-space table names can only start with 'word characters': i.e. >> >> > [a-zA-Z_0-9]: .META. >> >> > >> >> > >> >> > And the hbase shell will only create user-space table names. >> Restarting >> >> the >> >> > hbase master also does not appear to re-create it. >> >> > >> >> > I'm using 0.20.6. >> >> > >> >> Whats the master log look like when you start up hbase? Whats it say >> >> about missing .META.? >> >> >> >> If you do a >> >> >> >> $ bin/hadoop fs -lsr /hbase/.META. >> >> >> >> ... is there anything in your filesystem? >> >> >> >> If nothing there, one thing you could do is run an hbase elsewhere. >> >> Let it create a new "/hbase" dir and then copy the .META. table from >> >> here up to the filesystem with data (Run add_table.rb after .META. is >> >> in place to repopulate its rows). >> >> >> >> St.Ack >> >> >> > >> > >
