Your cluster seems fine. Item #1 above should probably not be WARN. It is over-sharing on what zk is up to during startup (#5 is related). #2 and #3 are just our logging that root and meta are not where zk says they are so we reassign which later in the log you'll see succeeds; this is just how we do startup (zk has last locations of these regions. When the master comes up it probes to see if these locations are still good...). #4 is complaining about a change in a metric after startup. See http://hbase.apache.org/0.94/xref/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.html#710 It explains why we log as ERROR. Agree it disturbing. File an issue if you are not doing anything here in this regard.
Thanks, St.Ack On Fri, Mar 21, 2014 at 12:23 AM, Vimal Jain <[email protected]> wrote: > Hi, > I could not stop myself from restarting the hbase cluster. :( > My predictions were correct > I restarted cluster and now I am getting those exceptions. > > logs after restarting hbase cluster. > > Zookeeper - http://pastebin.com/UBbbiERk > Master - http://pastebin.com/6GZH5hZK > Region - http://pastebin.com/Fi38vUeS > > > Please help in understanding this and let me know if there is any > confusion. > > > On Fri, Mar 21, 2014 at 12:38 PM, Vimal Jain <[email protected]> wrote: > > > Hi Stack/Lars, > > I configured Hbase again in pseudo distributed mode. > > Again i got some exceptions which have shared in pastebin > > > > zookeeper log - http://pastebin.com/6vgbm987 > > master log - http://pastebin.com/kcjpd1Zp > > region log - http://pastebin.com/vmfE2HG2 > > > > There is no error in all 3 hadoop processes. > > And i am sure if i restart the hbase , i will get errors like 'Region is > > not online: -ROOT-' , 'Region is not online: .META' > > This is the pattern i have observed. > > > > Let me know if you want me to restart the hbase cluster and then i will > > attach fresh logs. > > > > Hbase - 0.94.17 and Hadoop 1.2.1 > > > > > > On Fri, Mar 21, 2014 at 11:47 AM, lars hofhansl <[email protected]> > wrote: > > > >> I do this *all* the time, I have never seen an issue like this. So this > >> is interesting. > >> Is it possible that ZK happened to have picked a use ephemeral port? > (but > >> in that case I would have only expected this to fail once and work the > next > >> time). > >> > >> -- Lars > >> > >> > >> > >> ________________________________ > >> From: Vimal Jain <[email protected]> > >> To: "[email protected]" <[email protected]> > >> Sent: Thursday, March 20, 2014 11:10 PM > >> Subject: Re: Getting exceptions in fresh installment of hbase > >> > >> > >> Hi Stack, > >> I will delete everything and reconfigure hbase from scratch. > >> I will then attach logs in pastebin if i find same issues again. > >> > >> > >> On Fri, Mar 21, 2014 at 11:20 AM, Stack <[email protected]> wrote: > >> > >> > Put the full logs up in pastebin. The below seem like symptoms, not > the > >> > problem. > >> > St.Ack > >> > > >> > > >> > On Thu, Mar 20, 2014 at 10:12 PM, Vimal Jain <[email protected]> > wrote: > >> > > >> > > Yes Stack. > >> > > > >> > > > >> > > On Fri, Mar 21, 2014 at 9:22 AM, Stack <[email protected]> wrote: > >> > > > >> > > > On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <[email protected]> > >> wrote: > >> > > > > >> > > > > Hi, > >> > > > > I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo > >> > > distributed > >> > > > > mode and i am getting below exceptions when starting hbase for > >> first > >> > > > time. > >> > > > > > >> > > > > > >> > > > You started fresh? Fresh HDFS and ZK in particular? > >> > > > St.Ack > >> > > > > >> > > > > >> > > > > >> > > > > Master logs :- > >> > > > > > >> > > > > 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: > >> Node > >> > > > > /hbase/backup-m > >> > > > > asters/hbase.rummycircle.com,60000,1395314029900 already > deleted, > >> > and > >> > > > this > >> > > > > is not a retry > >> > > > > > >> > > > > 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed > >> > verification > >> > > of > >> > > > > -ROOT > >> > > > > -,,0 at address=hbase.rummycircle.com,60020,1395313673569; > >> > > > > org.apache.hadoop.hbase.NotServingRegionException: > >> > > > > org.apache.hadoop.hbase.N > >> > > > > otServingRegionException: Region is not online: -ROOT-,,0 > >> > > > > > >> > > > > 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed > >> > verification > >> > > of > >> > > > > .META > >> > > > > .,,1 at address=hbase.rummycircle.com,60020,1395313673569; > >> > > > > org.apache.hadoop.hbase.NotServingRegionException: > >> > > > > org.apache.hadoop.hbase.N > >> > > > > otServingRegionException: Region is not online: .META.,,1 > >> > > > > > >> > > > > 4) ERROR > >> org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics: > >> > > > > Inconsist > >> > > > > ent configuration. Previous configuration for using table name > in > >> > > > metrics: > >> > > > > true, new configuration: false > >> > > > > > >> > > > > 5) Got user-level KeeperEx > >> > > > > ception when processing sessionid:0x144df32705d0000 type:delete > >> > > cxid:0xc > >> > > > > zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast > >> > > > > ers/hbase.rummycircle.com,60000,1395314029900 > >> Error:KeeperErrorCode > >> > = > >> > > > > NoNode for /hbase/backup-masters/hbase.rummycircle.com > ,60000,1395 > >> > > > > 314029900 > >> > > > > > >> > > > > 3rd and 5th ones getting repeated multiple times. > >> > > > > > >> > > > > Please help. > >> > > > > > >> > > > > > >> > > > > > >> > > > > -- > >> > > > > Thanks and Regards, > >> > >> > > > > Vimal Jain > >> > > > > > >> > > > > >> > > > >> > > > >> > > > >> > > -- > >> > > Thanks and Regards, > >> > > Vimal Jain > >> > > > >> > > >> > >> > >> > >> -- > >> Thanks and Regards, > >> Vimal Jain > >> > > > > > > > > -- > > Thanks and Regards, > > Vimal Jain > > > > > > -- > Thanks and Regards, > Vimal Jain >
