Hi, One more issue i found is that zookeeper logs are not getting rotated. I started cluster 3-4 days back and still only master and region server logs are rotated properly. Zookeeper logs are still appended to "hbase-hadoop-zookeeper-watto.log". Please help on this and also on whether to use hadoop 1.2.1 or hadop 1.1.2 ?
On Sat, Mar 22, 2014 at 11:15 AM, Vimal Jain <[email protected]> wrote: > Stack, > Thanks a lot for the analysis , so nothing to worry about ? > Lars, > I did not built hbase from source. > I simply downloaded tars from hbase and hadoop download page( hbase from > "stable" directory ( i.e 0.94.17 ) and hadoop from "stable1" directory ( > i.e. 1.2.1) ). > Do i need to build from source ? > If yes , is there any guidelines in doing same as i have never used Maven > ( i use Ant for build and deployment ). > > > On Sat, Mar 22, 2014 at 5:22 AM, lars hofhansl <[email protected]> wrote: > >> Hi Vimal, >> >> did you rebuild HBase against Hadoop 1.2.1? The released tarballs are >> built against Hadoop 1.0.4 and won't work with Hadoop 1.2.x or 2.x.y. >> >> -- Lars >> >> ------------------------------ >> *From:* Vimal Jain <[email protected]> >> *To:* "[email protected]" <[email protected]>; lars hofhansl < >> [email protected]> >> *Sent:* Friday, March 21, 2014 12:08 AM >> >> *Subject:* Re: Getting exceptions in fresh installment of hbase >> >> Hi Stack/Lars, >> I configured Hbase again in pseudo distributed mode. >> Again i got some exceptions which have shared in pastebin >> >> zookeeper log - http://pastebin.com/6vgbm987 >> master log - http://pastebin.com/kcjpd1Zp >> region log - http://pastebin.com/vmfE2HG2 >> >> There is no error in all 3 hadoop processes. >> And i am sure if i restart the hbase , i will get errors like 'Region is >> not online: -ROOT-' , 'Region is not online: .META' >> This is the pattern i have observed. >> >> Let me know if you want me to restart the hbase cluster and then i will >> attach fresh logs. >> >> Hbase - 0.94.17 and Hadoop 1.2.1 >> >> >> On Fri, Mar 21, 2014 at 11:47 AM, lars hofhansl <[email protected]> wrote: >> >> I do this *all* the time, I have never seen an issue like this. So this >> is interesting. >> Is it possible that ZK happened to have picked a use ephemeral port? (but >> in that case I would have only expected this to fail once and work the next >> time). >> >> -- Lars >> >> >> >> ________________________________ >> From: Vimal Jain <[email protected]> >> To: "[email protected]" <[email protected]> >> Sent: Thursday, March 20, 2014 11:10 PM >> Subject: Re: Getting exceptions in fresh installment of hbase >> >> >> Hi Stack, >> I will delete everything and reconfigure hbase from scratch. >> I will then attach logs in pastebin if i find same issues again. >> >> >> On Fri, Mar 21, 2014 at 11:20 AM, Stack <[email protected]> wrote: >> >> > Put the full logs up in pastebin. The below seem like symptoms, not the >> > problem. >> > St.Ack >> > >> > >> > On Thu, Mar 20, 2014 at 10:12 PM, Vimal Jain <[email protected]> wrote: >> > >> > > Yes Stack. >> > > >> > > >> > > On Fri, Mar 21, 2014 at 9:22 AM, Stack <[email protected]> wrote: >> > > >> > > > On Thu, Mar 20, 2014 at 4:19 AM, Vimal Jain <[email protected]> >> wrote: >> > > > >> > > > > Hi, >> > > > > I am configuring hbase ( 0.94.17 and hadoop 1.2.1 ) in pseudo >> > > distributed >> > > > > mode and i am getting below exceptions when starting hbase for >> first >> > > > time. >> > > > > >> > > > > >> > > > You started fresh? Fresh HDFS and ZK in particular? >> > > > St.Ack >> > > > >> > > > >> > > > >> > > > > Master logs :- >> > > > > >> > > > > 1) WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: >> Node >> > > > > /hbase/backup-m >> > > > > asters/hbase.rummycircle.com,60000,1395314029900 already deleted, >> > and >> > > > this >> > > > > is not a retry >> > > > > >> > > > > 2) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed >> > verification >> > > of >> > > > > -ROOT >> > > > > -,,0 at address=hbase.rummycircle.com,60020,1395313673569; >> > > > > org.apache.hadoop.hbase.NotServingRegionException: >> > > > > org.apache.hadoop.hbase.N >> > > > > otServingRegionException: Region is not online: -ROOT-,,0 >> > > > > >> > > > > 3) org.apache.hadoop.hbase.catalog.CatalogTracker: Failed >> > verification >> > > of >> > > > > .META >> > > > > .,,1 at address=hbase.rummycircle.com,60020,1395313673569; >> > > > > org.apache.hadoop.hbase.NotServingRegionException: >> > > > > org.apache.hadoop.hbase.N >> > > > > otServingRegionException: Region is not online: .META.,,1 >> > > > > >> > > > > 4) ERROR >> org.apache.hadoop.hbase.regionserver.metrics.SchemaMetrics: >> > > > > Inconsist >> > > > > ent configuration. Previous configuration for using table name in >> > > > metrics: >> > > > > true, new configuration: false >> > > > > >> > > > > 5) Got user-level KeeperEx >> > > > > ception when processing sessionid:0x144df32705d0000 type:delete >> > > cxid:0xc >> > > > > zxid:0x53 txntype:-1 reqpath:n/a Error Path:/hbase/backup-mast >> > > > > ers/hbase.rummycircle.com,60000,1395314029900 >> Error:KeeperErrorCode >> > = >> > > > > NoNode for /hbase/backup-masters/hbase.rummycircle.com,60000,1395 >> > > > > 314029900 >> > > > > >> > > > > 3rd and 5th ones getting repeated multiple times. >> > > > > >> > > > > Please help. >> > > > > >> > > > > >> > > > > >> > > > > -- >> > > > > Thanks and Regards, >> >> > > > > Vimal Jain >> > > > > >> > > > >> > > >> > > >> > > >> > > -- >> > > Thanks and Regards, >> > > Vimal Jain >> > > >> > >> >> >> >> -- >> Thanks and Regards, >> Vimal Jain >> >> >> >> >> -- >> Thanks and Regards, >> Vimal Jain >> >> >> > > > -- > Thanks and Regards, > Vimal Jain > -- Thanks and Regards, Vimal Jain
