Actually it happened when I did a test to manually reboot one of the region servers. Note that the region server I rebooted was the one that contained -ROOT-, &.META as online regions. After I rebooted the server, the cluster crashed. I had to reboot the entire cluster , hoping that the cluster would fix this itself. But to my disappointed, the cluster didn't recover and the log showed the error which have been listed in this mail. Speaking of this, do you know anything about my problem?shouldn't the -ROOT-, &.META keep synchronized among all the servers? Please contact with me if u are interested in this issue. Much Thanks.
2011/7/29 Gan, Xiyun <[email protected]> > Following is my understanding. Please correct me if there are any mistakes. > > Did you change Zookeeper quorum configuration? > Zookeeper tracks table schema and regionservers. If ZK quorum changes, > the data Zookeeper saved is corrupted since the integration of the > data has been destroyed. > You may see some snippets from the log which says the cluster can't > connect to a previously selected ZK node. > > As far as I know, table data will not be lost after cleaning up the > zookeeper data, but I'm not sure. > > On Fri, Jul 29, 2011 at 11:33 AM, Xian Woo <[email protected]> wrote: > > Wow,it really helps~~ Thanks pal~~ > > But in fact, I really don't understand why this issue can be solved just > by > > cleaning up the zookeeper data on each zk node. So I really hope you can > > tell me something more about that. Thanks~ > > Best wishes~ > > Woo > > 2011/7/29 Gan, Xiyun <[email protected]> > >> > >> Hi, Woo > >> I have been confronted with this issue in our cluster. Have you > >> started hbase cluster successfully ever before? If yes, try to clean > >> up the zookeeper data on each zk node. > >> > >> > >> On Fri, Jul 29, 2011 at 1:03 AM, Xian Woo <[email protected]> > wrote: > >> > Excuse me for clicking the 'send' button accidently.. > >> > > >> > Log in Server4.yun.com: > >> > Here are some segments: > >> > > >> > 2011-07-28 20:48:41,123 DEBUG > >> > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 > >> > MB, > >> > free=197.54 MB, max=199.18 MB, blocks=0, accesses=0, hits=0, > >> > hitRatio=�%, > >> > cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%, evictions=0, > >> > evicted=0, evictedPerRun=NaN > >> > > >> > 2011-07-28 20:38:41,109 INFO org.apache.hadoop.ipc.HBaseServer: IPC > >> > Server > >> > handler 2 on 60020: starting > >> > > >> > 2011-07-28 20:38:40,598 DEBUG > >> > org.apache.hadoop.hbase.executor.ExecutorService: Starting executor > >> > service > >> > name=RS_OPEN_ROOT-server4.yun.com,60020,1311856719820, > corePoolSize=1, > >> > maxPoolSize=1 > >> > > >> > 2011-07-28 20:38:40,261 INFO > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every > >> > 10000000ms > >> > > >> > > >> > Log in server5.yun.com: > >> > > >> > 1.It first keeps showing: > >> > > >> > 2011-07-28 20:37:05,616 DEBUG > >> > org.apache.hadoop.hbase.regionserver.HRegionServer: > >> > NotServingRegionException; Region is not online: -ROOT-,,0 > >> > > >> > 2. Then it writes this per 5 minutes: > >> > > >> > 2011-07-28 20:40:52,814 DEBUG > >> > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 > >> > MB, > >> > free=197.54 MB, max=199.18 MB, blocks=0, accesses=0, hits=0, > >> > hitRatio=锟?, > >> > cachingAccesses=0, cachingHits=0, cachingHitsRatio=锟?, evictions=0, > >> > evicted=0, evictedPerRun=NaN > >> > > >> > > >> > I would really appreciate it if you can give me some advice on this > >> > issue. > >> > > >> > Thanks a lot ~ > >> > > >> > Best Wishes~ > >> > > >> > Woo > >> > > >> > > >> > > >> > > >> > > >> > > >> > > >> > 2011/7/29 Xian Woo <[email protected]> > >> > > >> >> > >> >> Hi everyone. I have just completed in installing my hbase and there > are > >> >> some errors when I try to start my cluster. > >> >> > >> >> Setup: > >> >> -cdh3u1 > >> >> - Hadoop 0.20.2 > >> >> - HBase 0.90.1 > >> >> - 1 Master Node running as NameNode & JobTracker(named > >> >> server3.yun.com) > >> >> -zookeeper quorum > >> >> - 2 child nodes(server4.yun.com & server5.yun.com) running as > >> >> Datanode, > >> >> TaskTracker and RegionServer each. > >> >> > >> >> When I try to start the master and the region servers, I find that > some > >> >> issues in my log: > >> >> > >> >> Master log(server3.yun.com): > >> >> > >> >> 1. It first keeps showing this: > >> >> "2011-07-28 20:30:16,056 INFO > >> >> org.apache.hadoop.hbase.master.AssignmentManager: Region has been > >> >> OPENING > >> >> for too long, reassigning region=-ROOT-,,0.70236052 > >> >> 2011-07-28 20:30:16,057 DEBUG > >> >> org.apache.hadoop.hbase.master.AssignmentManager: Region has > >> >> transitioned to > >> >> OPENED, allowing watched event handlers to process" > >> >> > >> >> 2.After several minutes of the same contents written in the log, > then: > >> >> "2011-07-28 20:30:16,412 DEBUG > >> >> > >> >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > >> >> Lookedup root region location, > >> >> > >> >> > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@7e24d517 > ; > >> >> hsa=server5.yun.com:60020 > >> >> 2011-07-28 20:30:16,413 DEBUG > >> >> > >> >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > >> >> locateRegionInMeta parentTable=-ROOT-, metaLocation=address: > >> >> server5.yun.com:60020, regioninfo: -ROOT-,,0.70236052, attempt=1 of > 10 > >> >> failed; retrying after sleep of 1000 because: > >> >> org.apache.hadoop.hbase.NotServingRegionException: Region is not > >> >> online: > >> >> -ROOT-,,0 > >> >> at > >> >> > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2318) > >> >> at > >> >> > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1614) > >> >> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > >> >> at > >> >> > >> >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> >> at java.lang.reflect.Method.invoke(Method.java:597) > >> >> at > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) > >> >> at > >> >> > >> >> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) > >> >> " > >> >> > >> >> 3.Then it moves back to 1 and continues checking... > >> >> > >> >> Log > >> >> > >> > > >> > >> > >> > >> -- > >> Best wishes > >> Gan, Xiyun > > > > > > > > -- > Best wishes > Gan, Xiyun >
