hmm, I am not sure about that , let me check again. Thank you about that. ^_^
2011/7/29 Gan, Xiyun <[email protected]> > Refer to https://issues.apache.org/jira/browse/HBASE-3669 > Probably it's a bug. > > Is this issue reproducible? > > On Fri, Jul 29, 2011 at 1:22 PM, Xian Woo <[email protected]> wrote: > > Actually it happened when I did a test to manually reboot one of the > region > > servers. Note that the region server I rebooted was the one that > contained > > -ROOT-, &.META as online regions. > > After I rebooted the server, the cluster crashed. I had to reboot the > entire > > cluster , hoping that the cluster would fix this itself. But to my > > disappointed, the cluster didn't recover and the log showed the error > which > > have been listed in this mail. > > Speaking of this, do you know anything about my problem?shouldn't > > the -ROOT-, &.META keep synchronized among all the servers? > > Please contact with me if u are interested in this issue. > > Much Thanks. > > > > 2011/7/29 Gan, Xiyun <[email protected]> > >> > >> Following is my understanding. Please correct me if there are any > >> mistakes. > >> > >> Did you change Zookeeper quorum configuration? > >> Zookeeper tracks table schema and regionservers. If ZK quorum changes, > >> the data Zookeeper saved is corrupted since the integration of the > >> data has been destroyed. > >> You may see some snippets from the log which says the cluster can't > >> connect to a previously selected ZK node. > >> > >> As far as I know, table data will not be lost after cleaning up the > >> zookeeper data, but I'm not sure. > >> > >> On Fri, Jul 29, 2011 at 11:33 AM, Xian Woo <[email protected]> > wrote: > >> > Wow,it really helps~~ Thanks pal~~ > >> > But in fact, I really don't understand why this issue can be solved > just > >> > by > >> > cleaning up the zookeeper data on each zk node. So I really hope you > >> > can > >> > tell me something more about that. Thanks~ > >> > Best wishes~ > >> > Woo > >> > 2011/7/29 Gan, Xiyun <[email protected]> > >> >> > >> >> Hi, Woo > >> >> I have been confronted with this issue in our cluster. Have you > >> >> started hbase cluster successfully ever before? If yes, try to clean > >> >> up the zookeeper data on each zk node. > >> >> > >> >> > >> >> On Fri, Jul 29, 2011 at 1:03 AM, Xian Woo <[email protected]> > >> >> wrote: > >> >> > Excuse me for clicking the 'send' button accidently.. > >> >> > > >> >> > Log in Server4.yun.com: > >> >> > Here are some segments: > >> >> > > >> >> > 2011-07-28 20:48:41,123 DEBUG > >> >> > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: > total=1.64 > >> >> > MB, > >> >> > free=197.54 MB, max=199.18 MB, blocks=0, accesses=0, hits=0, > >> >> > hitRatio=�%, > >> >> > cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%, evictions=0, > >> >> > evicted=0, evictedPerRun=NaN > >> >> > > >> >> > 2011-07-28 20:38:41,109 INFO org.apache.hadoop.ipc.HBaseServer: IPC > >> >> > Server > >> >> > handler 2 on 60020: starting > >> >> > > >> >> > 2011-07-28 20:38:40,598 DEBUG > >> >> > org.apache.hadoop.hbase.executor.ExecutorService: Starting executor > >> >> > service > >> >> > name=RS_OPEN_ROOT-server4.yun.com,60020,1311856719820, > >> >> > corePoolSize=1, > >> >> > maxPoolSize=1 > >> >> > > >> >> > 2011-07-28 20:38:40,261 INFO > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every > >> >> > 10000000ms > >> >> > > >> >> > > >> >> > Log in server5.yun.com: > >> >> > > >> >> > 1.It first keeps showing: > >> >> > > >> >> > 2011-07-28 20:37:05,616 DEBUG > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: > >> >> > NotServingRegionException; Region is not online: -ROOT-,,0 > >> >> > > >> >> > 2. Then it writes this per 5 minutes: > >> >> > > >> >> > 2011-07-28 20:40:52,814 DEBUG > >> >> > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: > total=1.64 > >> >> > MB, > >> >> > free=197.54 MB, max=199.18 MB, blocks=0, accesses=0, hits=0, > >> >> > hitRatio=锟?, > >> >> > cachingAccesses=0, cachingHits=0, cachingHitsRatio=锟?, evictions=0, > >> >> > evicted=0, evictedPerRun=NaN > >> >> > > >> >> > > >> >> > I would really appreciate it if you can give me some advice on > this > >> >> > issue. > >> >> > > >> >> > Thanks a lot ~ > >> >> > > >> >> > Best Wishes~ > >> >> > > >> >> > Woo > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > > >> >> > 2011/7/29 Xian Woo <[email protected]> > >> >> > > >> >> >> > >> >> >> Hi everyone. I have just completed in installing my hbase and > there > >> >> >> are > >> >> >> some errors when I try to start my cluster. > >> >> >> > >> >> >> Setup: > >> >> >> -cdh3u1 > >> >> >> - Hadoop 0.20.2 > >> >> >> - HBase 0.90.1 > >> >> >> - 1 Master Node running as NameNode & JobTracker(named > >> >> >> server3.yun.com) > >> >> >> -zookeeper quorum > >> >> >> - 2 child nodes(server4.yun.com & server5.yun.com) running as > >> >> >> Datanode, > >> >> >> TaskTracker and RegionServer each. > >> >> >> > >> >> >> When I try to start the master and the region servers, I find that > >> >> >> some > >> >> >> issues in my log: > >> >> >> > >> >> >> Master log(server3.yun.com): > >> >> >> > >> >> >> 1. It first keeps showing this: > >> >> >> "2011-07-28 20:30:16,056 INFO > >> >> >> org.apache.hadoop.hbase.master.AssignmentManager: Region has been > >> >> >> OPENING > >> >> >> for too long, reassigning region=-ROOT-,,0.70236052 > >> >> >> 2011-07-28 20:30:16,057 DEBUG > >> >> >> org.apache.hadoop.hbase.master.AssignmentManager: Region has > >> >> >> transitioned to > >> >> >> OPENED, allowing watched event handlers to process" > >> >> >> > >> >> >> 2.After several minutes of the same contents written in the log, > >> >> >> then: > >> >> >> "2011-07-28 20:30:16,412 DEBUG > >> >> >> > >> >> >> > >> >> >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > >> >> >> Lookedup root region location, > >> >> >> > >> >> >> > >> >> >> > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@7e24d517 > ; > >> >> >> hsa=server5.yun.com:60020 > >> >> >> 2011-07-28 20:30:16,413 DEBUG > >> >> >> > >> >> >> > >> >> >> > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: > >> >> >> locateRegionInMeta parentTable=-ROOT-, metaLocation=address: > >> >> >> server5.yun.com:60020, regioninfo: -ROOT-,,0.70236052, attempt=1 > of > >> >> >> 10 > >> >> >> failed; retrying after sleep of 1000 because: > >> >> >> org.apache.hadoop.hbase.NotServingRegionException: Region is not > >> >> >> online: > >> >> >> -ROOT-,,0 > >> >> >> at > >> >> >> > >> >> >> > >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2318) > >> >> >> at > >> >> >> > >> >> >> > >> >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1614) > >> >> >> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) > >> >> >> at > >> >> >> > >> >> >> > >> >> >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > >> >> >> at java.lang.reflect.Method.invoke(Method.java:597) > >> >> >> at > >> >> >> > org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) > >> >> >> at > >> >> >> > >> >> >> > >> >> >> > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) > >> >> >> " > >> >> >> > >> >> >> 3.Then it moves back to 1 and continues checking... > >> >> >> > >> >> >> Log > >> >> >> > >> >> > > >> >> > >> >> > >> >> > >> >> -- > >> >> Best wishes > >> >> Gan, Xiyun > >> > > >> > > >> > >> > >> > >> -- > >> Best wishes > >> Gan, Xiyun > > > > > > > > -- > Best wishes > Gan, Xiyun >
