Refer to https://issues.apache.org/jira/browse/HBASE-3669 Probably it's a bug.
Is this issue reproducible? On Fri, Jul 29, 2011 at 1:22 PM, Xian Woo <[email protected]> wrote: > Actually it happened when I did a test to manually reboot one of the region > servers. Note that the region server I rebooted was the one that contained > -ROOT-, &.META as online regions. > After I rebooted the server, the cluster crashed. I had to reboot the entire > cluster , hoping that the cluster would fix this itself. But to my > disappointed, the cluster didn't recover and the log showed the error which > have been listed in this mail. > Speaking of this, do you know anything about my problem?shouldn't > the -ROOT-, &.META keep synchronized among all the servers? > Please contact with me if u are interested in this issue. > Much Thanks. > > 2011/7/29 Gan, Xiyun <[email protected]> >> >> Following is my understanding. Please correct me if there are any >> mistakes. >> >> Did you change Zookeeper quorum configuration? >> Zookeeper tracks table schema and regionservers. If ZK quorum changes, >> the data Zookeeper saved is corrupted since the integration of the >> data has been destroyed. >> You may see some snippets from the log which says the cluster can't >> connect to a previously selected ZK node. >> >> As far as I know, table data will not be lost after cleaning up the >> zookeeper data, but I'm not sure. >> >> On Fri, Jul 29, 2011 at 11:33 AM, Xian Woo <[email protected]> wrote: >> > Wow,it really helps~~ Thanks pal~~ >> > But in fact, I really don't understand why this issue can be solved just >> > by >> > cleaning up the zookeeper data on each zk node. So I really hope you >> > can >> > tell me something more about that. Thanks~ >> > Best wishes~ >> > Woo >> > 2011/7/29 Gan, Xiyun <[email protected]> >> >> >> >> Hi, Woo >> >> I have been confronted with this issue in our cluster. Have you >> >> started hbase cluster successfully ever before? If yes, try to clean >> >> up the zookeeper data on each zk node. >> >> >> >> >> >> On Fri, Jul 29, 2011 at 1:03 AM, Xian Woo <[email protected]> >> >> wrote: >> >> > Excuse me for clicking the 'send' button accidently.. >> >> > >> >> > Log in Server4.yun.com: >> >> > Here are some segments: >> >> > >> >> > 2011-07-28 20:48:41,123 DEBUG >> >> > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 >> >> > MB, >> >> > free=197.54 MB, max=199.18 MB, blocks=0, accesses=0, hits=0, >> >> > hitRatio=�%, >> >> > cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%, evictions=0, >> >> > evicted=0, evictedPerRun=NaN >> >> > >> >> > 2011-07-28 20:38:41,109 INFO org.apache.hadoop.ipc.HBaseServer: IPC >> >> > Server >> >> > handler 2 on 60020: starting >> >> > >> >> > 2011-07-28 20:38:40,598 DEBUG >> >> > org.apache.hadoop.hbase.executor.ExecutorService: Starting executor >> >> > service >> >> > name=RS_OPEN_ROOT-server4.yun.com,60020,1311856719820, >> >> > corePoolSize=1, >> >> > maxPoolSize=1 >> >> > >> >> > 2011-07-28 20:38:40,261 INFO >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every >> >> > 10000000ms >> >> > >> >> > >> >> > Log in server5.yun.com: >> >> > >> >> > 1.It first keeps showing: >> >> > >> >> > 2011-07-28 20:37:05,616 DEBUG >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: >> >> > NotServingRegionException; Region is not online: -ROOT-,,0 >> >> > >> >> > 2. Then it writes this per 5 minutes: >> >> > >> >> > 2011-07-28 20:40:52,814 DEBUG >> >> > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64 >> >> > MB, >> >> > free=197.54 MB, max=199.18 MB, blocks=0, accesses=0, hits=0, >> >> > hitRatio=锟?, >> >> > cachingAccesses=0, cachingHits=0, cachingHitsRatio=锟?, evictions=0, >> >> > evicted=0, evictedPerRun=NaN >> >> > >> >> > >> >> > I would really appreciate it if you can give me some advice on this >> >> > issue. >> >> > >> >> > Thanks a lot ~ >> >> > >> >> > Best Wishes~ >> >> > >> >> > Woo >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> > 2011/7/29 Xian Woo <[email protected]> >> >> > >> >> >> >> >> >> Hi everyone. I have just completed in installing my hbase and there >> >> >> are >> >> >> some errors when I try to start my cluster. >> >> >> >> >> >> Setup: >> >> >> -cdh3u1 >> >> >> - Hadoop 0.20.2 >> >> >> - HBase 0.90.1 >> >> >> - 1 Master Node running as NameNode & JobTracker(named >> >> >> server3.yun.com) >> >> >> -zookeeper quorum >> >> >> - 2 child nodes(server4.yun.com & server5.yun.com) running as >> >> >> Datanode, >> >> >> TaskTracker and RegionServer each. >> >> >> >> >> >> When I try to start the master and the region servers, I find that >> >> >> some >> >> >> issues in my log: >> >> >> >> >> >> Master log(server3.yun.com): >> >> >> >> >> >> 1. It first keeps showing this: >> >> >> "2011-07-28 20:30:16,056 INFO >> >> >> org.apache.hadoop.hbase.master.AssignmentManager: Region has been >> >> >> OPENING >> >> >> for too long, reassigning region=-ROOT-,,0.70236052 >> >> >> 2011-07-28 20:30:16,057 DEBUG >> >> >> org.apache.hadoop.hbase.master.AssignmentManager: Region has >> >> >> transitioned to >> >> >> OPENED, allowing watched event handlers to process" >> >> >> >> >> >> 2.After several minutes of the same contents written in the log, >> >> >> then: >> >> >> "2011-07-28 20:30:16,412 DEBUG >> >> >> >> >> >> >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: >> >> >> Lookedup root region location, >> >> >> >> >> >> >> >> >> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@7e24d517; >> >> >> hsa=server5.yun.com:60020 >> >> >> 2011-07-28 20:30:16,413 DEBUG >> >> >> >> >> >> >> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: >> >> >> locateRegionInMeta parentTable=-ROOT-, metaLocation=address: >> >> >> server5.yun.com:60020, regioninfo: -ROOT-,,0.70236052, attempt=1 of >> >> >> 10 >> >> >> failed; retrying after sleep of 1000 because: >> >> >> org.apache.hadoop.hbase.NotServingRegionException: Region is not >> >> >> online: >> >> >> -ROOT-,,0 >> >> >> at >> >> >> >> >> >> >> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2318) >> >> >> at >> >> >> >> >> >> >> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1614) >> >> >> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source) >> >> >> at >> >> >> >> >> >> >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) >> >> >> at java.lang.reflect.Method.invoke(Method.java:597) >> >> >> at >> >> >> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570) >> >> >> at >> >> >> >> >> >> >> >> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039) >> >> >> " >> >> >> >> >> >> 3.Then it moves back to 1 and continues checking... >> >> >> >> >> >> Log >> >> >> >> >> > >> >> >> >> >> >> >> >> -- >> >> Best wishes >> >> Gan, Xiyun >> > >> > >> >> >> >> -- >> Best wishes >> Gan, Xiyun > > -- Best wishes Gan, Xiyun
