Refer to https://issues.apache.org/jira/browse/HBASE-3669
Probably it's a bug.

Is this issue reproducible?

On Fri, Jul 29, 2011 at 1:22 PM, Xian Woo <[email protected]> wrote:
> Actually it happened when I did a test to manually reboot one of the  region
> servers.  Note that the region server I rebooted was the one that contained
> -ROOT-, &.META as online regions.
> After I rebooted the server, the cluster crashed. I had to reboot the entire
> cluster , hoping that the cluster would fix this itself.  But to my
> disappointed, the cluster didn't recover and the log showed the error which
> have been listed in this mail.
> Speaking of this, do you know anything about my problem?shouldn't
> the  -ROOT-, &.META keep synchronized among all the servers?
> Please contact with me if u are interested in this issue.
> Much Thanks.
>
> 2011/7/29 Gan, Xiyun <[email protected]>
>>
>> Following is my understanding. Please correct me if there are any
>> mistakes.
>>
>> Did you change Zookeeper quorum configuration?
>> Zookeeper tracks table schema and regionservers. If ZK quorum changes,
>> the data Zookeeper saved is corrupted since the integration of the
>> data has been destroyed.
>> You may see some snippets from the log which says  the cluster can't
>> connect to a previously selected ZK node.
>>
>> As far as I know, table data will not be lost after cleaning up the
>> zookeeper data, but I'm not sure.
>>
>> On Fri, Jul 29, 2011 at 11:33 AM, Xian Woo <[email protected]> wrote:
>> > Wow,it really helps~~ Thanks pal~~
>> > But in fact, I really don't understand why this issue can be solved just
>> > by
>> > cleaning up the zookeeper data on each zk node.  So I really hope you
>> > can
>> > tell me something more about that. Thanks~
>> > Best wishes~
>> > Woo
>> > 2011/7/29 Gan, Xiyun <[email protected]>
>> >>
>> >> Hi, Woo
>> >>     I have been confronted with this issue in our cluster. Have you
>> >> started hbase cluster successfully ever before? If yes, try to clean
>> >> up the zookeeper data on each zk node.
>> >>
>> >>
>> >> On Fri, Jul 29, 2011 at 1:03 AM, Xian Woo <[email protected]>
>> >> wrote:
>> >> > Excuse me for clicking the 'send' button accidently..
>> >> >
>> >> > Log in Server4.yun.com:
>> >> > Here are some segments:
>> >> >
>> >> > 2011-07-28 20:48:41,123 DEBUG
>> >> > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64
>> >> > MB,
>> >> > free=197.54 MB, max=199.18 MB, blocks=0, accesses=0, hits=0,
>> >> > hitRatio=�%,
>> >> > cachingAccesses=0, cachingHits=0, cachingHitsRatio=�%, evictions=0,
>> >> > evicted=0, evictedPerRun=NaN
>> >> >
>> >> > 2011-07-28 20:38:41,109 INFO org.apache.hadoop.ipc.HBaseServer: IPC
>> >> > Server
>> >> > handler 2 on 60020: starting
>> >> >
>> >> > 2011-07-28 20:38:40,598 DEBUG
>> >> > org.apache.hadoop.hbase.executor.ExecutorService: Starting executor
>> >> > service
>> >> > name=RS_OPEN_ROOT-server4.yun.com,60020,1311856719820,
>> >> > corePoolSize=1,
>> >> > maxPoolSize=1
>> >> >
>> >> > 2011-07-28 20:38:40,261 INFO
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: Runs every
>> >> > 10000000ms
>> >> >
>> >> >
>> >> > Log in server5.yun.com:
>> >> >
>> >> > 1.It first keeps showing:
>> >> >
>> >> > 2011-07-28 20:37:05,616 DEBUG
>> >> > org.apache.hadoop.hbase.regionserver.HRegionServer:
>> >> > NotServingRegionException; Region is not online: -ROOT-,,0
>> >> >
>> >> > 2. Then it writes this per 5 minutes:
>> >> >
>> >> > 2011-07-28 20:40:52,814 DEBUG
>> >> > org.apache.hadoop.hbase.io.hfile.LruBlockCache: LRU Stats: total=1.64
>> >> > MB,
>> >> > free=197.54 MB, max=199.18 MB, blocks=0, accesses=0, hits=0,
>> >> > hitRatio=锟?,
>> >> > cachingAccesses=0, cachingHits=0, cachingHitsRatio=锟?, evictions=0,
>> >> > evicted=0, evictedPerRun=NaN
>> >> >
>> >> >
>> >> > I would really appreciate it if  you can give me some advice on this
>> >> > issue.
>> >> >
>> >> > Thanks a lot ~
>> >> >
>> >> > Best Wishes~
>> >> >
>> >> > Woo
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > 2011/7/29 Xian Woo <[email protected]>
>> >> >
>> >> >>
>> >> >> Hi everyone. I have just completed in installing my hbase and there
>> >> >> are
>> >> >> some errors when I try to start my cluster.
>> >> >>
>> >> >> Setup:
>> >> >>    -cdh3u1
>> >> >>    - Hadoop 0.20.2
>> >> >>    - HBase 0.90.1
>> >> >>    - 1 Master Node running as NameNode & JobTracker(named
>> >> >> server3.yun.com)
>> >> >>    -zookeeper quorum
>> >> >>    - 2 child nodes(server4.yun.com & server5.yun.com) running as
>> >> >> Datanode,
>> >> >> TaskTracker and RegionServer each.
>> >> >>
>> >> >> When I try to start the master and the region servers, I find that
>> >> >> some
>> >> >> issues in my log:
>> >> >>
>> >> >> Master log(server3.yun.com):
>> >> >>
>> >> >> 1. It first keeps showing this:
>> >> >> "2011-07-28 20:30:16,056 INFO
>> >> >> org.apache.hadoop.hbase.master.AssignmentManager: Region has been
>> >> >> OPENING
>> >> >> for too long, reassigning region=-ROOT-,,0.70236052
>> >> >> 2011-07-28 20:30:16,057 DEBUG
>> >> >> org.apache.hadoop.hbase.master.AssignmentManager: Region has
>> >> >> transitioned to
>> >> >> OPENED, allowing watched event handlers to process"
>> >> >>
>> >> >> 2.After several minutes of the same contents written in the log,
>> >> >>  then:
>> >> >> "2011-07-28 20:30:16,412 DEBUG
>> >> >>
>> >> >>
>> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> >> >> Lookedup root region location,
>> >> >>
>> >> >>
>> >> >> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@7e24d517;
>> >> >> hsa=server5.yun.com:60020
>> >> >> 2011-07-28 20:30:16,413 DEBUG
>> >> >>
>> >> >>
>> >> >> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
>> >> >> locateRegionInMeta parentTable=-ROOT-, metaLocation=address:
>> >> >> server5.yun.com:60020, regioninfo: -ROOT-,,0.70236052, attempt=1 of
>> >> >> 10
>> >> >> failed; retrying after sleep of 1000 because:
>> >> >> org.apache.hadoop.hbase.NotServingRegionException: Region is not
>> >> >> online:
>> >> >> -ROOT-,,0
>> >> >>  at
>> >> >>
>> >> >>
>> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:2318)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1614)
>> >> >>  at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
>> >> >> at
>> >> >>
>> >> >>
>> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >> >>  at java.lang.reflect.Method.invoke(Method.java:597)
>> >> >> at
>> >> >> org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:570)
>> >> >>  at
>> >> >>
>> >> >>
>> >> >> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1039)
>> >> >> "
>> >> >>
>> >> >> 3.Then it moves back to 1 and continues checking...
>> >> >>
>> >> >> Log
>> >> >>
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Best wishes
>> >> Gan, Xiyun
>> >
>> >
>>
>>
>>
>> --
>> Best wishes
>> Gan, Xiyun
>
>



-- 
Best wishes
Gan, Xiyun

Reply via email to