Hello J-D,
Thank you for your reply first.
>How many CPUs do you have?
Every server has 2 Dual-Core cpus.
>Are you swapping?
Now I'm not sure about it with our monitor tools, but now we have written
a script to record vmstat log every 2 seconds. If something wrong happen
again, we can take it.
>Also if the only you are using this system currently to batch load
>data or as an analytics backend, you probably want to set the timeout
>higher:
But our value of this property is already 240000.
We will try to optimize our garbage collector and we will see what will
happen.
Thanks again, J-D,
LvZheng
2010/3/25 Jean-Daniel Cryans <[email protected]>
> 2010-03-24 11:33:52,331 WARN org.apache.hadoop.hbase.util.Sleeper: We
> slept 54963ms, ten times longer than scheduled: 3000
>
> You had an important garbage collector pause (aka pause of the world
> in java-speak) and your region server's session with zookeeper expired
> (it literally stopped responding for too long, so long it was
> considered dead). Are you swapping? How many CPUs do you have? If you
> are slowing down the garbage collecting process, it will take more
> time.
>
> Also if the only you are using this system currently to batch load
> data or as an analytics backend, you probably want to set the timeout
> higher:
>
> <property>
> <name>zookeeper.session.timeout</name>
> <value>60000</value>
> <description>ZooKeeper session timeout.
> HBase passes this to the zk quorum as suggested maximum time for a
> session. See
>
> http://hadoop.apache.org/zookeeper/docs/current/zookeeperProgrammers.html#ch_zkSessions
> "The client sends a requested timeout, the server responds with the
> timeout that it can give the client. The current implementation
> requires that the timeout be a minimum of 2 times the tickTime
> (as set in the server configuration) and a maximum of 20 times
> the tickTime." Set the zk ticktime with
> hbase.zookeeper.property.tickTime.
> In milliseconds.
> </description>
> </property>
>
> This value can only be 20 times bigger than this:
>
> <property>
> <name>hbase.zookeeper.property.tickTime</name>
> <value>3000</value>
> <description>Property from ZooKeeper's config zoo.cfg.
> The number of milliseconds of each tick. See
> zookeeper.session.timeout description.
> </description>
> </property>
>
>
> So you could set tick to 6000, timeout to 120000 for a 2min timeout.
>
> J-D
>
> On Wed, Mar 24, 2010 at 8:01 PM, Zheng Lv <[email protected]>
> wrote:
> > Hello Stack,
> > Yesterday we got another problem about "zookeeper session expired",
> > leading rs shutdown, which never happened before.
> > I googled it, finding some docs about it, but didnot get things
> > really certain about how it happened and how to avoid it.
> > Now I have put the corresponding logs to
> > http://rapidshare.com/files/367820690/208-0324.log.html.
> > Look forward to your reply.
> > Thank you.
> > LvZheng
> >
> > 2010/3/24 Zheng Lv <[email protected]>
> >
> >> Hello Stack,
> >> Thank you for your explainations, it's very helpful, Thank you.
> >> If I get something new, I'll connect you.
> >> Regards,
> >> LvZheng
> >>
> >> 2010/3/24 Stack <[email protected]>
> >>
> >> On Tue, Mar 23, 2010 at 8:42 PM, Zheng Lv <[email protected]>
> >>> wrote:
> >>> > Hello Stack,
> >>> > >So, for sure ugly stuff is going on. I filed
> >>> > >https://issues.apache.org/jira/browse/HBASE-2365. It looks like
> >>> we're
> >>> > >doubly assigning a region.
> >>> > Can you tell me how this happened in detail? Thanks a lot.
> >>> >
> >>>
> >>> Yes.
> >>>
> >>> Splits are run by the regionserver. It figures a region needs to be
> >>> split and goes ahead closing the parent and creating the daughter
> >>> regions. It then adds edits to the meta table offlining the parent
> >>> and inserting the two new daughter regions. Next it sends a message
> >>> to the master telling it that a region has been split. The message
> >>> contains names of the daughter regions. On receipt of the message,
> >>> the master adds the new daughter regions to the unassigned regions
> >>> list so they'll be passed out the next time a regionserver checks in.
> >>>
> >>> Concurrently, the master is running a scan of the meta table every
> >>> minute making sure all is in order. One thing it does is if it finds
> >>> unassigned regions, it'll add them to the unassigned regions (this
> >>> process is what gets all regions assigned after a startup).
> >>>
> >>> In your case, whats happening is that there is a long period between
> >>> the add of the new split regions to the meta table and the report of
> >>> split to the master. During this time, the master meta scan ran,
> >>> found one of the daughters and went and assigned it. Then the split
> >>> message came in and the daughter was assigned again!
> >>>
> >>> There was supposed to be protection against this happening IIRC.
> >>> Looking at responsible code, we are trying to defend against this
> >>> happening in ServerManager:
> >>>
> >>> /*
> >>> * Assign new daughter-of-a-split UNLESS its already been assigned.
> >>> * It could have been assigned already in rare case where there was a
> >>> large
> >>> * gap between insertion of the daughter region into .META. by the
> >>> * splitting regionserver and receipt of the split message in master
> (See
> >>> * HBASE-1784).
> >>> * @param hri Region to assign.
> >>> */
> >>> private void assignSplitDaughter(final HRegionInfo hri) {
> >>> MetaRegion mr =
> >>> this.master.regionManager.getFirstMetaRegionForRegion(hri);
> >>> Get g = new Get(hri.getRegionName());
> >>> g.addFamily(HConstants.CATALOG_FAMILY);
> >>> try {
> >>> HRegionInterface server =
> >>> master.connection.getHRegionConnection(mr.getServer());
> >>> Result r = server.get(mr.getRegionName(), g);
> >>> // If size > 3 -- presume regioninfo, startcode and server -- then
> >>> presume
> >>> // that this daughter already assigned and return.
> >>> if (r.size() >= 3) return;
> >>> } catch (IOException e) {
> >>> LOG.warn("Failed get on " + HConstants.CATALOG_FAMILY_STR +
> >>> "; possible double-assignment?", e);
> >>> }
> >>> this.master.regionManager.setUnassigned(hri, false);
> >>> }
> >>>
> >>> So, the above is not working in your case for some reason. I'll take
> >>> a look but I'm not sure I can figure it w/o DEBUG (thanks for letting
> >>> me know about the out-of-sync clocks... Now I can have more faith in
> >>> what the logs are telling me).
> >>>
> >>> >
> >>> > >With DEBUG enabled have you been able to reproduce?
> >>> > These days the exception did not appera again, if it would, I'll
> show
> >>> you
> >>> > the logs.
> >>> >
> >>>
> >>> For sure, if you come across it again, I'm interested.
> >>>
> >>> Thanks Zheng,
> >>> St.Ack
> >>>
> >>
> >>
> >
>
