Yes, we are. We currently don't handle SessionExpired very well at all in
HBase. There are two things going on in parallel to fix it:
1) Reinitialize the ZooKeeper handler (and everything else that depends on
it) on the node in question when a SessionExpired event occurs.
2) Reduce the number of SessionExpired events we get by using Joey's JNI
solution. After the various talks about session timeout, different GC flags,
etc, we decided to pursue the JNI solution. We plan on contributing his work
back to ZooKeeper, under some contrib, so that others can use it.
In the really short term, for folks that are seeing it, using the concurrent
GC and bumping up the session timeout to 30 seconds or so seems to reduce
the frequency of the problem.
I'm curious if your problems are the same as ours. You should try tweaking
the GC parameters and session timeout to see if the problems you're having
are the same as ours.
On Tue, Apr 14, 2009 at 6:34 PM, Ted Dunning <ted.dunn...@gmail.com> wrote:
> Very good pointer. Thanks.
> Are you still having your problems?
> On Tue, Apr 14, 2009 at 6:09 PM, Nitay <nit...@gmail.com> wrote:
> > Hi Ted,
> > Fellow user coming from HBase. We were recently seeing lots of
> > SessionExpired events as well. Check out this mail thread:
> > Perhaps this might have something to do with what you're seeing.
> > Cheers,
> > -n
> > On Tue, Apr 14, 2009 at 5:48 PM, Ted Dunning <ted.dunn...@gmail.com>
> > wrote:
> > > We have been using EC2 as a substrate for our search cluster with
> > zookeeper
> > > as our coordination layer and have been seeing some strange problems.
> > >
> > > These problems seem to manifest around getting lots of anomalous
> > > disconnects
> > > and session expirations even though we have the timeout values set to 2
> > > seconds on the server side and 5 seconds on the client side.
> > >
> > > Has anybody else been seeing this?
> > >
> > > Is this related to clock jumps in a virtualized setting?
> > >
> > > On a related note, what is best practice for handling session
> > > Just deal with it as if it is a new start?
> > >
> Ted Dunning, CTO
> 111 West Evelyn Ave. Ste. 202
> Sunnyvale, CA 94086
> 858-414-0013 (m)
> 408-773-0220 (fax)