Take a look at this section to start:

What type of monitoring are you doing on your cluster? You could monitor at both the host and at the java (jmx) level. That will give you some insight on where to look; cpu, memory, disk, network, etc... Also the ZooKeeper JMX will give you information about latencies and such (you can even use the "four letter words" for that if you want to hack up some scripts instead of using jmx). JMX will also give you insight into the JVM workings - so for example you could confirm/ruleout the scenario outlined by Nitay (gc causing the jvm java threads to hang for > 30sec at a time, including the ZK heartbeat).

I've seen similar to what you describe a few times now, in each case it was something different. In one case for example there was a cluster of 5k clients attaching to a ZK cluster, ~20% of the clients had mis-configured nics, that was causing high tcp packet loss (and therefore high network latency), which caused a similar situation to what you are seeing, but only under fairly high network load (which made it hard to track down!).

I've also seen situations where ppl run the entire zk cluster on a set of VMWare vms, all on the same host system. Latency on this configuration was >>> 10sec in some cases due to resource issues (in particular io - see the link I provided above, dedicated log devices are critical to low latency operation of the ZK cluster).

In your scenario I think 5 sec timeout is too low, probably much too low. Why? You are running in virtualized environments on non-dedicated hardware outside your control/inspection. There is typically no way to tell (unless you are running on the 8 core ec2 systems) if the ec2 host you are running on is over/under subscribed (other vms). There is no way to control disk latency either. You could be seeing large latencies due to resource contention on the ec2 host alone. In addition to that I've heard that network latencies in ec2 are high relative to what you would see if you were running on your own dedicated environment. It's hard to tell the latency btw the servers and client->server w/in the ec2 environment you are seeing w/out measuring it.

Keep in mind the the timeout period is used by both the client and the server. If the ZK leader doesn't hear from the client w/in the timeout (say it's 5 sec) it will expire the session. The client is sending a ping after 1/3 of the timeout period. It expects to hear a response before another 1/3 of the timeout elapses, after which it will attempt to re-sync to another server in the cluster. In the 5 sec timeout case you are allowing 1.3 seconds for the request to go to the server, the server to respond back to the client, and the client to process the response. Check the latencies in ZK's JMX as I suggested to the hbase team in order to get insight into this (i.e. if the server latency is high, say because of io issues, or jvm swapping, vm latency, etc... that will cause the client/sessions to timeout)

Hope this helps.


Mahadev Konar wrote:
Hi Ted,
These problems seem to manifest around getting lots of anomalous disconnects
and session expirations even though we have the timeout values set to 2
seconds on the server side and 5 seconds on the client side.

 Your scenario might be a little differetn from what Nitay (Hbase) is
seeing. In their scenario the zookeeper client was not able to send out
pings to the server due to gc stalling threads in their zookeeper
application process.

The latencies in zookeeper clients are directly related to Zookeeper server
machines. It is very much dependant on the disk io latencies that you would
get on the zookeeper servers and network latencies with your cluster.

I am not sure how much sensitive you want your zookeeper application to be
-- but increasing the timeout should help. Also, we recommend using
dedicated disk for zookeeper log transactions.


Also, we have seen Ntp having problems and clocks going back on one of our
vm setup. This would lead to session getting timed out earler than the set
session timeout.

I hope this helps.


On 4/14/09 5:48 PM, "Ted Dunning" <ted.dunn...@gmail.com> wrote:

We have been using EC2 as a substrate for our search cluster with zookeeper
as our coordination layer and have been seeing some strange problems.

These problems seem to manifest around getting lots of anomalous disconnects
and session expirations even though we have the timeout values set to 2
seconds on the server side and 5 seconds on the client side.

Has anybody else been seeing this?

Is this related to clock jumps in a virtualized setting?

On a related note, what is best practice for handling session expiration?
Just deal with it as if it is a new start?

Reply via email to