How large/small are the writes?

Can you run the following, then try your test again and report the results for the time period where your test is running?

iostat -x -d 1

also note that ZK JMX allows you to reset the latency attributes (look under "operations" in jconsole). If you reset the latency during your test what do you see happen wrt the min/max/avg latency? Keep an eye on this (and perhaps resetting the stats every so often) during your test. Anything interesting happening that you notice?

I suspect that given a single disk is being used (not a dedicated disk for the transaction log), and also given that this host is highly virtualized (ec2), it seems to me that the most likely cause is IO. Specifically when the zk cluster writes data to disk (due to client write) it must sync the transaction log to disk. This sync behavior can impact the latency seen by the clients. What type of ec2 node are you using? Ted, do you have any insight on this? Any guidelines for the type of ec2 node to use for running a zk cluster?


Satish Bhatti wrote:
According to the jconsole, max memory usage is 30MB, 14 live threads and
peak CPU usage about 4%, average is under 1%.  We are not really hammering
it.  Doing about 10 read/writes every second max.

On Tue, Sep 1, 2009 at 5:20 PM, Ted Dunning <> wrote:

This is outrageously large.  Max should be more like 50ms.

Either you are doing this somehow, or you have an anomaly on your ZK

How much data is in ZK?  How many transaction per seoncd?

On Tue, Sep 1, 2009 at 5:11 PM, Satish Bhatti <> wrote:

MaxRequestLatency 55767

Ted Dunning, CTO

Reply via email to