How large/small are the writes?
Can you run the following, then try your test again and report the
results for the time period where your test is running?
iostat -x -d 1
also note that ZK JMX allows you to reset the latency attributes (look
under "operations" in jconsole). If you reset the latency during your
test what do you see happen wrt the min/max/avg latency? Keep an eye on
this (and perhaps resetting the stats every so often) during your test.
Anything interesting happening that you notice?
I suspect that given a single disk is being used (not a dedicated disk
for the transaction log), and also given that this host is highly
virtualized (ec2), it seems to me that the most likely cause is IO.
Specifically when the zk cluster writes data to disk (due to client
write) it must sync the transaction log to disk. This sync behavior can
impact the latency seen by the clients. What type of ec2 node are you
using? Ted, do you have any insight on this? Any guidelines for the type
of ec2 node to use for running a zk cluster?
Satish Bhatti wrote:
According to the jconsole, max memory usage is 30MB, 14 live threads and
peak CPU usage about 4%, average is under 1%. We are not really hammering
it. Doing about 10 read/writes every second max.
On Tue, Sep 1, 2009 at 5:20 PM, Ted Dunning <ted.dunn...@gmail.com> wrote:
This is outrageously large. Max should be more like 50ms.
Either you are doing this somehow, or you have an anomaly on your ZK
How much data is in ZK? How many transaction per seoncd?
On Tue, Sep 1, 2009 at 5:11 PM, Satish Bhatti <cthd2...@gmail.com> wrote:
Ted Dunning, CTO