Yes - the session drop happened again. I did the stat. The max latency is huge (I assume that's in ms).
Zookeeper version: 3.3.0-925362, built on 03/19/2010 18:38 GMT Clients: /10.243.14.179:57300(queued=0,recved=0,sent=0) /22.214.171.124:51493(queued=0,recved=1,sent=0) /10.243.13.191:44414(queued=0,recved=0,sent=0) /10.243.13.191:44415(queued=0,recved=0,sent=0) /10.244.135.100:58965(queued=0,recved=0,sent=0) /10.243.15.192:45057(queued=0,recved=0,sent=0) Latency min/avg/max: 0/6/965759 Received: 2203048 Sent: 2202979 Outstanding: 0 Zxid: 0xef2b2 Mode: standalone Node count: 347 On Jun 15, 2010, at 11:02 AM, Ted Dunning wrote: > Jordan, > > Good step to get this info. > > I have to ask, did you have your disconnect problem last night as well? > (just checking) > > What does the stat command on ZK give you for each server? > > On Tue, Jun 15, 2010 at 10:33 AM, Jordan Zimmerman < > jzimmer...@proofpoint.com> wrote: > >> More on this... >> >> I ran last night with verbose GC on our client. I analyzed the GC log in >> gchisto and 99% of the GCs are 1 or 2 ms. The longest gc is 30 ms. On the >> Zookeeper server side, the longest gc is 130 ms. So, I submit, GC is not the >> problem. NOTE we're running on Amazon EC2. >> >>