Setting up a little process to run overnight that appends a timestamp to a file once per second or so can be a very effective tool for ruling out, for example, "extra-dimensional" VM effects.
On 06/16/2010 12:15 AM, Patrick Hunt wrote: > I'm not very experienced personally with running zk on ec2 smalls, Ted > usually has the ec2 related insight. Given these boxes are not loaded > or lightly loaded, and you've ruled out gc/swap, the only thing I can > think of is that something is going on under the covers at the vm > level that's causing the high latency you're seeing. > > You're seeing 15 _minutes_ max latency. I can't think of what would > cause that inside zk. Any chance that the VM is shutting down or > "freezing" during that period? I dont' know. Are you monitoring that > system from a second system? Perhaps that might shed some light > (monitor the cpu/disk activity using some monitoring tool like > ganglia, nagios, etc... or even more primitive, perhaps doing a ping > to that system and tracking the round trip time/packet loss, dump to a > file and review the next day, etc...) > > Patrick > > On 06/15/2010 03:59 PM, Jordan Zimmerman wrote: >> They're small instances. The thing is that these machines are doing >> next to no work. We're just running simple little tests. The session >> expiration has not happened while I've been watching. It tends to >> happen over night. >> >> -JZ >> >> On Jun 15, 2010, at 1:50 PM, Ted Dunning wrote: >> >>> As usual, the ZK team provides the best feedback. >>> >>> I would be bold enough to ask what kind of ec2 instances you are >>> running on. Small instances are small chunks of larger machines >>> and are sometimes subject to competition for resources from the >>> other tenants. >>> >>> On Tue, Jun 15, 2010 at 12:30 PM, Patrick Hunt<ph...@apache.org> >>> wrote: 3) under-provisioned virtual machines (ie vmware) >>> >>> ... >>> >>> Given that you've ruled out the gc (most common), disk utilization >>> would be the next thing to check. >>> >> >> > -- Eric Bowman Boboco Ltd ebow...@boboco.ie http://www.boboco.ie/ebowman/pubkey.pgp +35318394189/+353872801532
