I'll toss out what we just recently discovered on an IBM 3500 SAN. After much log generating and reviewing counters showing excessive disk latency in VMware, IBM Hardware Tech support reviewing logs and finally getting to LSI to have them look at the controller info, LSI discovered 1 of the disks had an issue with writing to the disk cache. Nobody noticed it and the counters didn't seem big enough to make a difference. After all, it was just one drive out of 12 in a Raid 5 array, what difference would that make? Well, replaced the drive and it dropped the latency from above 300ms for the array down to 5 to 10 ms.
Long story short, check your drive diagnostics for any issues before you lay the blame on your other hardware. rr From: Joseph L. Casale [mailto:[email protected]] Sent: Wednesday, April 06, 2011 8:33 PM To: NT System Admin Issues Subject: RE: VMware disk latency Yes, and that is why I asked. HP has several low end controllers for those boxes and they are mind boggling POS's. Toss in a good battery backed (cache enabled) controller and see the difference. Said while staring at a 350 G5 and G6 running esxi after having been through this... jlc From: Bill Songstad [mailto:[email protected]] Sent: Wednesday, April 06, 2011 5:17 PM To: NT System Admin Issues Subject: Re: VMware disk latency Joseph, I can't find the build info for that box, but it looks to be a built in SATA controller. (SmartArray E200i) Does that sound like a viable option? Thanks, Bill On Wed, Apr 6, 2011 at 2:28 PM, Joseph L. Casale <[email protected]<mailto:[email protected]>> wrote: Which controller on the ML? From: Bill Songstad [mailto:[email protected]<mailto:[email protected]>] Sent: Wednesday, April 06, 2011 1:20 PM To: NT System Admin Issues Subject: OT: VMware disk latency I have an ESXi4 setup on a HP ML350 G5 running a RAID5 array of sata drives for the primary datastore and a drobo iscsi array for the secondary datastore. 2 VMs on each datastore. 4 VMs total. All w2k8. When I do a large file copy to any of the VMs from a share on a different physical machine there is huge latency, and not just on the machines hosted on the drobo. Okay, not the fastest disk arrays in the world, but I wonder if my bottleneck is elsewhere though. Durring the file copy, latency on the physical disks shoots up to 200-500 Milliseconds. But on the Virtual disks, the latency is literally 10 times that, with the other machine on that datastore essentially mirroring the physical disk. Subsequently, the machines effectively stop responding. Durring a file copy to vm1 the latencies look like this: Physical DAS latency 300 ms vm1 vdisk on DAS latency 3,000 ms vm2 vdisk on DAS latency 300 ms Physical iSCSI latency 0 ms vm3 vdisk latency 0 ms vm4 vdisk latency 0 ms Is this to be expected? Why the ten-fold discrepancy between the actual disk and the virtual disk? Any insights would be appreciated, -Bill ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected]<mailto:[email protected]> with the body: unsubscribe ntsysadmin ~ Finally, powerful endpoint security that ISN'T a resource hog! ~ ~ <http://www.sunbeltsoftware.com/Business/VIPRE-Enterprise/> ~ --- To manage subscriptions click here: http://lyris.sunbelt-software.com/read/my_forums/ or send an email to [email protected] with the body: unsubscribe ntsysadmin
