On Wed, Jul 29, 2009 at 12:07 PM, Scott Carey <[email protected]>wrote:
> > On 7/29/09 11:47 AM, "Todd Lipcon" <[email protected]> wrote: > > > On Wed, Jul 29, 2009 at 2:17 AM, Steve Loughran <[email protected]> > wrote: > > > >> Scott Carey wrote: > >> > >>> Well, the first thing to do in any performance bottleneck investigation > is > >>> to look at the machine hardware resource usage. > >>> > >>> During your test, what is the CPU use and disk usage? What about > network > >>> utilization? > >>> Top, vmstat, iostat, and some network usage monitoring would be useful. > >>> It > >>> could be many things causing your lack of scalability, but without > >>> actually > >>> monitoring your machines to see if there is an obvious bottleneck its > just > >>> random guessing and hunches. > >>> > >> > >> iotop is very good here > >> > > > > But sadly not available with kernels before 2.6.20 :( I was looking into > > this recently and the next best thing I could find was blktrace, but it's > > not nearly as nice. > > > > Any other tools people know about, especially that work on RHEL5/CentOS5? > > > > Iftop worked for me for network traffic: > http://www.ex-parrot.com/pdw/iftop/ > > For disk monitoring on CentOS/RHEL 5 Linux, every time I try and drill down > a problem with iostat I wish I was on Windows or Solaris instead where > monitoring and tracing disk I/O isn't a decade behind. On both of those, > not only can you see what process is causing the disk activity, you can > find > out what FILES the activity is on. > Yep, dtrace is great for this. I imagine if one were clever they could use strace to look at IO patterns on a per-process level. If anyone knows of any handy wrappers to do this, that would be great for looking at performance issues like this. -Todd
