Your first starting point should be the traditional tools (sar, iostat). I would try to generate outputs during times, when the app is reported to be running smoothly and badly for comparison.
Next would probably be bonnie++. You might get useful results using filebench (http://www.solarisinternals.com/wiki/index.php/FileBench), which is capable of simulating different app specific IO patterns. It also provides the opportunity to shape your own IO pattern for testing. Lately, I stumbled across swat/vdbench (http://blogs.sun.com/henk/). Swat (java) records and visualizes the actual IO patterns on given disks and vdbench is capable of replaying these patterns for (see pdfs on download vdbench page). I haven't tried it yet, but the idea of recording IO pattern on disk1 and applying it to disk2 for comparison is pretty cool. This might give you something without actually instrumenting your app for performance testing. Unfortunately none of these is reported to run on FreeBSD. Random thoughts: - what about the average block_size of your IO? - what about side effects from shared resources (iSCSI, network tuning, other systems using same storage) Thomas --------------- Thomas Leyer ([email protected]) Cologne - Germany On Feb 27, 2010, at 7:43 AM, Nicholas Tang wrote: > I'll forward this to a coworker who has a lot of knowledge about the various > tools, and has put together a reasonable test bench for our storage. > > However, one comment: doing benchmarks is not going to give you what you need > to tell the app team that they're hallucinating. Any generic artificial > benchmark is not going to tell you which truly perform better for your app, > because apps require more than just IOPS. They care about things like file > system structure, like the number of files and directories, they care about > latency, they care about random IOPS, they care about disk throughput, they > care about caching, all of that stuff. And unless your benchmark IS their > application running under an actual production load, you're not really going > to get an accurate picture of our their application will perform on different > types of storage. > > It's probably worth putting together benchmarks of the filesystem/hardware > setup combinations so that you have that data handy, but ultimately, > measuring the actual utilization of the storage platform(s) in the production > environment plus having ways to measure the delays in the application will > probably give you a much more compelling argument. That may require > instrumenting the app itself, which can theoretically be done without the > devs modifying it but is typically done more easily with their help. > > As an initial start, have you used anything like SAR or nmon or iostat or the > various other tools out there to measure the utilization of the actual > systems they're complaining about, especially during a period of time that > they've complained about? I'd start there, personally, and see if you see > the system waiting on IO or if you see some other resource getting maxed out. > > Nicholas > > P.S. It's 1:40 am here, I had been asleep but just woke up briefly and while > I was up decided to randomly check email, so I'm not entirely awake or > coherent. If I say anything to offend or that just sounds stupid, I'm > blaming it on that. ;) > > On Sat, Feb 27, 2010 at 1:12 AM, Atom Powers <[email protected]> wrote: > I have a wide variety of storage devices, and I suspect most of them > are underutilized. To make the best use of them I'd like to measure > what their actual performance is: disk IOps per second. > > The storage is SATA, SCSI, and iSCSI attached to Linux and/or FreeBSD, > UFS and ext 3 or 4 on a variety of i386 hardware. I'm not especially > interested in which ones /should/ perform better, I want to measure > which ones /do/ perform better. (And so I can tell my applications > team that they are hallucinating when they blame performance on the > file system.) > > What tools can I use to measure disk performance? > > -- > Perfection is just a word I use occasionally with mustard. > --Atom Powers-- > _______________________________________________ > Tech mailing list > [email protected] > http://lopsa.org/cgi-bin/mailman/listinfo/tech > This list provided by the League of Professional System Administrators > http://lopsa.org/ > > _______________________________________________ > Tech mailing list > [email protected] > http://lopsa.org/cgi-bin/mailman/listinfo/tech > This list provided by the League of Professional System Administrators > http://lopsa.org/
_______________________________________________ Tech mailing list [email protected] http://lopsa.org/cgi-bin/mailman/listinfo/tech This list provided by the League of Professional System Administrators http://lopsa.org/
