On 12/19/05 5:02 PM, Perrin Harkins wrote: > On Mon, 2005-12-19 at 08:33 -0500, John Siracusa wrote: >> Yes, the db itself will usually be the biggest performance drain. But the >> point of this benchmark suite is to isolate the overhead caused by the ORM >> itself. > > Aren't unnecessary extra queries the biggest source of overhead though?
Well, that's really just an extension of "the db is the slowest part" rule. > That seems worth measuring to me. Now I'm thinking maybe this benchmark > should measure wall time instead of CPU as well. That would make extra > queries carry a higher penalty. Pass the "--time" and (optionally) the "--hi-res-time" flags to bench.pl to see a wall clock comparison. You have to take great pains to ensure that no other significant processes are running when you do this, however. If, say, new mail arrives in your email client just as one ORM begins its test run, it can throw off the wallclock time significantly. This is especially true when the run times are within a second or two of each other. It doesn't take much to throw it off. One defense against this is to set the number of iterations and min CPU time to be huge, in the hopes of evening out the noise. But that can take a looong time to finish. Anyway, the way the bench is designed, the i/o wait and other db overhead should be exactly equal for each ORM . They're all using the same db, and they all have their own private subset of rows, in an attempt to defeat OS and db caching effects across ORMs. Pointing the bench at different dbs is also useful. (I keep meaning to add SQLite to the bench, but MySQL and Pg work now. Informix worked in the bench the last time I checked, but that was a while ago...) On 12/19/05 4:51 PM, Sam Vilain wrote: > On Mon, 2005-12-19 at 08:33 -0500, John Siracusa wrote: >>> This is one of the major reasons why I've never bothered with optimising >>> performance for this situation - by coding in the correct manner, the >>> number of very small database hits is minimised. Then it doesn't really >>> matter how long they take. >> I think you misunderstand what the tests are doing. They're meant to >> highlight exactly the features that you're describing. Fetching objects and >> some of their associated objects in other tables is a common task. Many of >> the tests revolve around this task (with various twists). There are other, >> earlier tests that measure the performance of single-row loads, and so on. > > Well, I still think it's timing of the wrong thing What should it be timing? The particular test you were looking at (search products and categories and code names) represents a common, practical task, and does in fact reward ORMs that can accomplish this task with as few queries as possible. > but I'll be happy to participate. Good :) I'll work on integrating Tangram into the bench for the next release. -John ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click _______________________________________________ Rose-db-object mailing list Rose-db-object@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/rose-db-object
