Jay Pipes <[email protected]> writes: > Paul McCullagh wrote: >> As has been mentioned elsewhere the most important thing is that Drizzle >> has automated regression tests in place. > > Yep, that plus we are discussing these regressions in a public forum > where many excellent folks like yourself can comment on them. :) The > suggestions and input we get from everyone is invaluable, and we follow > up on all suggestions.
One thing I am curious about: what is the smallest regression that you can discover without it being lost in noise? I think this one was like 20% or so, but do you think you could catch one of 5%? 1%? I strongly agree with letting performance regression testing control optimizations. I have seen _so_ many code decisions based on "assumed" performance benefits over clear code. With modern CPUs it's just plain wrong 90% of the time. Even "every instruction counts" is not right, as typical code I see utilise only 40-50% of instruction bandwidth on something like a Core 2 (hint: latency and dependency chains matter, not instruction counts). What I have in mind is something like Buildbot/Pushbuild, but for performance. So there would be performance regression tests for each major piece of code, and any push that caused a slowdown would be flagged by a red in the status display. The really hard part of this is the noise. Usually there is quite a bit of random noise in benchmarks, so I was looking at ways to reduce that noise to make automation feasible. I worked a bit with the perfmon2 library, and got some really good results. For micro-benchmarks of smaller pieces of code, I was able to measure with about 1 in 10000 accuracy, which is really quite good, and would enable to detect automatically something down to like a 0.1% slowdown or so perhaps. However, this requires having a test suite of micro-benchmarks, I do not think it is possible to get something like sysbench down to that accuracy. Anyway, then I got involved in MariaDB and had to put the project away for a while (I hope to return to it eventually). But I was just curious about your thoughts and experience with handling benchmark measurement noise. - Kristian. _______________________________________________ Mailing list: https://launchpad.net/~drizzle-discuss Post to : [email protected] Unsubscribe : https://launchpad.net/~drizzle-discuss More help : https://help.launchpad.net/ListHelp
