Kristian Nielsen wrote:
Jay Pipes <[email protected]> writes:
Paul McCullagh wrote:
As has been mentioned elsewhere the most important thing is that Drizzle
has automated regression tests in place.
Yep, that plus we are discussing these regressions in a public forum
where many excellent folks like yourself can comment on them. :) The
suggestions and input we get from everyone is invaluable, and we follow
up on all suggestions.
One thing I am curious about: what is the smallest regression that you can
discover without it being lost in noise? I think this one was like 20% or so,
but do you think you could catch one of 5%? 1%?
Hi! Actually, I was wondering the same thing the other day:
https://lists.launchpad.net/drizzle-discuss/msg03904.html
I think that post got lost in the flurry of ML posts that day...
Anyway, the automated regression tests store all results of all revision
benchmarks, so really, it's just about deciding what percentage change
should trigger a notification to the mailing list or IRC channel to
prompt further investigation (like the one I did to identify the
std::bitset regression)...
1%? 5%? Averaged across all concurrency levels? Compared with prior
revision or with the past X revisions? There's many ways we can do
it...just have to decide on the variables :)
I strongly agree with letting performance regression testing control
optimizations. I have seen _so_ many code decisions based on "assumed"
performance benefits over clear code. With modern CPUs it's just plain wrong
90% of the time. Even "every instruction counts" is not right, as typical code
I see utilise only 40-50% of instruction bandwidth on something like a Core 2
(hint: latency and dependency chains matter, not instruction counts).
What I have in mind is something like Buildbot/Pushbuild, but for
performance. So there would be performance regression tests for each major
piece of code, and any push that caused a slowdown would be flagged by a red
in the status display.
Have you checked out the Drizzle Automation project yet? I'm currently
documenting it, but it is the tool we've used in all our benchmarking so
far and provides a clean interface for benchmarking series of revisions
on a Drizzle branch and also has the ability to use MySQL Sandbox for
benchmarking stand-alone MySQL servers (we can pop in an adapter for
MariaDb fairly easily if you guys have a separate/different build process?
The really hard part of this is the noise. Usually there is quite a bit of
random noise in benchmarks, so I was looking at ways to reduce that noise to
make automation feasible.
Well, currently, the benchmarks are pretty consistent. I run 60 second
readonly and readwrite runs at these concurrency levels:
2,4,8,16,32,64,128,256,512,1024,2048
For readwrite, I only run up to 1024 connections because of a limitation
in InnoDB previously discussed on the list:
https://lists.launchpad.net/drizzle-discuss/msg03956.html
For all runs, I do 3 6-second iterations and the automation suite stores
the results of each run in a database. All I do to produce the results
is a simple SQL query which averages across all 3 iterations of the
results for each concurrency level. Today, I'll be finishing up the
automatic polling mechanism which will put us into fully-automated mode,
where any push to the trunk/staging tree will automatically run our
regression suite (benchmarks and LCOV coverage reports).
Sahil Cooner (neuro_damage on IRC) is working on the website part of
things, where we'll work together to produce pretty graphs in a
dashboard-like output.
I worked a bit with the perfmon2 library, and got some really good
results. For micro-benchmarks of smaller pieces of code, I was able to measure
with about 1 in 10000 accuracy, which is really quite good, and would enable
to detect automatically something down to like a 0.1% slowdown or so perhaps.
However, this requires having a test suite of micro-benchmarks, I do not think
it is possible to get something like sysbench down to that accuracy.
No, me neither. What we're after right now is making sure we don't have
significant regressions from one revision to another, and catching them
if they happen (like has been done on the bitset work)
Cheers!
Jay
Anyway, then I got involved in MariaDB and had to put the project away for a
while (I hope to return to it eventually). But I was just curious about your
thoughts and experience with handling benchmark measurement noise.
- Kristian.
_______________________________________________
Mailing list: https://launchpad.net/~drizzle-discuss
Post to : [email protected]
Unsubscribe : https://launchpad.net/~drizzle-discuss
More help : https://help.launchpad.net/ListHelp
_______________________________________________
Mailing list: https://launchpad.net/~drizzle-discuss
Post to : [email protected]
Unsubscribe : https://launchpad.net/~drizzle-discuss
More help : https://help.launchpad.net/ListHelp
