Re: our benchmark-suite
Andy Wingo wi...@pobox.com writes: Neat :) (Do you pngcrush these? They seem a little slow to serve.) I just tried running pngcrush on all the .pngs, and didn't get more than 6-8% reduction. So unfortunately it doesn't look like that would help much. Thanks for the idea though! Neil
Re: our benchmark-suite
Howdy! On Wed 25 Apr 2012 22:39, l...@gnu.org (Ludovic Courtès) writes: So, those are the problems: benchmarks running for inappropriate, inconsistent durations; I don’t really see such a problem. It doesn’t matter to me if ‘arithmetic.bm’ takes 2mn while ‘vlists.bm’ takes 40s, since I’m not comparing them. Running a benchmark for 2 minutes is not harmful to the results, but it is a bit needless. One second is enough. However, running a benchmark for just a few milliseconds is not very interesting: ;; (if.bm: if-bool-then: executing then 33 real 0.011994627 real/iteration 3.63473545454545e-8 run/iteration 3.62829060606061e-8 core/iteration 9.61427360606058e-10 gc 0.0) That's 12 milliseconds. The jitter there is too much. inappropriate benchmarks; I agree that things like ‘if.bm’ are not very relevant now. But there are also appropriate benchmarks, and benchmarks are always better than wild guess. ;-) Agreed :-) and benchmarks being optimized out. That should be fixed. In what way? It would make those benchmarks different. Thesis: anything for which you would want to turn off the optimizer is not a good benchmark anyway. See also: http://www.azulsystems.com/presentations/art-of-java-benchmarking My proposal is to rebase the iteration count in 0-reference.bm to run for 0.5s on some modern machine, and adjust all benchmarks to match, removing those benchmarks that do not measure anything useful. Sounds good. However, adjusting iteration counts of the benchmarks themselves should be done rarely, as it breaks performance tracking like http://ossau.homelinux.net/~neil/bm_master_i.html. I think we've established that this isn't the case -- modulo the effect that such a change would have on GC (process image size, etc) Finally we should perhaps enable automatic scaling of the iteration count. What do folks think about that? On the positive side, all of our benchmarks are very clear that they are a time per number of iterations, and so this change should not affect users that measure time per iteration. If the reported time is divided by the global iteration count, then automatic scaling of the global iteration count would be good, yes. OK, will do. Speak now or be surprised by a commit! ;-) Andy -- http://wingolog.org/
Re: our benchmark-suite
Hi! Andy Wingo wi...@pobox.com skribis: On Wed 25 Apr 2012 22:39, l...@gnu.org (Ludovic Courtès) writes: So, those are the problems: benchmarks running for inappropriate, inconsistent durations; I don’t really see such a problem. It doesn’t matter to me if ‘arithmetic.bm’ takes 2mn while ‘vlists.bm’ takes 40s, since I’m not comparing them. Running a benchmark for 2 minutes is not harmful to the results, but it is a bit needless. One second is enough. Well, duration has to be chosen such that the jitter is small enough. Sometimes it could be 2mn, sometimes 1s. [...] and benchmarks being optimized out. That should be fixed. In what way? It would make those benchmarks different. Thesis: anything for which you would want to turn off the optimizer is not a good benchmark anyway. Yes, it depends on the benchmarks. For instance, I once added benchmarks for ‘1+’ and ‘1-’, because I wanted to see the impact of an optimization to the corresponding VM instructions. Nowadays peval would optimize those benchmarks out. Yet, the fact is that I was interested in the performance of the underlying VM instructions, regardless of what the compiler might be doing. Thanks, Ludo’.
Re: our benchmark-suite
Heya Neil, On Fri 04 May 2012 23:43, Neil Jerram n...@ossau.homelinux.net writes: It turns out I'm already scaling by iteration count - in fact since November 2009. :-) Excellent, so we can scale iteration counts in Guile's git with impunity :) It would be nice for the graphs for individual benchmarks to have an absolute Y axis, in terms of microseconds I guess. Still, I wanted to do something new, so I've added further graphs showing just the last 50 measurements for each benchmark (whereas the existing graphs showed all measurements since my data collection began). The generation of those is still running at the moment, but should be complete in an hour or so. Neat :) (Do you pngcrush these? They seem a little slow to serve.) It would also be nice to have overview graphs from the last 50 days as well, should you have time to hack that up. Thanks for this tool, it's neat :) Andy -- http://wingolog.org/
Re: our benchmark-suite
Hi Neil! Neil Jerram n...@ossau.homelinux.net skribis: Still, I wanted to do something new, so I've added further graphs showing just the last 50 measurements for each benchmark (whereas the existing graphs showed all measurements since my data collection began). The generation of those is still running at the moment, but should be complete in an hour or so. In case you have spare time in your hands ;-), Flot [0] provides a very nice UI for plots (Hydra uses it for its history charts.). Also, the GNUnet people have developed a complete tool for performance tracking, Gauger [1]. I haven’t managed to display a single plot from there, but the idea seems nice. Thanks, Ludo’. [0] http://code.google.com/p/flot/ [1] https://gnunet.org/gauger/
Re: our benchmark-suite
For http://ossau.homelinux.net/~neil I do still have all of the raw data including iteration counts, so I could easily implement dividing by the iteration count, and hence allow for future iteration count changes. Is there any downside from doing that? (I don't think so.) No, I guess. And as you show, having raw data instead of synthesized figures gives more freedom. It turns out I'm already scaling by iteration count - in fact since November 2009. :-) Still, I wanted to do something new, so I've added further graphs showing just the last 50 measurements for each benchmark (whereas the existing graphs showed all measurements since my data collection began). The generation of those is still running at the moment, but should be complete in an hour or so. Regards, Neil
Re: our benchmark-suite
Hi, Neil Jerram n...@ossau.homelinux.net skribis: l...@gnu.org (Ludovic Courtès) writes: My proposal is to rebase the iteration count in 0-reference.bm to run for 0.5s on some modern machine, and adjust all benchmarks to match, removing those benchmarks that do not measure anything useful. Sounds good. However, adjusting iteration counts of the benchmarks themselves should be done rarely, as it breaks performance tracking like http://ossau.homelinux.net/~neil/bm_master_i.html. Finally we should perhaps enable automatic scaling of the iteration count. What do folks think about that? On the positive side, all of our benchmarks are very clear that they are a time per number of iterations, and so this change should not affect users that measure time per iteration. If the reported time is divided by the global iteration count, then automatic scaling of the global iteration count would be good, yes. For http://ossau.homelinux.net/~neil I do still have all of the raw data including iteration counts, so I could easily implement dividing by the iteration count, and hence allow for future iteration count changes. Is there any downside from doing that? (I don't think so.) No, I guess. And as you show, having raw data instead of synthesized figures gives more freedom. Thanks, Ludo’.
Re: our benchmark-suite
l...@gnu.org (Ludovic Courtès) writes: My proposal is to rebase the iteration count in 0-reference.bm to run for 0.5s on some modern machine, and adjust all benchmarks to match, removing those benchmarks that do not measure anything useful. Sounds good. However, adjusting iteration counts of the benchmarks themselves should be done rarely, as it breaks performance tracking like http://ossau.homelinux.net/~neil/bm_master_i.html. Finally we should perhaps enable automatic scaling of the iteration count. What do folks think about that? On the positive side, all of our benchmarks are very clear that they are a time per number of iterations, and so this change should not affect users that measure time per iteration. If the reported time is divided by the global iteration count, then automatic scaling of the global iteration count would be good, yes. For http://ossau.homelinux.net/~neil I do still have all of the raw data including iteration counts, so I could easily implement dividing by the iteration count, and hence allow for future iteration count changes. Is there any downside from doing that? (I don't think so.) Regards, Neil
Re: our benchmark-suite
Heya Neil, I pushed a change to the format of the text logged to the console when you do a ./benchmark-guile. It seems that this affected your benchmarking bot. I was hoping that this would not be the case, because the benchmark suite also writes a log to `guile-benchmark.log', and I tried to avoid changing the format of that file. Can you take a look at your bot and see if it's possible to switch to use benchmark-guile.log instead of the console output? Other suggestions as to a solution are also most welcome. Thanks! Andy On Mon 23 Apr 2012 11:22, Andy Wingo wi...@pobox.com writes: Hi, I was going to try to optimize vhash-assoc, but I wanted a good benchmark first, so I started to look at our benchmark suite. We have some issues to deal with. For those of you who are not familiar with the benchmark suite, we have a bunch of benchmarks in benchmark-suite/benchmarks/: those files that end in .bm. The format of a .bm file is like our .test files, except that instead of `pass-if' and the like, we have `benchmark'. You run benchmarks via ./benchmark-guile in the $top_builddir. The benchmarking framework tries to be appropriate for microbenchmarks, as the `benchmark' form includes a suggested number of iterations. Ideally when you create a benchmark, you give it a number of iterations that makes it run approximately as long as the other benchmarks. When the benchmarking suite was first made, 10 years ago, there was an empty reference benchmark that was created to run for approximately 1 second. Currently it runs in 0.012 seconds. This is one problem: the overall suite has old iteration counts. There is a facility for scaling the iteration counts of the suite as a whole, but it is unused. Another problem is that the actual runtime of the various benchmarks varies quite a lot, from 3.3 seconds for assoc (srfi-1), to 0.012 for if.bm. Short runtimes magnify imprecisions in measurement. It used to be that the measurement function was times, but I just changed that to the higher-precision get-internal-real-time / get-internal-run-time. Still, though, there is nothing you can do for a benchmark that runs in a few milliseconds or less. Another big problem is that some effect-free microbenchmarks optimize away. For example, the computations in arithmetic.bm fold entirely. The same goes for if.bm. These benchmarks do not measure anything useful. The benchmarking suite attempts to compensate for the overhead of the test by providing for core time: the time taken to run a benchmark, minus the time taken to run an empty benchmark with the same number of iterations. The benchmark itself is compiled as a thunk, and the framework calls the thunk repeatedly. In theory this sounds good. In practice however, for high-iteration microbenchmarks, the overhead of the thunk call outweighs any micro-benchmark being called. For what it's worth, the current overhead of the benchmark appears to be about 35 microseconds per iteration, on my laptop. If we inline the iteration into the benchmark itself, rather than calling a thunk repeatedly, we can bring that down to around 13 microseconds. However it's probably best to leave it as it is, because if we inline the loop, it's liable to be optimized out. So, those are the problems: benchmarks running for inappropriate, inconsistent durations; inappropriate benchmarks; and benchmarks being optimized out. My proposal is to rebase the iteration count in 0-reference.bm to run for 0.5s on some modern machine, and adjust all benchmarks to match, removing those benchmarks that do not measure anything useful. Finally we should perhaps enable automatic scaling of the iteration count. What do folks think about that? On the positive side, all of our benchmarks are very clear that they are a time per number of iterations, and so this change should not affect users that measure time per iteration. Regards, Andy -- http://wingolog.org/
our benchmark-suite
Hi, I was going to try to optimize vhash-assoc, but I wanted a good benchmark first, so I started to look at our benchmark suite. We have some issues to deal with. For those of you who are not familiar with the benchmark suite, we have a bunch of benchmarks in benchmark-suite/benchmarks/: those files that end in .bm. The format of a .bm file is like our .test files, except that instead of `pass-if' and the like, we have `benchmark'. You run benchmarks via ./benchmark-guile in the $top_builddir. The benchmarking framework tries to be appropriate for microbenchmarks, as the `benchmark' form includes a suggested number of iterations. Ideally when you create a benchmark, you give it a number of iterations that makes it run approximately as long as the other benchmarks. When the benchmarking suite was first made, 10 years ago, there was an empty reference benchmark that was created to run for approximately 1 second. Currently it runs in 0.012 seconds. This is one problem: the overall suite has old iteration counts. There is a facility for scaling the iteration counts of the suite as a whole, but it is unused. Another problem is that the actual runtime of the various benchmarks varies quite a lot, from 3.3 seconds for assoc (srfi-1), to 0.012 for if.bm. Short runtimes magnify imprecisions in measurement. It used to be that the measurement function was times, but I just changed that to the higher-precision get-internal-real-time / get-internal-run-time. Still, though, there is nothing you can do for a benchmark that runs in a few milliseconds or less. Another big problem is that some effect-free microbenchmarks optimize away. For example, the computations in arithmetic.bm fold entirely. The same goes for if.bm. These benchmarks do not measure anything useful. The benchmarking suite attempts to compensate for the overhead of the test by providing for core time: the time taken to run a benchmark, minus the time taken to run an empty benchmark with the same number of iterations. The benchmark itself is compiled as a thunk, and the framework calls the thunk repeatedly. In theory this sounds good. In practice however, for high-iteration microbenchmarks, the overhead of the thunk call outweighs any micro-benchmark being called. For what it's worth, the current overhead of the benchmark appears to be about 35 microseconds per iteration, on my laptop. If we inline the iteration into the benchmark itself, rather than calling a thunk repeatedly, we can bring that down to around 13 microseconds. However it's probably best to leave it as it is, because if we inline the loop, it's liable to be optimized out. So, those are the problems: benchmarks running for inappropriate, inconsistent durations; inappropriate benchmarks; and benchmarks being optimized out. My proposal is to rebase the iteration count in 0-reference.bm to run for 0.5s on some modern machine, and adjust all benchmarks to match, removing those benchmarks that do not measure anything useful. Finally we should perhaps enable automatic scaling of the iteration count. What do folks think about that? On the positive side, all of our benchmarks are very clear that they are a time per number of iterations, and so this change should not affect users that measure time per iteration. Regards, Andy -- http://wingolog.org/
