our benchmark-suite
Hi, I was going to try to optimize vhash-assoc, but I wanted a good benchmark first, so I started to look at our benchmark suite. We have some issues to deal with. For those of you who are not familiar with the benchmark suite, we have a bunch of benchmarks in benchmark-suite/benchmarks/: those files that end in .bm. The format of a .bm file is like our .test files, except that instead of `pass-if' and the like, we have `benchmark'. You run benchmarks via ./benchmark-guile in the $top_builddir. The benchmarking framework tries to be appropriate for microbenchmarks, as the `benchmark' form includes a suggested number of iterations. Ideally when you create a benchmark, you give it a number of iterations that makes it run approximately as long as the other benchmarks. When the benchmarking suite was first made, 10 years ago, there was an empty reference benchmark that was created to run for approximately 1 second. Currently it runs in 0.012 seconds. This is one problem: the overall suite has old iteration counts. There is a facility for scaling the iteration counts of the suite as a whole, but it is unused. Another problem is that the actual runtime of the various benchmarks varies quite a lot, from 3.3 seconds for assoc (srfi-1), to 0.012 for if.bm. Short runtimes magnify imprecisions in measurement. It used to be that the measurement function was times, but I just changed that to the higher-precision get-internal-real-time / get-internal-run-time. Still, though, there is nothing you can do for a benchmark that runs in a few milliseconds or less. Another big problem is that some effect-free microbenchmarks optimize away. For example, the computations in arithmetic.bm fold entirely. The same goes for if.bm. These benchmarks do not measure anything useful. The benchmarking suite attempts to compensate for the overhead of the test by providing for core time: the time taken to run a benchmark, minus the time taken to run an empty benchmark with the same number of iterations. The benchmark itself is compiled as a thunk, and the framework calls the thunk repeatedly. In theory this sounds good. In practice however, for high-iteration microbenchmarks, the overhead of the thunk call outweighs any micro-benchmark being called. For what it's worth, the current overhead of the benchmark appears to be about 35 microseconds per iteration, on my laptop. If we inline the iteration into the benchmark itself, rather than calling a thunk repeatedly, we can bring that down to around 13 microseconds. However it's probably best to leave it as it is, because if we inline the loop, it's liable to be optimized out. So, those are the problems: benchmarks running for inappropriate, inconsistent durations; inappropriate benchmarks; and benchmarks being optimized out. My proposal is to rebase the iteration count in 0-reference.bm to run for 0.5s on some modern machine, and adjust all benchmarks to match, removing those benchmarks that do not measure anything useful. Finally we should perhaps enable automatic scaling of the iteration count. What do folks think about that? On the positive side, all of our benchmarks are very clear that they are a time per number of iterations, and so this change should not affect users that measure time per iteration. Regards, Andy -- http://wingolog.org/
Problems with compilation on Trisquel 5.5
Dear readers, Recently I upgraded my machine to Trisquel 5.5 (x86_64). While I used Trisquel 5, I had no issues compiling guile, however now it appears all hell broke losoe. When compiling I get to the generation stage of GEN guile-procedures.texi. The stage ends up with a Segmentation fault. I have been debugging the lt_guile process with gdb, and some interesting things happened. The process received SIGPWR and SIGXCPU signals (in a loop), and eventually ends up with a SIGSEGV message from the procedure: GC_generic_malloc_inner. I ended up attempting to build the bdw-gc from scratch and link in the 7.1 stable version, but resulting in the same situation. I have no idea where the problem initiates, as it appears to be happening in a certain time interval (probably thread switching) and there is no way to pinpoint the problem in scheme itself. Even just looping around generates this error. Now, when I went through the make output I noticed some interesting things while compiling libguile_2.2_la-weak-table.lo: weak-table.c: In function 'move_disappearing_links': weak-table.c:164:7: warning: passing argument 1 of 'GC_unregister_disappearing_link' from incompatible pointer type [enabled by default] /usr/local/include/gc/gc.h:771:12: note: expected 'void **' but argument is of type 'scm_t_bits *' weak-table.c:165:7: warning: passing argument 1 of 'GC_general_register_disappearing_link' from incompatible pointer type [enabled by default] /usr/local/include/gc/gc.h:744:12: note: expected 'void **' but argument is of type 'scm_t_bits *' weak-table.c:175:7: warning: passing argument 1 of 'GC_unregister_disappearing_link' from incompatible pointer type [enabled by default] /usr/local/include/gc/gc.h:771:12: note: expected 'void **' but argument is of type 'scm_t_bits *' weak-table.c:176:7: warning: passing argument 1 of 'GC_general_register_disappearing_link' from incompatible pointer type [enabled by default] /usr/local/include/gc/gc.h:744:12: note: expected 'void **' but argument is of type 'scm_t_bits *' And although perhaps less important, when compiling libguile_2.2_la-gc.lo: gc.c:212:1: warning: 'GC_get_free_space_divisor' defined but not used [-Wunused-function] As I can't see anyone (yet) having this problem, I consider it might be rather unique, but I really would like to know what possibly could be going wrong and where I should start debugging.
Re: Problems with compilation on Trisquel 5.5
I found out, as a response to myself that the tag guile-2.0.5 does not share the same issues as the master branch. I will attempt to figure out where and when the problem is introduced, but it will be helpful if anyone knows about a change which might involve different/weird GC behavior. Regards, Sjoerd 2012/4/23 Sjoerd van Leent svanle...@gmail.com Dear readers, Recently I upgraded my machine to Trisquel 5.5 (x86_64). While I used Trisquel 5, I had no issues compiling guile, however now it appears all hell broke losoe. When compiling I get to the generation stage of GEN guile-procedures.texi. The stage ends up with a Segmentation fault. I have been debugging the lt_guile process with gdb, and some interesting things happened. The process received SIGPWR and SIGXCPU signals (in a loop), and eventually ends up with a SIGSEGV message from the procedure: GC_generic_malloc_inner. I ended up attempting to build the bdw-gc from scratch and link in the 7.1 stable version, but resulting in the same situation. I have no idea where the problem initiates, as it appears to be happening in a certain time interval (probably thread switching) and there is no way to pinpoint the problem in scheme itself. Even just looping around generates this error. Now, when I went through the make output I noticed some interesting things while compiling libguile_2.2_la-weak-table.lo: weak-table.c: In function 'move_disappearing_links': weak-table.c:164:7: warning: passing argument 1 of 'GC_unregister_disappearing_link' from incompatible pointer type [enabled by default] /usr/local/include/gc/gc.h:771:12: note: expected 'void **' but argument is of type 'scm_t_bits *' weak-table.c:165:7: warning: passing argument 1 of 'GC_general_register_disappearing_link' from incompatible pointer type [enabled by default] /usr/local/include/gc/gc.h:744:12: note: expected 'void **' but argument is of type 'scm_t_bits *' weak-table.c:175:7: warning: passing argument 1 of 'GC_unregister_disappearing_link' from incompatible pointer type [enabled by default] /usr/local/include/gc/gc.h:771:12: note: expected 'void **' but argument is of type 'scm_t_bits *' weak-table.c:176:7: warning: passing argument 1 of 'GC_general_register_disappearing_link' from incompatible pointer type [enabled by default] /usr/local/include/gc/gc.h:744:12: note: expected 'void **' but argument is of type 'scm_t_bits *' And although perhaps less important, when compiling libguile_2.2_la-gc.lo: gc.c:212:1: warning: 'GC_get_free_space_divisor' defined but not used [-Wunused-function] As I can't see anyone (yet) having this problem, I consider it might be rather unique, but I really would like to know what possibly could be going wrong and where I should start debugging.
Re: Broken Backtraces, and Part of a Solution
Hello, After looking at it more, there aren't really enough stack functions to warrant a test suite. Sounds like a fallacious argument to me. ;-) Would it be possible for you to add test cases? There are a few tests under “stacks” in eval.test; perhaps something along these lines would work? Yes, you're right. Those tests showed me how to get a stack. I pushed the fix with two new test cases, and then cleaned up the stack-test code a bit. Thanks a lot, Noah
