Hi,
I'm looking at benchmarking several different concurrency libraries
against each other. The benchmarks involve things like repeatedly
sending values between two threads. I'm likely to use Criterion for the
task.
However, one thing I've found is that the libraries have noticeably
different behaviour in terms of the amount of garbage created.
Criterion has an option to perform GC between each benchmark, but I
think that the benchmark is only fair if it takes into account the GC
time for each system; it doesn't seem right for two systems to be
counted as equal if the times to get the results are the same, but then
one has to spend twice as long as the other in GC afterwards. Here's
some options I've considered:
* I could make a small change to Criterion to include the time for
performing GC in each benchmark run, but I worry that running the GC so
often is also misrepresentative (might 100 small GCs take a lot longer
than one large GC of the same data?) -- it may also add a lot of
overhead to quick benchmarks, but I can avoid that problem.
* Alternatively, I could run the GC once at the end of all the runs,
then apportion the cost equally to each of the benchmark times (so if
100 benchmarks require 0.7s of GC, add 0.007s to each time) -- but if GC
is triggered somewhere in the middle of the runs, that upsets the
strategy a little.
* I guess a further alternative is to make each benchmark a whole
program (and decently long), then just time the whole thing, rather than
using Criterion.
Has anyone run into these issues before, and can anyone offer an opinion
on what the best option is?
Thanks,
Neil.
_______________________________________________
Haskell-Cafe mailing list
Haskell-Cafe@haskell.org
http://www.haskell.org/mailman/listinfo/haskell-cafe
