Hi Stefan, OK, you proved I was wrong. I said I could. Thanks for clarification!
Jan On Thu, 2015-11-05 at 17:17 +0100, [email protected] wrote: > Hi Jan, > Hi Max: > > I guess, the main issues is missing documentation… > Even so, there are class comments… > > > On 01 Nov 2015, at 23:45, Jan Vrany <[email protected]> wrote: > > > > Hi Max, > > > > I looked at some version of SMark years ago and never used > > it extensively, so I might be wrong, but: > > > > * SMark executor does some magic with numbers. > > Nope. It does only do that if you ask for it. However, granted, > that’s the standard setting, because it is supposed to be used > conveniently from within the image. > > The SMark design knows the concepts reporter (how and what data to > report), runner (how to execute benchmarks), suite (the benchmarks), > timer (should be named gauge or something, can be everything, doesn’t > have to be time). > > > > It tries to > > calculate a number of iterations to run in order to get > > "statistically meaningful results". Maybe it's me, but > > I could not fully understand what it does and why it does it > > so. > > CalipeL does no magic - it gives you raw numbers (no average, no > > mean, > > rather a sequence of measurements). > > See the ReBenchHarness, that’s giving you exactly that as alternative > standard setting. > > > * SMark, IIRC, requires benchmarks to inherit from some base class > > (like SUnit). > > Require is a strong word, as long as you implement the interface of > SMarkSuite you can inherit from where ever you want. It’s Smalltalk > after all. > > > > Also, not sure if SMark allows you to specify a warmup- > > phase (handy for example to measure peak performance when caches > > are > > filled or so). > > There is the concept of #setup/teardown methods. > And, a runner can do what it wants/needs to reach warmup, too. > For instance, the SMarkCogRunner will make sure that all code is > compiled before starting to measure. > > > CalipeL, OTOH, uses method annotations to describe the benchmark, > > so one can turn a regular SUnit test method into benchmark as > > simply > > as annotating it with <benchmark>. > > Ok, that’s not possible. > > > A warmup method and setup/teardown > > methods can be specified per-benchmark. > > We got that too. > > > * SMark has no support for parametrization. > > Well, there is the #problemSize parameter, but that is indeed rather > simplistic. > > > > > * SMark measures time only. > > Nope, the SMarkTimer can measure what they want. (and it even got a > class comment ;)) > > > * SMark had no support for “system" profilers and similar. > > That’s absent, true. > > > * Finally, SMark spits out a report and that’s it. > > Well, reports and raw data. I use ReBench [1], and pipe the raw data > directly into my latex/knitr/R tool chain to generate graphs/numbers > in my papers (example sec. 4 [2], that’s based on a latex file with > embedded R). > > So, I’d say there are some interesting differences. > But, much of the mentioned things seem just to be missing > ‘documentation’/communication ;) > > Best regards > Stefan > > [1] https://github.com/smarr/ReBench > [2] http://stefan-marr.de/papers/oopsla-marr-ducasse-meta-tracing-vs- > partial-evaluation/ > > > > >
