> Its a fairly straight forward implementation based on UnitTests... >
Yeah. Actually, a very good benchmark is running unit tests. People from docomo do that. Alexandre > I see several issues in your benchmark implementation which I think are > solved much cleaner with my approach: > > - currently there is one single class with tons of benchmarks in it > - no statistically valid output ( just the average doesn't mean anything! see > http://dx.doi.org/10.1145/1297105.1297033 for the basic scientific > backgrounds) > - you interleave model (benchmarks and results) and view (transcript output) > which is really evil, there is no way you can ever use this on the command > line! > > so what I suggest, is that you have a look at my implementation and see how > we can improve the current situation. > > > m(^_-)m > camillo > > On 2011-03-11, at 15:05, Alexandre Bergel wrote: > >> I am not sure what you mean with vm benchmarks, but in almost all the tools >> I am contributing come with some benchmarks (spy, Mondrian, I wrote some >> benchmarks for Glamour as well). Naturally, those are macro benchmarks, >> which is probably what matter the most. >> >> Cheers, >> Alexandre >> >> >> >> Le 11 mars 2011 à 10:55, Mariano Martinez Peck <[email protected]> a >> écrit : >> >>> does anyone have news regarding this topic? we VERY welcome people helping >>> with benchmarks. >>> please feel free to improve http://www.squeaksource.com/PharoBenchmarks >>> >>> >>> >>> On Tue, Jan 4, 2011 at 11:51 PM, Stefan Marr <[email protected]> wrote: >>> Hi: >>> >>> On 04 Jan 2011, at 23:40, Mariano Martinez Peck wrote: >>> >>>> On Tue, Jan 4, 2011 at 11:35 PM, Stefan Marr <[email protected]> wrote: >>>> Hi Igor: >>>> >>>> On 04 Jan 2011, at 22:40, Igor Stasenko wrote: >>>> >>>>> Okay, how about creating a separate >>>>> VMBenchmarks repository >>>>> and putting VMBenchmarks package there? >>>> >>>> Sure, sounds good. There are also the Systems benchmarks at >>>> http://www.squeaksource.com/PharoBenchmarks. >>>> >>>> >>>> So, at least, I was able to run all benchmarks :) >>>> >>>> However, it still needs cleaning, improvement, testing, blah. But it is a >>>> good start point I think. >>> >>> Well, I am not to sure about the general value of those benchmarks. >>> There are many microbenchmarks which do not tell you a lot. All those test* >>> things. >>> And well, their value for testing is also questionable. They only can help >>> you to identify where it goes *boom* and crashes the VM, but they do not >>> actually assert for anything. >>> >>> Also not sure what the value of Slopstone and Smopstone (names from the top >>> of my head might be slightly different) is nowadays. >>> >>> The compiler benchmark is a reasonable application benchmark. >>> Would be good to have a few others in that collection, too. >>> >>> Best regards >>> Stefan >>> >>> >>> >>> >>> -- >>> Stefan Marr >>> Software Languages Lab >>> Vrije Universiteit Brussel >>> Pleinlaan 2 / B-1050 Brussels / Belgium >>> http://soft.vub.ac.be/~smarr >>> Phone: +32 2 629 2974 >>> Fax: +32 2 629 3525 >>> >>> >>> > >
