On 2011-03-11, at 15:35, Mariano Martinez Peck wrote: > On Fri, Mar 11, 2011 at 3:23 PM, Camillo Bruni <[email protected]>wrote: > >> I am maybe an ignorant here writing like that. I like to see a nice set of >> benchmarks popping up in Pharo. >> >> But did you have a look at my benchmark implementation we use in Pinocchio? >> Its in the PBenchmark package of the Pinocchio project: >> >> MCHttpRepository >> location: 'http://www.squeaksource.com/p' >> user: '' >> password: '' >> >> Its a fairly straight forward implementation based on UnitTests... >> >> I see several issues in your benchmark implementation which I think are >> solved much cleaner with my approach: >> >> - currently there is one single class with tons of benchmarks in it >> - no statistically valid output ( just the average doesn't mean anything! >> see http://dx.doi.org/10.1145/1297105.1297033 for the basic scientific >> backgrounds) >> - you interleave model (benchmarks and results) and view (transcript >> output) which is really evil, there is no way you can ever use this on the >> command line! >> >> so what I suggest, is that you have a look at my implementation and see how >> we can improve the current situation. >> > > There is no class comment, no wiki, no class side examples.... > I just tried > PBenchmarkSuite new runAll > but I didn't get anything. > > cheers > > mariano
Sorry, indeed no documentation yet :). But as I said it works like the UnitTest suite, hence there is nothing to run in there. Try one of the actual test-classes like PBString run and you will get a PBenchmarkRun back, as you would with a TestCase. let me know if that helps :). camillo > >> >> >> m(^_-)m >> camillo >> >> On 2011-03-11, at 15:05, Alexandre Bergel wrote: >> >>> I am not sure what you mean with vm benchmarks, but in almost all the >> tools I am contributing come with some benchmarks (spy, Mondrian, I wrote >> some benchmarks for Glamour as well). Naturally, those are macro benchmarks, >> which is probably what matter the most. >>> >>> Cheers, >>> Alexandre >>> >>> >>> >>> Le 11 mars 2011 à 10:55, Mariano Martinez Peck <[email protected]> a >> écrit : >>> >>>> does anyone have news regarding this topic? we VERY welcome people >> helping with benchmarks. >>>> please feel free to improve http://www.squeaksource.com/PharoBenchmarks >>>> >>>> >>>> >>>> On Tue, Jan 4, 2011 at 11:51 PM, Stefan Marr <[email protected]> >> wrote: >>>> Hi: >>>> >>>> On 04 Jan 2011, at 23:40, Mariano Martinez Peck wrote: >>>> >>>>> On Tue, Jan 4, 2011 at 11:35 PM, Stefan Marr <[email protected]> >> wrote: >>>>> Hi Igor: >>>>> >>>>> On 04 Jan 2011, at 22:40, Igor Stasenko wrote: >>>>> >>>>>> Okay, how about creating a separate >>>>>> VMBenchmarks repository >>>>>> and putting VMBenchmarks package there? >>>>> >>>>> Sure, sounds good. There are also the Systems benchmarks at >> http://www.squeaksource.com/PharoBenchmarks. >>>>> >>>>> >>>>> So, at least, I was able to run all benchmarks :) >>>>> >>>>> However, it still needs cleaning, improvement, testing, blah. But it >> is a good start point I think. >>>> >>>> Well, I am not to sure about the general value of those benchmarks. >>>> There are many microbenchmarks which do not tell you a lot. All those >> test* things. >>>> And well, their value for testing is also questionable. They only can >> help you to identify where it goes *boom* and crashes the VM, but they do >> not actually assert for anything. >>>> >>>> Also not sure what the value of Slopstone and Smopstone (names from the >> top of my head might be slightly different) is nowadays. >>>> >>>> The compiler benchmark is a reasonable application benchmark. >>>> Would be good to have a few others in that collection, too. >>>> >>>> Best regards >>>> Stefan >>>> >>>> >>>> >>>> >>>> -- >>>> Stefan Marr >>>> Software Languages Lab >>>> Vrije Universiteit Brussel >>>> Pleinlaan 2 / B-1050 Brussels / Belgium >>>> http://soft.vub.ac.be/~smarr >>>> Phone: +32 2 629 2974 >>>> Fax: +32 2 629 3525 >>>> >>>> >>>> >> >> >>
