On Fri, Mar 11, 2011 at 3:23 PM, Camillo Bruni <[email protected]>wrote:
> I am maybe an ignorant here writing like that. I like to see a nice set of > benchmarks popping up in Pharo. > > But did you have a look at my benchmark implementation we use in Pinocchio? > Its in the PBenchmark package of the Pinocchio project: > > MCHttpRepository > location: 'http://www.squeaksource.com/p' > user: '' > password: '' > > Its a fairly straight forward implementation based on UnitTests... > > I see several issues in your benchmark implementation which I think are > solved much cleaner with my approach: > > - currently there is one single class with tons of benchmarks in it > - no statistically valid output ( just the average doesn't mean anything! > see http://dx.doi.org/10.1145/1297105.1297033 for the basic scientific > backgrounds) > - you interleave model (benchmarks and results) and view (transcript > output) which is really evil, there is no way you can ever use this on the > command line! > > so what I suggest, is that you have a look at my implementation and see how > we can improve the current situation. > There is no class comment, no wiki, no class side examples.... I just tried PBenchmarkSuite new runAll but I didn't get anything. cheers mariano > > > m(^_-)m > camillo > > On 2011-03-11, at 15:05, Alexandre Bergel wrote: > > > I am not sure what you mean with vm benchmarks, but in almost all the > tools I am contributing come with some benchmarks (spy, Mondrian, I wrote > some benchmarks for Glamour as well). Naturally, those are macro benchmarks, > which is probably what matter the most. > > > > Cheers, > > Alexandre > > > > > > > > Le 11 mars 2011 à 10:55, Mariano Martinez Peck <[email protected]> a > écrit : > > > >> does anyone have news regarding this topic? we VERY welcome people > helping with benchmarks. > >> please feel free to improve http://www.squeaksource.com/PharoBenchmarks > >> > >> > >> > >> On Tue, Jan 4, 2011 at 11:51 PM, Stefan Marr <[email protected]> > wrote: > >> Hi: > >> > >> On 04 Jan 2011, at 23:40, Mariano Martinez Peck wrote: > >> > >>> On Tue, Jan 4, 2011 at 11:35 PM, Stefan Marr <[email protected]> > wrote: > >>> Hi Igor: > >>> > >>> On 04 Jan 2011, at 22:40, Igor Stasenko wrote: > >>> > >>>> Okay, how about creating a separate > >>>> VMBenchmarks repository > >>>> and putting VMBenchmarks package there? > >>> > >>> Sure, sounds good. There are also the Systems benchmarks at > http://www.squeaksource.com/PharoBenchmarks. > >>> > >>> > >>> So, at least, I was able to run all benchmarks :) > >>> > >>> However, it still needs cleaning, improvement, testing, blah. But it > is a good start point I think. > >> > >> Well, I am not to sure about the general value of those benchmarks. > >> There are many microbenchmarks which do not tell you a lot. All those > test* things. > >> And well, their value for testing is also questionable. They only can > help you to identify where it goes *boom* and crashes the VM, but they do > not actually assert for anything. > >> > >> Also not sure what the value of Slopstone and Smopstone (names from the > top of my head might be slightly different) is nowadays. > >> > >> The compiler benchmark is a reasonable application benchmark. > >> Would be good to have a few others in that collection, too. > >> > >> Best regards > >> Stefan > >> > >> > >> > >> > >> -- > >> Stefan Marr > >> Software Languages Lab > >> Vrije Universiteit Brussel > >> Pleinlaan 2 / B-1050 Brussels / Belgium > >> http://soft.vub.ac.be/~smarr > >> Phone: +32 2 629 2974 > >> Fax: +32 2 629 3525 > >> > >> > >> > > >
