I am maybe an ignorant here writing like that. I like to see a nice set of
benchmarks popping up in Pharo.
But did you have a look at my benchmark implementation we use in Pinocchio?
Its in the PBenchmark package of the Pinocchio project:
MCHttpRepository
location: 'http://www.squeaksource.com/p'
user: ''
password: ''
Its a fairly straight forward implementation based on UnitTests...
I see several issues in your benchmark implementation which I think are solved
much cleaner with my approach:
- currently there is one single class with tons of benchmarks in it
- no statistically valid output ( just the average doesn't mean anything! see
http://dx.doi.org/10.1145/1297105.1297033 for the basic scientific backgrounds)
- you interleave model (benchmarks and results) and view (transcript output)
which is really evil, there is no way you can ever use this on the command line!
so what I suggest, is that you have a look at my implementation and see how we
can improve the current situation.
m(^_-)m
camillo
On 2011-03-11, at 15:05, Alexandre Bergel wrote:
> I am not sure what you mean with vm benchmarks, but in almost all the tools I
> am contributing come with some benchmarks (spy, Mondrian, I wrote some
> benchmarks for Glamour as well). Naturally, those are macro benchmarks, which
> is probably what matter the most.
>
> Cheers,
> Alexandre
>
>
>
> Le 11 mars 2011 à 10:55, Mariano Martinez Peck <[email protected]> a
> écrit :
>
>> does anyone have news regarding this topic? we VERY welcome people helping
>> with benchmarks.
>> please feel free to improve http://www.squeaksource.com/PharoBenchmarks
>>
>>
>>
>> On Tue, Jan 4, 2011 at 11:51 PM, Stefan Marr <[email protected]> wrote:
>> Hi:
>>
>> On 04 Jan 2011, at 23:40, Mariano Martinez Peck wrote:
>>
>>> On Tue, Jan 4, 2011 at 11:35 PM, Stefan Marr <[email protected]> wrote:
>>> Hi Igor:
>>>
>>> On 04 Jan 2011, at 22:40, Igor Stasenko wrote:
>>>
>>>> Okay, how about creating a separate
>>>> VMBenchmarks repository
>>>> and putting VMBenchmarks package there?
>>>
>>> Sure, sounds good. There are also the Systems benchmarks at
>>> http://www.squeaksource.com/PharoBenchmarks.
>>>
>>>
>>> So, at least, I was able to run all benchmarks :)
>>>
>>> However, it still needs cleaning, improvement, testing, blah. But it is a
>>> good start point I think.
>>
>> Well, I am not to sure about the general value of those benchmarks.
>> There are many microbenchmarks which do not tell you a lot. All those test*
>> things.
>> And well, their value for testing is also questionable. They only can help
>> you to identify where it goes *boom* and crashes the VM, but they do not
>> actually assert for anything.
>>
>> Also not sure what the value of Slopstone and Smopstone (names from the top
>> of my head might be slightly different) is nowadays.
>>
>> The compiler benchmark is a reasonable application benchmark.
>> Would be good to have a few others in that collection, too.
>>
>> Best regards
>> Stefan
>>
>>
>>
>>
>> --
>> Stefan Marr
>> Software Languages Lab
>> Vrije Universiteit Brussel
>> Pleinlaan 2 / B-1050 Brussels / Belgium
>> http://soft.vub.ac.be/~smarr
>> Phone: +32 2 629 2974
>> Fax: +32 2 629 3525
>>
>>
>>
