This is exactly why you have to provide some confidence interval / deviation, otherwise it is hard to make any reasonable conclusion.
run it 100 times and take the average and provide the standard deviation. I am not a big fan of relying on incomplete benchmarking results: Please read: http://portal.acm.org/citation.cfm?id=1297033 http://www.squeaksource.com/p.html provides a basic benchmarking framework under the NBenchmark package. You subclass from PBenchmarkSuite implement a method #benchXXX and run it. r := PBFloat run: 100. r asString which will give decent results back :). This way it is much easier to make sense out of the numbers. So here again to remember: - number of samples - average run times - standard deviation If one of these results is missing the benchmark results are incomplete. best regards, camillo On 2011-04-05, at 13:56, Igor Stasenko wrote: > VariableNode initialize. > Compiler recompileAll. > > [ > TestCase allSubclasses do: [ :cls| > cls isAbstract > ifFalse: [cls suite run]]. > ] timeToRun > > 178938 > 183963 > > > > (ParseNode classVarNamed: 'StdSelectors') removeKey: #class ifAbsent: []. > Compiler recompileAll. > > [ > TestCase allSubclasses do: [ :cls| > cls isAbstract > ifFalse: [cls suite run]]. > ] timeToRun > > 187168 > 184992 > > the deviation is too big to see if its really so big overhead. > > if you compare worst , you got 187/178 ~ 5% > and if you compare the best you got > 184/183 ~ 0.5% > > > -- > Best regards, > Igor Stasenko AKA sig. >
