Better than average, take the median Nicolas
2011/4/5 Camillo Bruni <[email protected]>: > This is exactly why you have to provide some confidence interval / deviation, > otherwise it is hard to make any reasonable conclusion. > > run it 100 times and take the average and provide the standard deviation. > > I am not a big fan of relying on incomplete benchmarking results: > > Please read: http://portal.acm.org/citation.cfm?id=1297033 > > http://www.squeaksource.com/p.html provides a basic benchmarking framework > under the NBenchmark package. You subclass from PBenchmarkSuite implement a > method #benchXXX and run it. > > r := PBFloat run: 100. > r asString > > which will give decent results back :). This way it is much easier to make > sense out of the numbers. > > So here again to remember: > > - number of samples > - average run times > - standard deviation > > If one of these results is missing the benchmark results are incomplete. > > best regards, > camillo > > > > On 2011-04-05, at 13:56, Igor Stasenko wrote: > >> VariableNode initialize. >> Compiler recompileAll. >> >> [ >> TestCase allSubclasses do: [ :cls| >> cls isAbstract >> ifFalse: [cls suite run]]. >> ] timeToRun >> >> 178938 >> 183963 >> >> >> >> (ParseNode classVarNamed: 'StdSelectors') removeKey: #class ifAbsent: []. >> Compiler recompileAll. >> >> [ >> TestCase allSubclasses do: [ :cls| >> cls isAbstract >> ifFalse: [cls suite run]]. >> ] timeToRun >> >> 187168 >> 184992 >> >> the deviation is too big to see if its really so big overhead. >> >> if you compare worst , you got 187/178 ~ 5% >> and if you compare the best you got >> 184/183 ~ 0.5% >> >> >> -- >> Best regards, >> Igor Stasenko AKA sig. >> > > >
