On 2011-04-05, at 15:11, Igor Stasenko wrote: > I don't care :) > > What i found that measurements confirms my previous expectation: > the 'slowdown' of removing #class from set of optimized selectors lies > in the range of macro-benchmark deviation. > > And running all tests for 100 times is not fun (it could take half of > the day), and i'm not ready to leave my computer for half of the day > and simply wait till it finish, because if i will do something > else,then it will affect the performance, and measurements will be > even more inaccurate.
thats why you run it over night :) (or the day, depending on your sleep rhythm). or you reduce the number of runs to something at least bigger than 1 :D. > On 5 April 2011 14:53, Camillo Bruni <[email protected]> wrote: >> This is exactly why you have to provide some confidence interval / >> deviation, otherwise it is hard to make any reasonable conclusion. >> >> run it 100 times and take the average and provide the standard deviation. >> >> I am not a big fan of relying on incomplete benchmarking results: >> >> Please read: http://portal.acm.org/citation.cfm?id=1297033 >> >> http://www.squeaksource.com/p.html provides a basic benchmarking framework >> under the NBenchmark package. You subclass from PBenchmarkSuite implement a >> method #benchXXX and run it. >> >> r := PBFloat run: 100. >> r asString >> >> which will give decent results back :). This way it is much easier to make >> sense out of the numbers. >> >> So here again to remember: >> >> - number of samples >> - average run times >> - standard deviation >> >> If one of these results is missing the benchmark results are incomplete. >> >> best regards, >> camillo >> >> >> >> On 2011-04-05, at 13:56, Igor Stasenko wrote: >> >>> VariableNode initialize. >>> Compiler recompileAll. >>> >>> [ >>> TestCase allSubclasses do: [ :cls| >>> cls isAbstract >>> ifFalse: [cls suite run]]. >>> ] timeToRun >>> >>> 178938 >>> 183963 >>> >>> >>> >>> (ParseNode classVarNamed: 'StdSelectors') removeKey: #class ifAbsent: []. >>> Compiler recompileAll. >>> >>> [ >>> TestCase allSubclasses do: [ :cls| >>> cls isAbstract >>> ifFalse: [cls suite run]]. >>> ] timeToRun >>> >>> 187168 >>> 184992 >>> >>> the deviation is too big to see if its really so big overhead. >>> >>> if you compare worst , you got 187/178 ~ 5% >>> and if you compare the best you got >>> 184/183 ~ 0.5% >>> >>> >>> -- >>> Best regards, >>> Igor Stasenko AKA sig. >>> >> >> >> > > > > -- > Best regards, > Igor Stasenko AKA sig. >
