Depends at which signal/noise ratio you are ready to drop the signal... Nicolas
2011/4/5 Igor Stasenko <[email protected]>: > On 5 April 2011 16:31, Stefan Marr <[email protected]> wrote: >> >> On 05 Apr 2011, at 15:11, Igor Stasenko wrote: >> >>> I don't care :) >> *sigh* then just flip a coin and you have the same amount of insight with >> less effort. >> >> It is always astonishing how few people know about the basics when it comes >> to empirical experiments. >> >> I disagree with Camillo's demand for 100x repetition, but it should be >> definitely run at least a few times. And then you have to look at the >> results and see how they vary. Otherwise you don't know anything and are >> just wasting your own time. >> > > that's what i did. i run two times and found that slowdown is at scale > of deviation. This is why i don't care about exact slowdown > anymore, because it is at the level of noise on macro benchmarks. > >> I will try at the next sprint to make some progress on the framework front. >> Perhaps, someone could trow together something like the unit-test runner to >> make at least the basics fool-proof. >> >> And at the same time the message to measure the execution time of a block >> should become deprecated or raise a warning that you are about to fool >> yourself... >> >> It does not have to be statistically rigorous for small experiments, but at >> the very least you have to convince yourself that your measurements are >> actually of any value. >> The computer in front of you is just to complex to make an educated guess. >> >> Best regards >> Stefan >> >> -- >> Stefan Marr >> Software Languages Lab >> Vrije Universiteit Brussel >> Pleinlaan 2 / B-1050 Brussels / Belgium >> http://soft.vub.ac.be/~smarr >> Phone: +32 2 629 2974 >> Fax: +32 2 629 3525 >> >> >> > > > > -- > Best regards, > Igor Stasenko AKA sig. > >
