On 5 April 2011 16:31, Stefan Marr <[email protected]> wrote: > > On 05 Apr 2011, at 15:11, Igor Stasenko wrote: > >> I don't care :) > *sigh* then just flip a coin and you have the same amount of insight with > less effort. > > It is always astonishing how few people know about the basics when it comes > to empirical experiments. > > I disagree with Camillo's demand for 100x repetition, but it should be > definitely run at least a few times. And then you have to look at the results > and see how they vary. Otherwise you don't know anything and are just wasting > your own time. >
that's what i did. i run two times and found that slowdown is at scale of deviation. This is why i don't care about exact slowdown anymore, because it is at the level of noise on macro benchmarks. > I will try at the next sprint to make some progress on the framework front. > Perhaps, someone could trow together something like the unit-test runner to > make at least the basics fool-proof. > > And at the same time the message to measure the execution time of a block > should become deprecated or raise a warning that you are about to fool > yourself... > > It does not have to be statistically rigorous for small experiments, but at > the very least you have to convince yourself that your measurements are > actually of any value. > The computer in front of you is just to complex to make an educated guess. > > Best regards > Stefan > > -- > Stefan Marr > Software Languages Lab > Vrije Universiteit Brussel > Pleinlaan 2 / B-1050 Brussels / Belgium > http://soft.vub.ac.be/~smarr > Phone: +32 2 629 2974 > Fax: +32 2 629 3525 > > > -- Best regards, Igor Stasenko AKA sig.
