On Tue, Sep 1, 2015 at 12:15 PM, Ian Romanick <i...@freedesktop.org> wrote: > For a bunch of the small changes, I don't care too much what the > difference is. I just want to know whether after is better than before.
And that gets back to my comment that you can't *measure* the impact of a change. Not with something where the outcome is a random variable. It can't be done. All you can do is answer the question "is X's mean more than N higher than Y's mean". And you change the number of trials in an experiment depending on N. (There's also more advanced concepts like 'power' and whatnot, I've done just fine without fully understanding them, I suspect you can too.) As an aside, increasing the number of trials until you get a significant result is a great way to arrive at incorrect decisions, due to the multi-look problem (95% CI means 1/20 gives you bad results). The proper way is to decide beforehand "I care about changes >0.1%, which means I need to run 5000 trial runs" (based on the assumption that 50 runs gets you 1%). After doing the 5k runs, your CI width should be ~0.1% and you should then be able to see if the delta in means is higher or lower than that. If it's higher, then you've detected a significant change. If it's not, that btw doesn't mean "no change", just not statistically significant. There's also a procedure for the null hypothesis (i.e. is a change's impact <1%) which is basically the same thing but involves doing a few more runs (like 50% more? I forget the details). Anyways, I'm sure I've bored everyone to death with these pedantic explanations, but IME statistics is one of the most misunderstood areas of math, especially among us engineers. -ilia _______________________________________________ mesa-dev mailing list mesa-dev@lists.freedesktop.org http://lists.freedesktop.org/mailman/listinfo/mesa-dev
