> better than another with 95% of confidence. However, my adviser is asking > me about not only the STATISTICAL significance of the results, but also the > PRACTICAL significance of them. I mean, if one system is, for example only > 1% better than another, with 99% of confidence, the result would have a > statistical significance, but wouldn't really matter in a practical sense.
It all comes down to defining "practical significance", which is subjective; but still a useful line of thought. I started thinking a definition could be: in any *short* series of games (e.g. best of 5) you'd be confident it would come out ahead. That would be my definition of clear superiority. 1% better isn't enough, as you need hundreds of games to be sure of seeing a difference. When you said a "23% difference" did you mean the win-ratio is 61.5:38.5? If so, what is the probability the stronger player will get 3+ wins from 5 games? If over 0.9 I think you can make the case that the strength difference is of practical significance. Or, yes, do it all in ELO ratings, and decide on how many ELO points feels significant. Darren -- Darren Cook, Software Researcher/Developer http://dcook.org/work/ (About me and my work) http://dcook.org/blogs.html (My blogs and articles) _______________________________________________ Computer-go mailing list [email protected] http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
