One general comment: Ratings are not transitive. For instance, A1 may score 25 % against B, and A2 may score 22 % against B. Then it can not be concluded that A1 will score more than 50 % in direct duel with A2.
It is rather easy it construct triples of "semi-simple" agents A, B, C for some "normal" game where A score 95+ percent against B, B scores 95+ percent against C, C scores 95+ percent against A. Ingo. -------- Original-Nachricht -------- > Datum: Sun, 25 Nov 2012 17:03:33 -0800 > Von: Leandro Marcolino <[email protected]> > An: [email protected] > Betreff: [Computer-go] Practical significance? > Hello all!.. > > I am currently doing a research about Computer Go. I can't tell the > details > about it yet, but I will post them here after (if) my paper is accepted... > > In my research I compare many systems (An), playing against a fixed strong > adversary (B). So A1 would have a percentage of victory x1 against B, > while > A2 would have a percentage of victory x2, etc... Then I compare the > percentage of victories, and for most cases I can show that one system is > better than another with 95% of confidence. However, my adviser is asking > me about not only the STATISTICAL significance of the results, but also > the > PRACTICAL significance of them. I mean, if one system is, for example only > 1% better than another, with 99% of confidence, the result would have a > statistical significance, but wouldn't really matter in a practical sense. > > In my case, the difference between the systems can range from about 4% to > about 23%. Doesn't seem to be enough to argue that one system would be > one-handicap stone better than another. But what would be the minimum > difference for me to argue that one system is significantly better than > another, in a practical sense? (or they are not, in the end?..) Would > calculating ELO-ratings help me in answering this question? > > I think it gets even more complex if we think that, let's say, changing > the > percentage of victory from 95% to 100% seems to be much more significant > (in a practical sense) than changing from 30% to 35%, even though the > difference between the two systems is still only 5%. In my case, I am > dealing with percentages of victories that range from around 30% to > around > 53%. > > What do you guys think?.. > > Thanks for your help!.. > > Regards, > Leandro _______________________________________________ Computer-go mailing list [email protected] http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
