About a year ago I tried collecting full histograms to see if I could extract useful information from them. On small boards the histograms tend to show multimodal distributions (killing the whole board leads to peaks at the two extreme scores). But I could not think of a good use for this information. I mean, how do you compare two multimodal distributions (instead of comparing only one statistic like the mean, the winrate, the median or the mode)? One could calculate skewness (or other moments), but when comparing distibutions with the same mean, should one prefer positive skewness or negative skewness? Using the median also seemed troublesome, because its value progression while adding playouts is not smooth. It changes in a steplike fashion: Adding a win or loss either either has no effect on the median (most of the time) or it lets the median jump from one value to the next (occasionally). Dave de Vos
________________________________ Van: [email protected] namens Lukasz Lew Verzonden: di 5-10-2010 18:57 Aan: [email protected] Onderwerp: Re: [Computer-go] Results from 19x19 Valkyria/1k H9 vs Valkyria/10k On Tue, Oct 5, 2010 at 18:40, <[email protected]> wrote: > No information is thrown away with maximizing win rate. That is not true. :) If you look for robustness median and quantile statistics are a good choice. But that is not necessary because playout results almost follwow Bernoulli distribution. Just look at the histogram. I say almost because it is not a sum of iid variables. Result is a sum of random components of not equall sizes. Also there is artefact of killing a whole board. (especially on 9x9) > > :-) > > This is because in go it is only the sign of the final board count that > matter. > > Yes I know I am very stubborn and narrow minded on this issue. > > It is a weird thing to fear throwing away "information" and then estimate > the expected score which will throw away the actual distributions of > outcomes. And so will any other statistical measure do. > > Also I do measure the score in at least two ways in Valkyria. One is simply > taking the average of score at the root and the other is to estimate > territory directly by looking at black/white membership of individual points > of the board. The latter is much more stable and the average is often off > several points also late in the endgame. > > It could also be that different programs have very different kind of > playouts. I know that the playouts of Valkyria contains really weird stuff. > Sometimes black wins a playout with +100 points because a lot of perfectly > safe white groups died because of some really unlikely combination of bugs, > omissions and randomness in the playout. Such a score does not give me > information with any value. Killing several groups due to bugs etc is not an issue since killing them is independent. > > I do agree that maybe expected score could be useful in the opening, because > the opening has no systematic bias yet. In the endgame however almost every > group will have a small probability of flipping state from alive to dead or > dead or alive, which has a little to do with actually theoretical score of > perfect play. But these probabilities will add up to something that on > average close to the true value but mostly it will be very wrong. > > So in short, I think win rates is the most robust thing to evaluate > positions using MC playouts. And someone else have to prove me wrong! I wont > do it. (But if someone do prove me wrong I will of course steal the idea and > implement it). > > -Magnus > > > > > > Quoting Lukasz Lew <[email protected]>: > >> On Tue, Oct 5, 2010 at 15:34, <[email protected]> wrote: >>> >>> Quoting Lukasz Lew <[email protected]>: >>> >>>> Your dynamic komi results are very convincing.. >>>> But shouldn't we just concentrate on maximizing score instead of >>>> winning rate in the beginning of the game? >>> >>> Maximizing winning rate means that the probability of having a score > 0 >>> at >>> the end of the game is maximum. >> >> I want to maximize expected score of a playout. >> I think that if we set komi so that around half of the playouts have >> score > 0 >> AND if the noise is large (the game is in the beginning) then >> maximixing winning rate >> is almost the same as maximizing score. >> >> The drawbacks of maximizing winning rate are: >> - we need to adhoc adjust komi to tell engine to concentrate on >> maximizing score >> - we throw away information. >> >> Of course maximizing winning rate is the right way in low-noise >> conditions (endgame). >> >>> >>> Exactly what do you mean with "maximizing score in the beginning of the >>> game"? >>> It is hard to estimate the score. Also in a game of go territory is not >>> everything. Aji and influence is also important. Win rate is as far as i >>> know the best way of capturing all these things in one measure that >>> guides >>> search. >>> >>> Magnus >>> >>> >>> _______________________________________________ >>> Computer-go mailing list >>> [email protected] >>> http://dvandva.org/cgi-bin/mailman/listinfo/computer-go >> >> >> >> -- >> Lukasz >> _______________________________________________ >> Computer-go mailing list >> [email protected] >> http://dvandva.org/cgi-bin/mailman/listinfo/computer-go > > > _______________________________________________ > Computer-go mailing list > [email protected] > http://dvandva.org/cgi-bin/mailman/listinfo/computer-go -- Lukasz _______________________________________________ Computer-go mailing list [email protected] http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
_______________________________________________ Computer-go mailing list [email protected] http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
