Is Jrefgo the pure version that does not use tricks like the futures map? If you use things like that, all bets are off - I can't be sure this is not negatively scalable.
You cannot draw any reasonable conclusions by stopping after 10 moves and letting gnugo judge the game either. Why didn't you play complete games? - Don On Mon, 2008-12-15 at 17:30 -0500, Weston Markham wrote: > Hi. This is a continuation of a month-old conversation about the > possibility that the quality of AMAF Monte Carlo can degrade, as the > number of simulations increases: > > Me: "running 10k playouts can be significantly worse than running 5k > playouts." > > On Tue, Nov 18, 2008 at 2:27 PM, Don Dailey <drdai...@cox.net> wrote: > > On Tue, 2008-11-18 at 14:17 -0500, Weston Markham wrote: > >> On Tue, Nov 18, 2008 at 12:02 PM, Michael Williams > >> <michaelwilliam...@gmail.com> wrote: > >> > It doesn't make any sense to me from a theoretical perspective. Do you > >> > have > >> > empirical evidence? > >> > >> I used to have data on this, from a program that I think was very > >> nearly identical to Don's reference spec. When I get a chance, I'll > >> try to reproduce it. > > > > Unless the difference is large, you will have to run thousands of games > > to back this up. > > > > - Don > > I am comparing the behavior of the AMAF reference bot with 5000 > playouts against the behavior with 100000 playouts, and I am only > considering the first ten moves (five from each player) of the (9x9) > games. I downloaded a copy of Don's reference bot, as well as a copy > of Mogo, which is used as an opponent for each of the two settings. > gnugo version 3.7.11 is also used, in order to judge which side won > (jrefgo or mogo) after each individual match. gnugo was used because > it is simple to set it up for this sort of thing via command-line > options, and it seems plausible that it should give a somewhat > realistic assessment of the situation. > > jrefgo always plays black, and Mogo plays white. Komi is set to 0.5, > so that jrefgo has a reasonable number of winning lines available to > it, although the general superiority of Mogo means that egregiously > bad individual moves will be punished. > > In the games played, Mogo would occasionally crash. (This was run > under Windows Vista; perhaps there is some incompatibility of the > binary I downloaded) I have discarded these games (about 1 out of 50, > I think) from the statistics gathered. As far as I know, there would > be no reason to think that this would skew the comparison between 5k > playouts and 100k playouts. Other than occasional crashes, the > behavior of Mogo seemed reasonable in other games that I observed. I > have no reason to think that it was not playing at a relatively high > level in the retained results. > > Out of 3637 matches using 5k playouts, jrefgo won (i.e., was ahead > after 10 moves, as estimated by gnugo) 1688 of them. (46.4%) > Out of 2949 matches using 100k playouts, jrefgo won 785. (26.6%) > > It appears clear to me that increasing the number of playouts from 5k > to 100k certainly degrades the performance of jrefgo. Below, I am > including the commands that I used to run the tests and tally the > results. > > Weston > > > $ cat scratch5k.sh > > ../gogui-1.1.3/bin/gogui-twogtp -auto -black "\"C:\\\\Program > Files\\\\Java\\\\j > dk1.6.0_06\\\\bin\\\\java.exe\" -jar jrefgo.jar 5000" -games 10000 -komi 0.5 > -ma > xmoves 10 -referee "gnugo --mode gtp --score aftermath --chinese-rules > --positio > nal-superko" -sgffile games/jr5k-v-mogo -size 9 -white > C:\\\\cygwin\\\\home\\\\E > xperience\\\\projects\\\\go\\\\MoGo_release3\\\\mogo.exe > > > $ grep B+ games/jr5k-v-mogo.dat | grep -v unexp | wc -l > 1688 > > $ grep W+ games/jr5k-v-mogo.dat | grep -v unexp | wc -l > 1949 > > $ grep B+ games/jr100k-v-mogo.dat | grep -v unexp | wc -l > 785 > > $ grep W+ games/jr100k-v-mogo.dat | grep -v unexp | wc -l > 2164 > _______________________________________________ > computer-go mailing list > computer-go@computer-go.org > http://www.computer-go.org/mailman/listinfo/computer-go/
signature.asc
Description: This is a digitally signed message part
_______________________________________________ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
