Sorry, I have just figured out that my loadsgf command did not set the komi correctly. Now that it is fixed, the result is much better: http://www.grappa.univ-lille3.fr/~coulom/seki-1k.html http://www.grappa.univ-lille3.fr/~coulom/seki-2k.html http://www.grappa.univ-lille3.fr/~coulom/seki-4k.html http://www.grappa.univ-lille3.fr/~coulom/seki-8k.html http://www.grappa.univ-lille3.fr/~coulom/seki-16k.html http://www.grappa.univ-lille3.fr/~coulom/seki-32k.html http://www.grappa.univ-lille3.fr/~coulom/seki-64k.html http://www.grappa.univ-lille3.fr/~coulom/seki-128k.html http://www.grappa.univ-lille3.fr/~coulom/seki-1024k.html
So only 3 errors at slow time control. I can't tell for sure they are really errors. The evaluation for ID 111 & 115 is losing. The evaluation for ID 191 is jigo. All the other (the correct ones) are either jigo or winning, except for case17 (ID 172), that is a very funny kind of seki, and that is evaluated as bad for W, although CS plays the game correctly. Rémi On 17 mai 2012, at 13:41, Rémi Coulom wrote: > Hi Aja, > > Thanks for this interesting test. This is Crazy Stone's output for seki_moves: > http://www.grappa.univ-lille3.fr/~coulom/seki-128k.html > Many correct answers are probably a bit lucky, because the evaluation is > rarely correct. > > It is not easy for me to implement sg_compare_float. But I'll try if I find > time. > > gnugo rules :-) > > Rémi > > On 15 mai 2012, at 04:39, Aja Huang wrote: > >> Dear all, >> >> Martin Mueller and I are writing a paper about exploring some limitations of >> current MCTS programs in Go. For this purpose we have carefully designed a >> regression test set which consists of 20 seki and 15 two-safe-groups cases >> on 9x9 board. If you are interested, it is available at >> >> http://webdocs.cs.ualberta.ca/~mmueller/ps/seki-and-two-safe-groups-regression-test.zip >> >> We will appreciate if you would like to run your program over our regression >> test and send us the results for our publication. >> >> It's easy to run your program through these positions (.sgf). Mainly, the >> script run.sh under /utility is able to run a given program for a given >> regression test file (.tst) and produce the result in a related html file. >> For example, for the seki test you can simply type >> >> ./run.sh -p PATH_TO_PROGRAM -t g_seki_moves.tst >> >> Some notes: >> 1. Your program must support the command sg_compare_float for the >> two-safe-groups test. If it doesn't support reg_genmove then the test file >> g_seki_moves.tst is good to use which performs genmove instead. >> >> 2. On Windows platform, you will be able to execute 'run.sh' directly at the >> command prompt after cygwin is installed. >> >> 3. If your program doesn't support the GTP command 'loadsgf', gogui-adapter >> is able to translate 'loadsgf' into a sequence of 'play' commands. The file >> gogui-adapter.jar under /utility is good to use because Markus has fixed >> some bugs for us, see >> >> https://sourceforge.net/tracker/?func=detail&aid=3522401&group_id=59117&atid=489964 >> https://sourceforge.net/tracker/?func=detail&aid=3519829&group_id=59117&atid=489964 >> >> Under /experimental results, there are results of several programs such as >> Fuego (tilburg version), pachi, ManyFaces and GnuGo. We thank David for >> providing us the valuable results of ManyFaces. The test set is really not >> easy because these programs all failed in many cases. >> >> Questions are very welcome. If you find any error in the test set please >> inform us. Thanks. >> >> Best regards, >> Aja >> _______________________________________________ >> Computer-go mailing list >> Computer-go@dvandva.org >> http://dvandva.org/cgi-bin/mailman/listinfo/computer-go > > _______________________________________________ > Computer-go mailing list > Computer-go@dvandva.org > http://dvandva.org/cgi-bin/mailman/listinfo/computer-go _______________________________________________ Computer-go mailing list Computer-go@dvandva.org http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
