Heikki Levanto wrote: > On Mon, Dec 10, 2007 at 04:08:48PM -0500, Don Dailey wrote: >> Would you rather be 95% confident of a win or 90% confident? There is >> only 1 correct answer to that question. >> Yes, if you can offer me reliable confidence numbers. We all (should) know
> that MC evaluations suffer from systematic problems that can not just be > averaged away statistically. > > Compare these two positions: > > playout_benchmark 10000 > = Initial board: > komi 7.5 > A B C D E F G H J > 9 . . . . . O O O O 9 > 8 O O O O O O O O O 8 > 7 O O O O O O O O O 7 > 6 O O O O O O O O O 6 > 5 # # # # # # # # # 5 > 4 O O O # # # # # # 4 > 3 O O O O . # # # # 3 > 2 . O O O . # # # . 2 > 1 # . O O . # # . # 1 > A B C D E F G H J > Performance: > 10000 playouts > 0.032002 seconds > 312.481 kpps > Black wins = 1937 > White wins = 8063 > P(black win) = 0.1937 > > > playout_benchmark 10000 > = Initial board: > komi 7.5 > A B C D E F G H J > 9 . # . . . O O O O 9 > 8 O O O O O O O O O 8 > 7 O O O O O O O O O 7 > 6 O O O O O O # # # 6 > 5 # # # # # # # # # 5 > 4 O O O # # # # # # 4 > 3 O O O O . # # # # 3 > 2 . O O O . # # # . 2 > 1 . . O O . # # . # 1 > A B C D E F G H J > Performance: > 10000 playouts > 0.084006 seconds > 119.039 kpps > Black wins = 7746 > White wins = 2254 > P(black win) = 0.7746 > > > Which one is better, 77% or 19%?
This reminds me of the first testcase I wrote when I started with
MonteGNU. Black to play, no komi.
A B C D E F G H J
9 . . O O X . X . X 9
8 . . . O X . X O X 8
7 O . O O X X O O X 7
6 O O O . X . X O O 6
5 X X X X X O O O . 5
4 . . X . O O . O . 4
3 X X O X O . + O . 3
2 X X O X O . . O . 2
1 . O O O O . . . . 1
A B C D E F G H J
Naturally B has to play B8, or white plays there and wins big. This is
trivial to find for a classic program and easy enough for a Monte
Carlo program. What's interesting is that it takes some work to make
black think that it has better than even winning chances after B8. The
Monte Carlo code in GNU Go CVS version gets 0.079 with 10k, 0.387 with
100k, and 0.475 with 1M simulations. I suspect that stronger programs
tend to be more optimistic about winning chances here. So please fill
in this table if you have an MC program:
10k 100k 1M
--------------------------------
GNU Go CVS 0.079 0.387 0.475
The sgf file is attached, load it before the first move. The positions
before move 3 and 5 are also relevant tests.
/Gunnar
mc1.sgf
Description: application/go-sgf
_______________________________________________ computer-go mailing list [email protected] http://www.computer-go.org/mailman/listinfo/computer-go/
