My favorite part:
"One natural idea is to use the learned simulation policy in Monte-Carlo search, and
generate new deep search values, in an iterative cycle."
But one thing confuses me: You are using the value from Fuego's 10k simulations as an approximation of the actual value of the position. But isn't the actual
value of the position either a win or a loss? On such small boards, can't you assume that Fuego is able to correctly determin who is winning and round it's
evaluation to the nearest win/loss? i.e. if it evaluates the position to 0.674, that gets rounded to 1. If such an assumption about Fuego's ability to read
the position on a small board is valid, then it should improve the results of your balanced simulation strategy, right? Or am I missing something?
David Silver wrote:
Hi everyone,
Please find attached my ICML paper with Gerry Tesauro on automatically
learning a simulation policy for Monte-Carlo Go. Our preliminary results
show a 200+ Elo improvement over previous approaches, although our
experiments were restricted to simple Monte-Carlo search with no tree on
small boards.
Abstract
In this paper we introduce the first algorithms for efficiently learning
a simulation policy for Monte-Carlo search. Our main idea is to optimise
the balance of a simulation policy, so that an accurate spread of
simulation outcomes is maintained, rather than optimising the direct
strength of the simulation policy. We develop two algorithms for
balancing a simulation policy by gradient descent. The first algorithm
optimises the balance of complete simulations, using a policy gradient
algorithm; whereas the second algorithm optimises the balance over every
two steps of simulation. We compare our algorithms to reinforcement
learning and supervised learning algorithms for maximising the strength
of the simulation policy. We test each algorithm in the domain of 5x5
and 6x6 Computer Go, using a softmax policy that is parameterised by
weights for a hundred simple patterns. When used in a simple Monte-Carlo
search, the policies learnt by simulation balancing achieved
significantly better performance, with half the mean squared error of a
uniform random policy, and equal overall performance to a sophisticated
Go engine.
-Dave
------------------------------------------------------------------------
_______________________________________________
computer-go mailing list
[email protected]
http://www.computer-go.org/mailman/listinfo/computer-go/
_______________________________________________
computer-go mailing list
[email protected]
http://www.computer-go.org/mailman/listinfo/computer-go/
