Re: [computer-go] scalability analysis with pachi
Oh, ok. I was a bit surprised. Last time I checked my program scaled quite nicely against GnuGo, at least for low numbers of simulations up to about 97% winning rate. I suppose there could be some kind of plateau when nearing 100% due to some missing knowledge/skills that only GnuGo has. Erik 2010/1/16 dhillism...@netscape.net: Well, I thought there seems to be a picture emerging was sufficiently hedged that it would be construed as a conjecture, not a conclusion. :) I am thinking, in particular, of the scalability studies with Zen that Hideki reported to this list in Oct. 2009. BTW, recently I've measured the strength (win rate) vs time for a move curves with Zen vs GNU Go and Zen vs Zen (self-play) on 19 x 19 board. Without opening book, it saturates between +400 and +500 Elo against GNU but doesn't upto +800 Elo in self-play. That's somewhat interesting (detail will be open soon at GPW-2009). Hideki There was a bit more information provided in a sequence of posts to this list during that month. I wonder if the paper is out now. - Dave Hillis -Original Message- From: Erik van der Werf erikvanderw...@gmail.com To: computer-go computer-go@computer-go.org Sent: Sat, Jan 16, 2010 12:55 pm Subject: Re: [computer-go] scalability analysis with pachi 2010/1/15 dhillism...@netscape.net Thank you for posting these interesting results There seems to be a picture emerging that MCTS engines scale very well in self play, and apparently against other MCTS engines, but not so well against the non-MCTS version of Gnugo. - Dave Hillis Do you have any data to back that conclusion? Erik ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] scalability analysis with pachi
Yes. And while worrying about what happens after a win rate of 97% sounds like splitting hairs, I think we're talking about an awkward way of measuring something that's of practical interest. Suppose Hideki's experiment were repeated, giving GNU GO a four stone handicap. If Zen plateau'd out at the same time-per-move, that would suggest a real limit in strength improvement against dissimilar opponents. If Zen plateau'd out at the same ELO point, that might suggest that a few percent of the games involved tactical situations that GNU GO could understand but Zen couldn't, even with much more time. The second possibility could be tested more efficiently by Zen taking the handicap. - Dave Hillis -Original Message- From: Erik van der Werf erikvanderw...@gmail.com To: computer-go computer-go@computer-go.org Sent: Sun, Jan 17, 2010 8:06 am Subject: Re: [computer-go] scalability analysis with pachi Oh, ok. I was a bit surprised. Last time I checked my program scaled uite nicely against GnuGo, at least for low numbers of simulations up o about 97% winning rate. I suppose there could be some kind of lateau when nearing 100% due to some missing knowledge/skills that nly GnuGo has. Erik 010/1/16 dhillism...@netscape.net: Well, I thought there seems to be a picture emerging was sufficiently hedged that it would be construed as a conjecture, not a conclusion. :) I am thinking, in particular, of the scalability studies with Zen that Hideki reported to this list in Oct. 2009. BTW, recently I've measured the strength (win rate) vs time for a move curves with Zen vs GNU Go and Zen vs Zen (self-play) on 19 x 19 board. Without opening book, it saturates between +400 and +500 Elo against GNU but doesn't upto +800 Elo in self-play. That's somewhat interesting (detail will be open soon at GPW-2009). Hideki There was a bit more information provided in a sequence of posts to this list during that month. I wonder if the paper is out now. - Dave Hillis -Original Message- From: Erik van der Werf erikvanderw...@gmail.com To: computer-go computer-go@computer-go.org Sent: Sat, Jan 16, 2010 12:55 pm Subject: Re: [computer-go] scalability analysis with pachi 2010/1/15 dhillism...@netscape.net Thank you for posting these interesting results There seems to be a picture emerging that MCTS engines scale very well in self play, and apparently against other MCTS engines, but not so well against the non-MCTS version of Gnugo. - Dave Hillis Do you have any data to back that conclusion? Erik ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ __ omputer-go mailing list omputer...@computer-go.org ttp://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] scalability analysis with pachi
Yes. And while worrying about what happens after a win rate of 97% sounds like splitting hairs, I think we're talking about an awkward way of measuring something that's of practical interest. Yes. How can a program be strong enough to win 97% and not win 100%. Over on the fuego list Martin Mueller discovered some interesting test positions (e.g. only winning move is playing inside your own benson-safe region, IIRC) by looking at games where fuego lost against weaker opponents. An addition to CGOS that shows Biggest Upsets Of The Month, excluding losses on time, would be an interesting way to build up a test suite. I don't know if that is a trivial SQL query for Don, or something harder though. Darren ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] scalability analysis with pachi
2010/1/15 dhillism...@netscape.net Thank you for posting these interesting results There seems to be a picture emerging that MCTS engines scale very well in self play, and apparently against other MCTS engines, but not so well against the non-MCTS version of Gnugo. - Dave Hillis Do you have any data to back that conclusion? Erik ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] scalability analysis with pachi
Well, I thought there seems to be a picture emerging was sufficiently hedged that it would be construed as a conjecture, not a conclusion. :) I am thinking, in particular, of the scalability studies with Zen that Hideki reported to this list in Oct. 2009. BTW, recently I've measured the strength (win rate) vs time for a move curves with Zen vs GNU Go and Zen vs Zen (self-play) on 19 x 19 board. Without opening book, it saturates between +400 and +500 Elo against GNU but doesn't upto +800 Elo in self-play. That's somewhat interesting (detail will be open soon at GPW-2009). Hideki There was a bit more information provided in a sequence of posts to this list during that month. I wonder if the paper is out now. - Dave Hillis -Original Message- From: Erik van der Werf erikvanderw...@gmail.com To: computer-go computer-go@computer-go.org Sent: Sat, Jan 16, 2010 12:55 pm Subject: Re: [computer-go] scalability analysis with pachi 2010/1/15 dhillism...@netscape.net Thank you for posting these interesting results There seems to be a picture emerging that MCTS engines scale very well in self play, and apparently against other MCTS engines, but not so well against the non-MCTS version of Gnugo. - Dave Hillis Do you have any data to back that conclusion? Erik ___ omputer-go mailing list omputer...@computer-go.org ttp://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] scalability analysis with pachi
Thank you for posting these interesting results There seems to be apicture emerging that MCTS engines scale very well in self play, and apparently against other MCTS engines, but not so well against the non-MCTS version of Gnugo. - Dave Hillis -Original Message- From: Jean-loup Gailly jl...@gailly.net To: computer-go computer-go@computer-go.org Sent: Thu, Jan 14, 2010 6:53 pm Subject: [computer-go] scalability analysis with pachi I did some 19x19 scalability experiments with pachi, written by Petr Baudis. This was run on one machine using 15 out of 16 2.2 GHz cores, against Fuego using 100K playouts on 15 cores of another machine. The results are encouraging. Pachi's strength continues scaling linearly (in elo) with each doubling of the number of playouts: pachi2 is an instance of pachi running on kgs. It temporarily got a 2d rating on 19x19 but many dans are now trying to lower this :-( Note that Petr should be congratulated for this, I only did minor changes myself. Jean-loup ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
[computer-go] scalability analysis with pachi
I did some 19x19 scalability experiments with pachi, written by Petr Baudis. This was run on one machine using 15 out of 16 2.2 GHz cores, against Fuego using 100K playouts on 15 cores of another machine. The results are encouraging. Pachi's strength continues scaling linearly (in elo) with each doubling of the number of playouts: [image: ?ui=2view=attth=1262f3dd69e42c69attid=0.1disp=attdrealattid=ii_1262f3dd69e42c69zw] pachi2 is an instance of pachi running on kgs. It temporarily got a 2d rating on 19x19 but many dans are now trying to lower this :-( Note that Petr should be congratulated for this, I only did minor changes myself. Jean-loup pachi-scalability.png___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
