The issue is testing time. To see if a change makes a small improvement I play 1000 or more test games. To do a test in a day requires a large number of cores or very short games.
David From: [email protected] [mailto:[email protected]] On Behalf Of terry mcintyre Sent: Tuesday, January 18, 2011 6:43 PM To: [email protected] Subject: Re: [Computer-go] semeai example of winning rate Old saying: make it correct first, then make it fast. Blitz games may give fast but wrong results, penalizing a new patch for lack of speed. As a kibbitzer put it recently, referring to a program running on 64 nodes: "64 wrongs don't make a right." Terry McIntyre <[email protected]> Unix/Linux Systems Administration Taking time to do it right saves having to do it twice. _____ From: Darren Cook <[email protected]> To: [email protected] Sent: Tue, January 18, 2011 6:23:09 PM Subject: Re: [Computer-go] semeai example of winning rate >> The main problem right now is that a lot of "smart" code actually >> weakens the program. > > ... > we are here essentially trying to patch a bot weakness. The usual way to > test strength is by playing other bots. Since they also have the > weakness, and anyhow won't steer the game in that direction, patching > that weakness does not make a huge difference against them, and can > easily be overshadowed by any collateral change, such as being slightly > slower. On the other hand, it might still make them stronger against > humans. I fully agree; in fact I'm getting increasingly passionate on this issue :-) I feel some good ideas are getting thrown away (along with lots of bad ones) because they are initially only considered on blitz time settings, and only on self-play or against one opponent. I think Very Large Test Suites are the way forward; supported with play on go servers. Darren -- Darren Cook, Software Researcher/Developer http://dcook.org/work/ (About me and my work) http://dcook.org/blogs.html (My blogs and articles) _______________________________________________ Computer-go mailing list [email protected] http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
_______________________________________________ Computer-go mailing list [email protected] http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
