Bojun Huang: <[email protected]>: >Thanks for all these responses and discussions here! I'd like to try to gather >pieces in this >thread so far and to make the consensuses more clear to me (and others >hopefully). > >Consensus 1: given the searching/playout policies, more playouts will >GENERALLY contribute to >a stronger program. > >Here the term "generally" means that the observations are from statistics (or >the sense we >got empirically) upon large number of games between various GO bots, rather >than refering to >any specific cases/situations. So an example-based arguement is not valid here. > >However, two questions for this consensus: > >(1.a) Inhttp://cgos.boardspace.net/study/, the ELO improvement from doubling >the number of >playouts seems converge to 3000 score. Is this because versions with 3000+ ELO >is "too >strong" to be measured accurately? More importantly, seems these results are >self-playing >game records, which is known to have amplification effect on the ELO score >improvement.
See my old post on this list and attached graphs (Message-Id: <4aecf4dd.166%[email protected]>) on Date: Sun, 01 Nov 2009 11:39:32 +0900. A reason of the convergence is the "systematic bias." For example, if a program doesn't know "nakade" (like MoGo in the study), his performance converges at certain level. When a program can't manage particular positions correctly, the same thing will happen. >(1.b) Generally speaking, we always want the progam to "show its best" in an >important >tournament like Olympaid, and assuming the consensus, more playouts will >always helps this. >As a consequence, there should be no serious player for the Olympaid who run >their program on >laptop (unless they are fully confident that the capability of their program >shown in laptop >is *already* strong enough to beat all others!). Note that "the authors prefer >the >algorithm-part improvement" or "software algorithm just matters much more" do >not answer this >question, since no matter what algorithmic improvement here, we just assume >that more >playouts give orthogonal contributions, right? NB: Classical programs such as Katsunari and GNU Go perform the same on laptops at the longer time-settings used for the Olympiads. >Consensus 2: there is a trade-off between the quality of playout and the speed >of playout. > >it's very interesting for me to find that consensus 2 actually has little >necessary relation >with consensus 1. First, the speed of playout translates to the number of >playout only under >fixed computation resources. So even the playout becomes slow due to a more >complex playout >policy, we can achieve the same number of playouts by simply using more CPU >cores. Second, >even though we do have the "quality vs number" trade-off given fixed CPU >cores, this does not >conflict with the fact that "number" brings additional contributions for given >"quality". > >But seems all strong programs just favor the "quality" in this trade-off, >because more >playouts tend to make the same errors with previous playouts so they can >hardly recover these >errors in reasonable time. On the other hand, everyone admit that the random >playout is >clearly out of the sweet spot area of this trade-off. I believe there are no trade-off now. Quality is the most important issue for developing strong programs. If the simulations of a program can manage _almost all_ positions correctly, then I mind the trade-off. But this never happen in a few tens years, I guess. Hideki >Finally, I'm trying to find a "lower bound quality" for any meaningful >trade-off, but there >seems no convincing answer. > >Bojun > > >---- inline file >_______________________________________________ >Computer-go mailing list >[email protected] >http://dvandva.org/cgi-bin/mailman/listinfo/computer-go -- Hideki Kato <mailto:[email protected]> _______________________________________________ Computer-go mailing list [email protected] http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
