some minor nits: consensus 2 seems muddled. everyone would generally prefer to use as much CPU as they can lay their hands on, but a few things stop them: programming concurrency is hard, top-end algorithms solving real fixed-parameter problems don't just magically scale, things like heuristics might need to be modified, ram limits are reached without fancy trickery, etc. also, given that resources are finite, the thing that you *do* have the most control over is how your program can run on a single machine, or perhaps a small group of machines. so of course that's the focus. writing code for, say, 1000 machines is a totally different task with a totally different set of optimizations to think about.
consensus 1: i wouldn't get too worked up about a top-end ELO estimate. the top end is whatever it is. what matters is advancing the ELO of real programs (and i'd argue that this measurement is more accurate when a machine is compared against humans, since there is a much richer pool of playing styles to compare yourself with). s. 2011/6/5 Bojun Huang <[email protected]>: > Thanks for all these responses and discussions here! I'd like to try to > gather pieces in this thread so far and to make the consensuses more clear > to me (and others hopefully). > > Consensus 1: given the searching/playout policies, > more playouts will GENERALLY contribute to a stronger program. > > Here the term "generally" means that the observations are from statistics > (or the sense we got empirically) upon large number of games between various > GO bots, rather than refering to any specific cases/situations. So an > example-based arguement is not valid here. > > However, two questions for this consensus: > > (1.a) In http://cgos.boardspace.net/study/, the ELO improvement from > doubling the number of playouts seems converge to 3000 score. Is this > because versions with 3000+ ELO is "too strong" to be measured accurately? > More importantly, seems these results are self-playing game records, which > is known to have amplification effect on the ELO score improvement. > > (1.b) Generally speaking, we always want the progam to "show its best" in an > important tournament like Olympaid, and assuming the consensus, more > playouts will always helps this. As a consequence, there should be no > serious player for the Olympaid who run their program on laptop (unless they > are fully confident that the capability of their program shown in laptop is > *already* strong enough to beat all others!). Note that "the authors prefer > the algorithm-part improvement" or "software algorithm just matters much > more" do not answer this question, since no matter what algorithmic > improvement here, we just assume that more playouts give orthogonal > contributions, right? > > Consensus 2: there is a trade-off between the quality of playout and the > speed of playout. > > it's very interesting for me to find that consensus 2 actually has little > necessary relation with consensus 1. First, the speed of playout translates > to the number of playout only under fixed computation resources. So even the > playout becomes slow due to a more complex playout policy, we can achieve > the same number of playouts by simply using more CPU cores. Second, > even though we do have the "quality vs number" trade-off given fixed CPU > cores, this does not conflict with the fact that "number" brings additional > contributions for given "quality". > > But seems all strong programs just favor the "quality" in this trade-off, > because more playouts tend to make the same errors with previous playouts so > they can hardly recover these errors in reasonable time. On the other > hand, everyone admit that the random playout is clearly out of the sweet > spot area of this trade-off. > > Finally, I'm trying to find a "lower bound quality" for any meaningful > trade-off, but there seems no convincing answer. > > Bojun > > > > > _______________________________________________ > Computer-go mailing list > [email protected] > http://dvandva.org/cgi-bin/mailman/listinfo/computer-go > _______________________________________________ Computer-go mailing list [email protected] http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
