Darren Cook: <[email protected]>: >>> I have done precisely this. The reports of scalability death are >>> greatly exaggerated, as you can see from the attached graph. To >>> avoid self play benchmarks which are misleading, I tested Pachi >>> against Fuego 1.1. ... >>> >> Well this gives a biased solution. Wrong sample so to speak. Fuego >> will not create complex semeais and hard-to-read ishi-no-shita nakade >> shapes i.e opponent that puts no pressure to known problems . So you >> prove that against an opponent who does not play like human you do >> scale. But as you advance up the ladder of human players these small >> issues tend to pop-up more often. > >+1. This is exactly what was nagging at my mind as I read this thread. > >Hideki's hypothesis (as I understood it) is that increased CPU stops >helping at a certain point because, against very strong human opponents: > * there is systematic error in the evaluations; > * doubling CPU doesn't help remove this error > >(I think Hideki is also saying that this point, where CPU stops helping, >is rapidly approaching and that therefore this is a practical problem.)
The rapid approach could be Zen's problem (but not limited to, I guess). Since Zen is a prototype of a commercial product, Yamato had tuned Zen as few playouts as possible (with some fixed performance) but this caused the search too selective and more thinking time contributes little, ie, rarely changes the optimal move. (I found this at a slow bot tournament, thanks Nick.) This was also caused by the benchmarks with very short time settings used for the tuning. Recently, Yamato uses longer time benchmarks and now Zen has more potential against time. So, other programs could have more room to be improved by time, but highly tuned MCTS bots with short time settings might be caught by this trap. >Playing a tournament between programs that all have this same problem >just means you get some very high level games where they all avoid the >types of games that will challenge them. Yes, such games could be limited in a subspace of real Go. (This could apply not only MCTS bots but also (weak) human players.) >Darren > >P.S. But, not wanting to be totally negative, here is one idea: > > * Collect a set of problem positions where the current strongest >programs mess up. > * Use each of these as the start position, instead of an empty board, >in your scaling study. > >(In other words, don't let the programs play the style of game they want >to play, but force them to start outside their comfort zone. > >If Hideki is right, you should see very little improvement for each >doubling. Darrent, at least one program that can correctly manage those positions is requred for your plan, isn't it? Hideki >This has its own bias: the selection of problems. However, it could make >for an interesting and educational grudge match: > In the red corner, those who agree that scaling won't get you much >further, hunt for some really great problem positions. > In the blue corner, those who disagree, scrounge even more CPU power to >see if just one more doubling will give a breakthrough. >_______________________________________________ >Computer-go mailing list >[email protected] >http://dvandva.org/cgi-bin/mailman/listinfo/computer-go -- Hideki Kato <mailto:[email protected]> _______________________________________________ Computer-go mailing list [email protected] http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
