it's a natural tendency to look for patterns in data as soon as you have any data at all. some of these patterns i'd be willing to bet will hold up over time -- but the bayesian in me would say that is simply because they have further given evidence for my prior beliefs.
requiring everyone to stay out of everyone else's first standard deviation will take quite a few more trials, and probably won't change the order of the existing mogos 1-12, for instance. if you were to plot these datapoints with their 1st or 2nd std. dev. errorbars and look at the possible set of curves that you could fit through them, though, it'd give quite a funny story, i'd agree. :) 95% "confidence" is a bit misleading and overrated, in my opinion. s. ----- Original Message ---- From: Hideki Kato <[EMAIL PROTECTED]> To: computer-go <[email protected]> Sent: Thursday, January 24, 2008 8:34:42 AM Subject: [computer-go] Re: Scalbility study: low end Heikki, The numbers of games are about 200 and their ratings' standard deviations (right of Elo) are 70 to 100, right now. To get 95% of reliability, you have to double them. Don't you think it's too early to conclude any? -Hideki Heikki Levanto: <[EMAIL PROTECTED]>: >Everyone is looking at the top end of the scalability study > http://cgos.boardspace.net/study/ > >But what happens in the low end? Both programs show linear progress to begin >with, then a corner, and more (almost?) linear development. > >Fatman's curve has a clear break at 3 doublings, when it suddenly starts to >improve much slower than before. This goes on until 12 doublings, after which >we get the mysterious decline. > >Mogo's curve is pretty well linear to 4 doublings, after that there is more >variation (I suppose random), but the overall scope is clearly not what it >was below 4. > > >It is possible that both programs have a subtle bug that starts to disturb >results around this point, but I find it quite unlikely. > >The breaks happe at 1350 - 1550 ELO points. Isn't that about the level where >plain MC stops improving with more playouts? Would be fun to see if we could >isolate the playout parts of those programs, and let them play pure MC. My >guess is that they would end up around this level. > > >Could it be that there are other limiting factors higher up? Perhaps Fatman >is hitting the next one around 12 doublings, and Mogo will follow at 14 or >15... We will see that in a few days, when the new Mogos join the study and >start producing results. > > - Heikki -- [EMAIL PROTECTED] (Kato) _______________________________________________ computer-go mailing list [email protected] http://www.computer-go.org/mailman/listinfo/computer-go/ ____________________________________________________________________________________ Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping _______________________________________________ computer-go mailing list [email protected] http://www.computer-go.org/mailman/listinfo/computer-go/
