Yes, several comments are right on the money. You cannot trust data based on less than a few hundred games - although you can clearly see very general trends (programs tend to get stronger.)
You may have noticed the scale has been expanding ever since we started. Mogo13 was one time around 2000, then 2100, then 2200 and it keeps going up and has not stopped. This is probably an artifact of bayeselo which assumes all players are equal and requires a lot of data to refute that. In other works if you win a 4 game match by the score of 3 out of 4, the rating bayeselo would assign would predict a much closer match. - Don Magnus Persson wrote: > Quoting Heikki Levanto <[EMAIL PROTECTED]>: > >> Of course we can wait until we get more data. I just wanted to share my >> observation that the curves seem to change around the level where MC >> playouts >> tend to flatten out, and hear if anyone would have some insightful >> comments >> to that. Even with the risk that more data may invalidate my >> observation... > > Nothing wrong with that, I do it myself all the time for my own tests. > But I have tricked myself at lot of time plotting curves like this. > > As a rule of thumb I want 300 games for each datapoint and at least > 500 if I am going to make any conclusions. > > Also note that this project collects data for a lot of datapoints at > the same time. This means that by chance there must be several > datapoints that are quite far off from where they should be, but the > human eye will not correct for that. So making conclusions from > details of these curves I would prefer even more games for each > datapoint. Perhaps a 1000 games or so. > > Interestingly the column of opponent rating indicate that many of the > programs has played against harder or weaker opposition than expected, > meaning that a lot of games played might have had a very predictable > outcome. Thus the number of games played are probably to some extent > misleading for how accurate the results are. > > -Magnus > > _______________________________________________ > computer-go mailing list > [email protected] > http://www.computer-go.org/mailman/listinfo/computer-go/ > _______________________________________________ computer-go mailing list [email protected] http://www.computer-go.org/mailman/listinfo/computer-go/
