I was not looking forward to the enormous amount of work involved in organizing another scalability study so I'm happy you have already done so.
One advantage however to involving a lot of people and their computers is that I COULD do the study out to enormous numbers of playouts, given enough help. I think it could be pretty simply done if it's done via ssh and the "loan" of linux machines if we can get people to agree to give us accounts on their machines for this study. At what point does memory become a scaling issue? My home-brewed chess auto tester multiplexes as many simulatenous games as I want (it was designed for multi-core automated tested) I would only have to convert it to GO (which I could do in half a day easy) It handles all the gory details and would be the easiest way to run such a study. All I would need is ssh accounts on linux machines. Do you think such a thing is worth the effort, or are you satisfied with the study you have done? I would probably have multiple instance of EACH program, not a single fixed opponent for one program but that does require a lot more games. Don On Fri, Jun 17, 2011 at 1:32 PM, Jean-loup Gailly <[email protected]> wrote: > Don writes: > > > Maybe another scalability study is in order? > > I have done precisely this. The reports of scalability death are greatly > exaggerated, as you can see from the attached graph. To avoid self play > benchmarks which are misleading, I tested Pachi against Fuego 1.1. Fuego > uses constant 550 000 playouts per move (but since Fuego 1.0 this includes > counts from reused subtrees) and always 16 cores. Pachi used from 1 to 16 > cores and from 15K to 8M playouts per move. At the mid range, where Pachi > uses 16 cores and 250K playouts per move, both engines use roughly the same > time (15.9mn per game for Fuego, 15.3mn for Pachi) and the same amount of > memory (10 GB each). With the same number of cores and same playing time, > Pachi is about 3.5 stones stronger than Fuego so Pachi always plays white > with a komi of -22.5 to enable the scalability study on a larger range. > > You can see on the graph that, at 16 cores, one doubling from 250K to 500K > playouts brings about 100 elo improvement. This confirms previous studies. > To confirm scalability at the high end (beyond 8M playouts per move) > I would need a stronger opponent but unfortunately I don't have any. But at > least on a single machine at reasonable time settings, my measurements show > that we can still scale for many doublings. We can scale to at least 8M > playouts/move which take 9 hours per game with 16 cores but would take only > 30mn with 256 cores. (It's much harder to scale on multiple machines.) > > > I think that 5,000 games for a given player gives us an error margin > > of something around plus or minus 10 ELO. > > I confirm this. I have used exactly 5000 games for each experiment > and the experiments are reproducible within 10 elo. > > Pasky and I are still writing a detailed paper about Pachi but it > takes a little longer than expected, sorry. > > Jean-loup > > > > _______________________________________________ > Computer-go mailing list > [email protected] > http://dvandva.org/cgi-bin/mailman/listinfo/computer-go >
_______________________________________________ Computer-go mailing list [email protected] http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
