Don writes: > Maybe another scalability study is in order?
I have done precisely this. The reports of scalability death are greatly exaggerated, as you can see from the attached graph. To avoid self play benchmarks which are misleading, I tested Pachi against Fuego 1.1. Fuego uses constant 550 000 playouts per move (but since Fuego 1.0 this includes counts from reused subtrees) and always 16 cores. Pachi used from 1 to 16 cores and from 15K to 8M playouts per move. At the mid range, where Pachi uses 16 cores and 250K playouts per move, both engines use roughly the same time (15.9mn per game for Fuego, 15.3mn for Pachi) and the same amount of memory (10 GB each). With the same number of cores and same playing time, Pachi is about 3.5 stones stronger than Fuego so Pachi always plays white with a komi of -22.5 to enable the scalability study on a larger range. You can see on the graph that, at 16 cores, one doubling from 250K to 500K playouts brings about 100 elo improvement. This confirms previous studies. To confirm scalability at the high end (beyond 8M playouts per move) I would need a stronger opponent but unfortunately I don't have any. But at least on a single machine at reasonable time settings, my measurements show that we can still scale for many doublings. We can scale to at least 8M playouts/move which take 9 hours per game with 16 cores but would take only 30mn with 256 cores. (It's much harder to scale on multiple machines.) > I think that 5,000 games for a given player gives us an error margin > of something around plus or minus 10 ELO. I confirm this. I have used exactly 5000 games for each experiment and the experiments are reproducible within 10 elo. Pasky and I are still writing a detailed paper about Pachi but it takes a little longer than expected, sorry. Jean-loup
<<attachment: pachi-scalability.gif>>
_______________________________________________ Computer-go mailing list [email protected] http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
