Re: [computer-go] 13x13 study. Time and memory
Le jeudi 21 février 2008, Don Dailey a écrit : If you look at the table you will notice that going from level 4 to level 11 (which is 7 doublings and should take 128X longer) only takes 59.43 X longer. So if we plot 9X9 rank vs time, maybe we have a straight line :) ELO vs size of the tree (or memory usage) should show the same ? On 13x13 study it does, but there are not enough data at high level. Alain ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] 13x13 study. Time and memory
2008/2/22, Alain Baeckeroot [EMAIL PROTECTED]: Le jeudi 21 février 2008, Don Dailey a écrit : If you look at the table you will notice that going from level 4 to level 11 (which is 7 doublings and should take 128X longer) only takes 59.43 X longer. So if we plot 9X9 rank vs time, maybe we have a straight line :) It would indeed be very interesting to see that plot! Sylvain ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] 13x13 study. Time and memory
Le vendredi 22 février 2008, Sylvain Gelly a écrit : 2008/2/22, Alain Baeckeroot [EMAIL PROTECTED]: Le jeudi 21 février 2008, Don Dailey a écrit : If you look at the table you will notice that going from level 4 to level 11 (which is 7 doublings and should take 128X longer) only takes 59.43 X longer. So if we plot 9X9 rank vs time, maybe we have a straight line :) It would indeed be very interesting to see that plot! and if i remember it was Don's initial claim, that doubling thinking time (for humans and scalable bots) will produce a fixed Elo increase, at least until exhaustion of other resources (human fall asleep, bot fill memory ...) Alain ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] 13x13 study. Time and memory
Alain Baeckeroot wrote: Le vendredi 22 février 2008, Sylvain Gelly a écrit : 2008/2/22, Alain Baeckeroot [EMAIL PROTECTED]: Le jeudi 21 février 2008, Don Dailey a écrit : If you look at the table you will notice that going from level 4 to level 11 (which is 7 doublings and should take 128X longer) only takes 59.43 X longer. So if we plot 9X9 rank vs time, maybe we have a straight line :) It would indeed be very interesting to see that plot! and if i remember it was Don's initial claim, that doubling thinking time (for humans and scalable bots) will produce a fixed Elo increase, at least until exhaustion of other resources (human fall asleep, bot fill memory ...) I didn't predict perfect linearity, I expect it to gradually curve until it's horizontal as the programs approach perfection ASSUMING the program is completely scalable and isn't hitting internal limits as you say. However, it should appear to us as nearly linear because we are so far from perfect play. Especially at 13x13 and beyond. Since I am no go expert I can't speak for 9x9, but my belief (I'm willing to be wrong here) is that mogo at really high levels of thinking time on fast computers is playing pretty strong - strong enough to bend the curve. This is despite the games that David Fotland looked at and criticized.I do not doubt his analysis that it played bad moves, but my experience has always been that human experts look at games and under-rate computers by enormous amounts based on a few moves that rub them wrong. The typical scenario is that they see one bad move - and from that point on they lose all objectivity. Besides, I don't think even a highly objective human can accurately assign ratings or rankings based on simply looking at moves. There is also the phenomenon that if you look at a LOT of data points, you can perceive the curve more easily when graphed.In chess studies over the years we have typically looked at only a few data points and saw a straight line.But with mogo we are looking at a huge range - from the very weak to the very strong. With 18 doublings for Mogo on the 9x9 study, there is an enormous difference between the best and the worst.We also saw some empirical evidence that mogo was suffering from memory limits and this was throttling it's strength downward at the upper levels. With chess I think the ELO advantage of a doubling has decreased from about 100 ELO to about 50 or 60 - I'm not sure of the exact figure. But that's pretty amazing that we can still keep finding 50 ELO points, already humans cannot compete. I would also mention that it is not difficult to find bad moves - they still make mistakes and sometimes they are ugly - but can you beat it? Nope! - Don Alain ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/ ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
[computer-go] 13x13 study.
The study is running very well. We have 32 computers being used so far, some participants are providing 2 (or even more) computers. It would be great to get even more as we get into higher levels, as it will take a LOT of power to get a lot of games when each games takes 2 or 3 hours or more. http://cgos.boardspace.net/study/13/index.html - Don ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] 13x13 study.
Le jeudi 21 février 2008, Don Dailey a écrit : The study is running very well. We have 32 computers being used so far, some participants are providing 2 (or even more) computers. It would be great to get even more as we get into higher levels, as it will take a LOT of power to get a lot of games when each games takes 2 or 3 hours or more. Is it possible to have a indicative time and memory consumption at each level, for 9x9 and 13x13 study ? I guess this does not vary too much on recent hardware (factor 2 at max), and it is interesting to extrapolate how the scalability is (im)possible. Thanks a lot for this interesting and convincing study. Alain ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] 13x13 study.
It would be great to get even more as we get into higher levels, as it
will take a LOT of power to get a lot of games when each games takes 2
or 3 hours or more.
Is it possible to have a indicative time and memory consumption at each
level, for 9x9 and 13x13 study ?
I guess this does not vary too much on recent hardware (factor 2 at max),
and it is interesting to extrapolate how the scalability is (im)possible.
I can give you the data for my locally running instance of the study - I
have a core 2 duo system.
One interesting thing about mogo is that even when you set it to a fixed
level, it has heuristics to stop the search early - so you see that the
following table does not indicate that each level take 2X time, instead
it is something less.
It's interesting also that mogo is much stronger than gnugo at the same
basic time control - at least 300 ELO stronger.
PLAYERTIME/GME RATING GAMES WIN% Total games: 1028
--- - ---
Mogo_13_11 645.36 2493.2 5986.44 Mogo at 131072 play-outs
Mogo_13_10 367.31 2322.5 6063.33 Mogo at 65536 play-outs
Mogo_13_09 194.76 2231.114166.67 Mogo at 32768 play-outs
Mogo_13_08 109.11 2103.812249.18 Mogo at 16384 play-outs
Mogo_13_07 79.86 2010.112551.20 Mogo at 8192 play-outs
Mogo_13_06 39.13 1823.013034.62 Mogo at 4096 play-outs
Gnugo-3.7.11137.56 1800.029768.69 Gnugo level 8
Mogo_13_05 19.80 1741.711642.24 Mogo at 2048 play-outs
Mogo_13_04 10.86 1629.126638.72 Mogo at 1024 play-outs
Mogo_13_035.29 1398.423855.46 Mogo at 512 play-outs
Mogo_13_022.90 1198.927657.97 Mogo at 256 play-outs
Mogo_13_011.69909.922612.39 Mogo at 128 play-outs
By the way, the ELO ratings are not very meaningful from this report
because they are only rough estimates for pairing purposes and they are
designed to converge very slowly.
- Don
Thanks a lot for this interesting and convincing study.
Alain
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] 13x13 study.
If you look at the table you will notice that going from level 4 to
level 11 (which is 7 doublings and should take 128X longer) only takes
59.43 X longer.
Some possible hypothesis:
1. Computer more loaded during early games?
2. Mogo resigns earlier at longer levels.
3. Mogo's stop early heuristic works better at longer levels.
PLAYERTIME/GME RATING GAMES WIN% Total games: 1028
--- - ---
Mogo_13_11 645.36 2493.2 5986.44 Mogo at 131072 play-outs
Mogo_13_10 367.31 2322.5 6063.33 Mogo at 65536 play-outs
Mogo_13_09 194.76 2231.114166.67 Mogo at 32768 play-outs
Mogo_13_08 109.11 2103.812249.18 Mogo at 16384 play-outs
Mogo_13_07 79.86 2010.112551.20 Mogo at 8192 play-outs
Mogo_13_06 39.13 1823.013034.62 Mogo at 4096 play-outs
Gnugo-3.7.11137.56 1800.029768.69 Gnugo level 8
Mogo_13_05 19.80 1741.711642.24 Mogo at 2048 play-outs
Mogo_13_04 10.86 1629.126638.72 Mogo at 1024 play-outs
Mogo_13_035.29 1398.423855.46 Mogo at 512 play-outs
Mogo_13_022.90 1198.927657.97 Mogo at 256 play-outs
Mogo_13_011.69909.922612.39 Mogo at 128 play-outs
- Don
___
computer-go mailing list
computer-go@computer-go.org
http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] 13x13 study.
Hi Don, 2008/2/21, Don Dailey [EMAIL PROTECTED]: If you look at the table you will notice that going from level 4 to level 11 (which is 7 doublings and should take 128X longer) only takes 59.43 X longer. Mogo's stop early heuristic works better at longer levels. That is actually very interesting, and may be a new hypothesis for the scalability limits we saw in 9x9. There are two kind of stop early heuristics - a safe one, in the following case: if we began to always simulate the second best move, it would not have more simulations that the first best move at the end of the time limit. As the chosen move is the one with the maximum number of simulations, there is no point to continue thinking. - a risky one, in the following case: if the first best move have more than x% of all simulations, and the ratio first best move/second best move (in number of simulations) is more than y, and the total number of simulations is greater than expected total of simulations / 2, then we stop. There is also a hard stop early in the following case: if the first best move have more than 1-(1-x%)/2 of all simulations, and the ratio first best move/second best move (in number of simulations) is more than 2 * y, and the total number of simulations is greater than expected total of simulations / 4, then we stop Maybe x and y are not adapted to long thinking time (stop too early in a loosing move). Or maybe they are, and it worth saving time :). Anyway, it is normal that we longer thinking time, even the first heuristic arrives much more often. Sylvain ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] 13x13 study.
On Thu, Feb 21, 2008 at 04:08:56PM -0500, Don Dailey wrote: It's also interesting how the graph up to level 11 seems to form 2 very straight lines, almost as if they were connected at an angle. This must be a by-product of how we started the test. We played only the first 4 levels as we were testing the system and that is where the bend point is.Then I added levels gradually.I cannot figure out why this would cause such strange behavior. I noticed the same angle in the 9x9 study, more on Fatman, but also on Mogo. To me it still seems to be an interesting coincidence (if that what it is) that it happens about the level where a MC program (without a tree search) levels off. On 9x9, it was around 1400 elo for Fatman and 1600 for Mogo. The same seems to apply here. It would be interesting to see a similar study with pure MC programs (no tree search of any kind). And/or to get a few 'in-between' data points near the corner. Or to make the MC simulations weaker/stronger, and see how that affects the performance of the UCT programs... If I had all the machine power, and nothing better to do... -H -- Heikki Levanto In Murphy We Turst heikki (at) lsd (dot) dk ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
Re: [computer-go] 13x13 study.
--- Don Dailey [EMAIL PROTECTED] wrote: It's also interesting how the graph up to level 11 seems to form 2 very straight lines, almost as if they were connected at an angle. This must be a by-product of how we started the test. We played only the first 4 levels as we were testing the system and that is where the bend point is.Then I added levels gradually. I cannot figure out why this would cause such strange behavior. Is it possible to generate a cross-table? Level x plays opponents 1,2,3,4,5... with so many games each and such-and-such winning percentage? Perhaps that would help understand the results. The upper part of the graph looks very interesting, but there aren't many games there yet. Terry McIntyre lt;[EMAIL PROTECTED]gt; Wherever is found what is called a paternal government, there is found state education. It has been discovered that the best way to insure implicit obedience is to commence tyranny in the nursery. Benjamin Disraeli, Speech in the House of Commons [June 15, 1874] Looking for last minute shopping deals? Find them fast with Yahoo! Search. http://tools.search.yahoo.com/newsearch/category.php?category=shopping ___ computer-go mailing list computer-go@computer-go.org http://www.computer-go.org/mailman/listinfo/computer-go/
