If anyone wants to TEST their program against one of the reference bots, you should get close to 50% score when playing lots of games.
If you play 1000 games, you should expect your score to be within the range 0.46838 - 0.53162 in order to be within two standard deviations, the amount usually considered to be within a reasonable amount of noise. If you play 10,000 games, you need to be within 49 - 51 percent. These ranges are such that you should expect the result to be outside that range less than 5% of the time (actually, about 4.5 percent) if the programs really are "equal." So far the Vala bot is well within 2 STD after almost 2000 games. It has a minor implementation defect so I am looking to see if it gets picked up. When 2 moves have equal scores, it does not choose between them randomly, but chooses the one first in the list (the highest rank first, then the lowest file.) With 1000 simulations that should make very little difference perhaps not even easily measurable. It's not even clear if it would make it play 1 or 2 ELO stronger or weaker if it does make a difference. Note: I can make a test that catches this too. - Don On Wed, 2008-10-22 at 14:10 -0400, Don Dailey wrote: > I have some preliminary performance numbers for various programming > languages with reference bots, I now have reference bots in 3 > languages. > > I also used the same data structure, and pretty much the same techniques > for each language. The Vala program is almost a direct port of the > java version (the language are VERY similar.) > > My disclaimer: I am sure the performance numbers for any given platform > could be easily improved, perhaps even with different compiler options > or simple changes to code. I am no expert in any of these languages. > > Below is the data as I have it now. > > - Don > > > > platform: core 2 duo e6700 > > vendor_id : GenuineIntel > cpu family : 6 > model : 15 > model name : Intel(R) Core(TM)2 CPU 6700 @ 2.66GHz > stepping : 6 > cpu MHz : 2667.000 > cache size : 4096 KB > > boardsize 9 > komi 0.5 > > Starting position > > PROGRAM SPEED TIME MOVE TOTAL NODES AVE > SCORE > ------------ ----- -------- ---- ------------- ------------ > crefgo 1.00 40.2 E5 111,040,853 0.523997 > valago 1.38 55.70 E5 111,060,969 0.523488 > jrefgo.jar 1.69 68.16 E5 111,037,354 0.524622 > jrefgo 1.75 70.5 E5 111,048,045 0.523744 > > > > crefgo > --------------------------------------------------------------- > gcc version 4.2.3 > options: -O3 -march=native > > > valago > --------------------------------------------------------------- > Vala version 0.3.5 > compile: valac --Xcc=-O3 --disable-assert --disable-checking > --disable-non-null -o vgo valaGo.vala > > > jrefgo.jar > ---------------------------------------------------------------- > version: ibm java 1.6.0 > compile: javac -O > > > jrefgo > ---------------------------------------------------------------- > gcj version 4.2.3 (gcc native code compiler) > compile: gcj -O3 > > _______________________________________________ > computer-go mailing list > [email protected] > http://www.computer-go.org/mailman/listinfo/computer-go/
signature.asc
Description: This is a digitally signed message part
_______________________________________________ computer-go mailing list [email protected] http://www.computer-go.org/mailman/listinfo/computer-go/
