Magnus, This is a really good experiment, I think it is well past due for someone to try something like this and report the results.
I know that it's not possible to make the experimental setup perfect, but it is important to establish some kind of baseline result to work from. I hope you are able to play enough games to be statistically significant. I have seen way too many papers on games of various kinds where some algorithm is tried and they "prove" it is helpful with 100 games and the score is 55% or 60% - that is just not enough to be statistically convincing. Don On Mon, Oct 4, 2010 at 2:05 PM, <[email protected]> wrote: > Hi Don! > > I just started an experiment! First some thought about what I believe is > the factors involved. And then I describe the setup and some preliminary > impressions and results. > > > Quoting Don Dailey <[email protected]>: > > 1. boardsize could be a factor. >> > > I would not expect it to be useful on 9x9, especially dynamic komi would > get incredibly unstable on small board towards the end of the game. So what > I am testing is 19x19. > > > 2. it may be more feasible in handicap games. >> > > My previous experiments for even game showed little or no gain. My > subjective feeling is that it is in high handicap games on 19x19 where > effect can be strong. > > > 3. it may work better against humans for psychological reasons. >> > > I cannot measure that. > > > 4. Maybe I did not stumble on the right approach when I did my >> experiments. >> > > With dynamic komi the MC evaluation suffers more from noise because when > the win rates get close to 100% it even makes MC programs play randomly (or > systematically from the top left corner or something similar). This problem > becomes greater the stronger the program is. It is probably not so bad > because a stronger program will also have larger difference in win rates for > strong and weak moves. This is my explanation why Valkyria probably only > benefits from. > > The other reason why dynamic komi may be good is that a MCTS program > prefers high win rates over a safe winning margin leading to the 0.5 points > syndrome. On 9x9 boards present programs do not appear to suffer from this. > They are so strong that with a "safe" 0.5 lead they win without mistakes. > And if they do suffer from it still it is not clearly visible. It looks like > a middle game mistake and then they realize they are behind a wild fight > occur. > > On 19x19 this is probably a major problem for most program without dynamic > komi or some other greedy search method. If a game is long and the score > drops to 0.5 long before the game is finished the program must play the rest > of the game as strongly as the opponent. This is a disaster for black in > high handicap games. In the worst case even an opponent just slightly > stronger can win with a large handicap, because the game become even too > soon. > > Ok, enough babbling about the stuff dynamic komi could possible fix. > > I just started an experiment to make Don happy. It is not so easy to make > such an experiment completely satisfactorily but it will be a first step to > test Valkyria at least. > > I want to show that Valkyria plays stronger as black with 9 handicap stones > on 19x19. The problem is that I would need an extremely strong opponent to > beat Valkyria as white under normal time constraint. And I have no access to > such an opponent. > > So I let Valkyria play Black with about 1s per move and White with 10s per > move approximately (1000 and 10000 playouts respectively). > > I run two conditions in parallel one where black use dynamic komi and one > where it play without it. My hope is that the modest difference in playing > strength will be enough to make a difference. In the worst case black wins > 100% of the games in both conditions. > > The good news is that I think the effect of dynamic komi in this particular > experiment is huge just from watching the very first games. > > With dynamic Komi black is really aggressive. It is tactically weak but > fights in a way that the white groups are eyeless and have no territory. It > kills large white groups in almost every game. > > Without dynamic komi white is allowed to establish relatively strong groups > all over the board without a fight and gets closer to an even game for > almost every move played. So far white won 5 out 7 games which I did not > expect. 10k vs 1k should not be more than a 3-4 stones I thought. > > White has won 1 game out of 6 against black with dynamic komi. > > It is too early to make any conclusions yet but if the effect is as large > at it seems to be I can probably give some convincing numbers already > tomorrow. > > Caution is important also. As in all self play experiments effect tend to > be exaggerated. The next step is this seems to be a good setup is to test > against Fuego. I would also like to scale it up using 10k playouts for black > and 100k playouts for white to if effect stays with better play for both > colors. > > It may also be that the effect is double. White with h9 has winrates below > 20% in these games and play desperate move rather than patient moves quite > often. Maybe white can use dynamic komi too to some degree? Or perhaps a > linear komi to make it play the opening more sound and then get more > desperate later? > > -Magnus > > _______________________________________________ > Computer-go mailing list > [email protected] > http://dvandva.org/cgi-bin/mailman/listinfo/computer-go >
_______________________________________________ Computer-go mailing list [email protected] http://dvandva.org/cgi-bin/mailman/listinfo/computer-go
