On Thu, Dec 5, 2019 at 6:34 PM Timothy Y. Chow <[email protected]> wrote:
> > > Regarding expected results, I also believe that backgammon bots are very > > close to perfection and whatever improvements (from any approach) will > > be marginal. > > In order to determine whether a new network is doing better than the old > network, it helps to have examples of positions where the old network is > clearly playing poorly. Here's one example of a game that I played > against eXtreme Gammon where the bot made a lot of obvious blunders: > > http://timothychow.net/cg/Games/7pt2015-05-24e%20Game%202.htm > > For example, search for "10/8 6/4(3)". The bot's ridiculous play here > would not be among the top 50 plays of any halfway decent human player. > Admittedly this was XG but I would expect GNU to behave similarly, if not > in these specific positions then in similar ones. > > I analyzed the positioned you mention using GNU Backgammon 1.06.002-mingw 32-Bit 20180802. Since I am not am experienced GNUBG user, if any GNUBG dev spots anything wrong with the following, feel free to correct me. I started with a 4-ply evaluation and the "correct" move is No 49 in the list of best moves. ID: AABCgDsTg4MAAA:AQHpACAAAAAE 1. Cubeful 4-ply 18/16 13/7* Eq.: +0.138 0.574 0.000 0.000 - 0.426 0.104 0.060 4-ply cubeful prune [4ply] 2. Cubeful 4-ply 10/4 6/4 Eq.: +0.136 (-0.003) 0.564 0.000 0.000 - 0.436 0.096 0.047 4-ply cubeful prune [4ply] 3. Cubeful 4-ply 23/21 13/7* Eq.: +0.117 (-0.022) 0.569 0.000 0.000 - 0.431 0.113 0.060 4-ply cubeful prune [4ply] 4. Cubeful 4-ply 18/16 10/6 5/3* Eq.: +0.109 (-0.029) 0.566 0.000 0.000 - 0.434 0.111 0.063 4-ply cubeful prune [4ply] ... 49. Cubeful 0-ply 13/7* 5/3* Eq.: +0.115 (-0.023) 0.552 0.000 0.000 - 0.448 0.087 0.050 0-ply cubeful prune [expert] What happened is that a good move got pruned at 0-ply because the default move filter for 4-ply eval at 0-ply is 16. So the best move did not reach deeper plies for better evaluation. I suspect something similar happened in your game with XG. Then I changed this setting to 50, and after waiting a minute or two, the move gets to the number 1 spot: 1. Cubeful 4-ply 13/7* 5/3* Eq.: +0.164 0.582 0.000 0.000 - 0.418 0.099 0.060 4-ply cubeful prune [4ply] 2. Cubeful 4-ply 18/16 13/7* Eq.: +0.138 (-0.026) 0.574 0.000 0.000 - 0.426 0.104 0.060 4-ply cubeful prune [4ply] 3. Cubeful 4-ply 10/4 6/4 Eq.: +0.136 (-0.028) 0.564 0.000 0.000 - 0.436 0.096 0.047 4-ply cubeful prune [4ply] 4. Cubeful 4-ply 13/7* 10/8 Eq.: +0.125 (-0.039) 0.571 0.000 0.000 - 0.429 0.110 0.060 4-ply cubeful prune [4ply] 5. Cubeful 2-ply 23/21 13/7* Eq.: +0.153 (-0.011) 0.582 0.000 0.000 - 0.418 0.109 0.061 2-ply cubeful prune [world class] The moral: If one needs to experience the full power of the bg bots one needs to change the default settings which are configured for the average user. Whatever errors bots occasionally make at their evaluations, they make up by searching deeper. Nikos Playing around with positions like this will quickly disabuse anyone of > the illusion that "backgammon bots are very close to perfection." > > As I recall, in the past, people have tried specifically training neural > nets on positions like these, as well as "snake" positions where you have > to roll a prime for a long distance, and the problem was that it seemed to > degrade performance on other types of positions. It's possible that, as > Papachristou suggests, recent incremental improvements in regularization > algorithms might be good enough to overcome these difficulties. Anecdotal > evidence from Robert Wachtel's revised version of "In the Game Until the > End" suggests that Xavier was able to improve eXtreme Gammon's post-coup > classique play significantly, without a wholesale switch to modern deep > learning methods. > > Tim > >
