Yes! I actually think we need to think this through.... I actually think we need two kinds of protocols. One protocol for connecting a bot to a Graphic User Interface and another protocol to head-to-head matching.
>From my experience - if the protocol is "stateless", i.e. the board state is passed at every new situation, a head-to-head match becomes very slow. Not only does the messages passed become unnecessarily long, but there is also a lot of waste in parsing the board position into the bot's internal representation and so on. This is part of what kills the performance when doing head-to-head matching. I tried to do head-2-head matching through the GNU Backgammon external interface (the external interface is stateless and passes FIBS-boards and returns moves). It works fine, but it's very slow. Then knowing that you need hundreds of thousands of games to cut through the variance of the dice rolls, makes it infeasible to run a head-to-head matching through such an interface. (OK - I have not tried to play head-2-head matching through a state aware interface either.. maybe it is just as slow?) So for comparing bots to each other, the best I have come up with is wrapping the different bots into an API interface and running them in one and the same process. This goes blitzing fast and it can even be naively threaded. But this of course loses the generality we are seeking. I know that Jonathan is working on a head-to-head dueller tool. Maybe that's the best way to compare bots? https://github.com/bungogood/bgci . Also Eran Lambooij is working on OGXF (Open Gammon eXchange Format) which is a fileformat that stores neural networks and input calculations, such that you can ship a bot as a binary file. This might also be an interface to match two bots against each other. (And if a bot can be extracted in a binary format, it could also be used in a user interface connection.) The main requirement for a head-2-head matching system is that it is fast and this is so important that we might even should pay some generality to keep it fast. So... enough about tools and potential protocols to do head-2-head matching. The protocol that I think is even more precarious is a protocol to bind a user interface to a bot engine. This should, IMHO, be a stateless text-based interface, pretty much like UCI for chess. However this requires design, collaboration and implementation. -Øystein ons. 27. mai 2026 kl. 22:24 skrev Frank Berger <[email protected]>: > Hi Mark, > > the procedure you mention (self play and judge by XG) is what I'm doing as > well. Making rollouts of the questionable positions should be done by > both/all bots, Otherwise the ground truth might be biased. If they still > disagree …. well then we don’t know what’s the truth. > > > Additionally what does stronger mean? Which car is better? An Unimog or a > Porsche 911? It depends whether you want to do an expedition in the > Amazonas or go to Le Mans for sure. > Stronger in Backgammon means "stronger on average" and not "always > better". And errors by XG (or GnuBG or BGBlitz or Octopus, BSage, > bg-engine, …. ) doesn't have to be marginal. > E.g. using XG as a judge in a deep containment positions might not be a > particular good idea. > > What is needed is an API to do bot battles, although I don’t think that > we’ll ever have XG implement it, Seeing three new super human AIs in the > last year is promising after a decade of drought and might shatter the > blind faith in XG. We have some proposals for an API that are well suited > for bot battles, but I think that keeps BG-AIs in a niche. I want end users > that have very little IT-knowledge to be able to install a 3rd-party AI and > use it. I will probably start on that in summer and would be very delighted > if GnuBG will cooperate/implement it as well (naturally I publish the API > and necessary source code even if you guys wont be interested). > > best > Frank > > > >
