Hi, > I actually think we need to think this through.... I actually think we need > two kinds of protocols. One protocol for connecting a bot to a Graphic User > Interface and another protocol to head-to-head matching. > > From my experience - if the protocol is "stateless", i.e. the board state is > passed at every new situation, a head-to-head match becomes very slow. Not > only does the messages passed become unnecessarily long, but there is also a > lot of waste in parsing the board position into the bot's internal > representation and so on. This is part of what kills the performance when > doing head-to-head matching.
I agree that stateless makes things easier. I disagree that it make a protocol slow. Even parsing the cli output of GnuBG was fast enough and only in the “move-analysis” window where about 70 requests had to be sent is only a slight pause perceivable compared to the included AI. A binary protocol (with ASCCI mode for debugging) will be pretty fast. I tend to suggest "protocol buffers” because it is battle proved, it supports many languages and the serializing code is generated. But I’m open to suggestions, e.g. I read cursory about Apache Fory yesterday which allows to use the native objects and don’t need transfer objects and has more advantages but it is not that widespread used as Protocol Buffers. But if anyone else says e.g.: should be JSON or whatever, I’m not dogmatic. > I tried to do head-2-head matching through the GNU Backgammon external > interface (the external interface is stateless and passes FIBS-boards and > returns moves). It works fine, but it's very slow. Then knowing that you need > hundreds of thousands of games to cut through the variance of the dice rolls, > makes it infeasible to run a head-to-head matching through such an interface. Probably even more games are needed. 0-ply (1-ply rest of the world) is good to see errors in the bot that might be fixed by deeper searches on the other hand a higher ply might be more realistic. Maybe a time based setting (e.g. ABT setting) might be regarded as fair, > > So for comparing bots to each other, the best I have come up with is wrapping > the different bots into an API interface and running them in one and the same > process. This goes blitzing fast and it can even be naively threaded. But > this of course loses the generality we are seeking. I know that Jonathan is > working on a head-to-head dueller tool. Maybe that's the best way to compare > bots? https://github.com/bungogood/bgci . Also Eran Lambooij is working on > OGXF (Open Gammon eXchange Format) which is a fileformat that stores neural > networks and input calculations, such that you can ship a bot as a binary > file. This might also be an interface to match two bots against each other. > (And if a bot can be extracted in a binary format, it could also be used in a > user interface connection.) > The main requirement for a head-2-head matching system is that it is fast and > this is so important that we might even should pay some generality to keep it > fast. > > So... enough about tools and potential protocols to do head-2-head matching. > The protocol that I think is even more precarious is a protocol to bind a > user interface to a bot engine. This should, IMHO, be a stateless text-based > interface, pretty much like UCI for chess. However this requires design, > collaboration and implementation. As soon as 3.4.2 is out I’ll make some suggestions. best Frank
