Hi Mark, the procedure you mention (self play and judge by XG) is what I'm doing as well. Making rollouts of the questionable positions should be done by both/all bots, Otherwise the ground truth might be biased. If they still disagree …. well then we don’t know what’s the truth.
Additionally what does stronger mean? Which car is better? An Unimog or a Porsche 911? It depends whether you want to do an expedition in the Amazonas or go to Le Mans for sure. Stronger in Backgammon means "stronger on average" and not "always better". And errors by XG (or GnuBG or BGBlitz or Octopus, BSage, bg-engine, …. ) doesn't have to be marginal. E.g. using XG as a judge in a deep containment positions might not be a particular good idea. What is needed is an API to do bot battles, although I don’t think that we’ll ever have XG implement it, Seeing three new super human AIs in the last year is promising after a decade of drought and might shatter the blind faith in XG. We have some proposals for an API that are well suited for bot battles, but I think that keeps BG-AIs in a niche. I want end users that have very little IT-knowledge to be able to install a 3rd-party AI and use it. I will probably start on that in summer and would be very delighted if GnuBG will cooperate/implement it as well (naturally I publish the API and necessary source code even if you guys wont be interested). best Frank
