On Thu, 2008-08-28 at 08:37 -0700, terry mcintyre wrote: > Regarding a rating system which provides more dimensions, may I suggest a > test suite of problems at different levels? > > Convert life-and-death problems to "solve this problem or lose the game" > situations which can be properly appreciated by monte carlo programs, and > make a guesstimate of the elo rating required to solve such problems. Tweak > the estimates depending on the results of tests with various programs. > > This will expose strengths and weaknesses of particular programs. You'll find > out the sort of things which humans discover - this program doesn't know how > to handle that joseki variation - can't deal with that sort of nakade - but > it is scary accurate in this situation.
That is a good way to instrument skill - just design problem sets to measure various skill aspects, a kind of I.Q. test for software players. The scary strong Rybka program claims to be weak tactically. The developers say that problem solving skill does not correlate strongly with playing strength and they don't tune or care about that. Of course there is a correlation, every program is a genius tactically but among the very best programs this is a very inaccurate and crude measure of playing strength. And I think I understand why. Winning games is a skill that involves so many different skills that it isn't funny. The enormous depths that chess program achieve came at some kind of sacrifice that turns out to be favorable for winning games. They are highly selective, and probably miss a LOT of stuff while picking up a LOT of stuff and the mixture seems to work. The key to this seems to be, however, that when you increase the hardware, for a given program EVERY weakness gets reduced. You will on average pick up things you would have missed with the weaker hardware. You are like a ratchet wrench, you cannot go backwards, only forward. Yesterdays intransitivity are minimized on todays hardware and so on. Null move selectivity is a thing that has added a lot of strength to chess programs when it was discovered and first used. It DID create some problems however. Those attacks I spoke of previously, where you build up slowly to attack the opponent king - null move exacerbated this problem. But it made programs stronger on average, and it didn't stop programs from improving with more hardware. That's why I skeptical of the idea implied by some that MCTS is the wrong approach for beating humans because it will surely hit some imaginary wall (but only against humans.) - Don > > > > _______________________________________________ > computer-go mailing list > [email protected] > http://www.computer-go.org/mailman/listinfo/computer-go/ _______________________________________________ computer-go mailing list [email protected] http://www.computer-go.org/mailman/listinfo/computer-go/
