On Tue, Feb 18, 2003 at 06:58:30PM -0500, Ben Goertzel wrote: > However, I do think he ended up making a good point about AIXItl, which is > that an AIXItl will probably be a lot worse at modeling other AIXItl's, than > a human is at modeling other humans. This suggests that AIXItl's playing > cooperative games with each other, will likely fare worse than humans > playing cooperative games with each other.
That's because AIXI wasn't designed with game theory in mind. I.e., the reason that it doesn't handle cooperative games is that it wasn't designed to. As the abstract says, AIXI is a combination of decision theory with Solomonoff's theory of universal induction. We know that game theory subsumes decision theory as a special case (where there is only one player) but not the other way around. Central to multi-player game theory is the concept of Nash equilibrium, which doesn't exist in decision theory. If you apply decision theory to multi-player games, you're going to end up with an infinite recursion where you try to predict the other players trying to predict you trying to predict the other players, and so on. If you cut this infinite recursion off at an arbitrary point, as AIXI-tl would, of course you're not going to get good results. > > I always thought that the biggest problem with the AIXI model is that it > > assumes that something in the environment is evaluating the AI and giving > > it rewards, so the easiest way for the AI to obtain its rewards would be > > to coerce or subvert the evaluator rather than to accomplish any real > > goals. I wrote a bit more about this problem at > > http://www.mail-archive.com/[email protected]/msg03620.html. > > I agree, this is a weakness of AIXI/AIXItl as a practical AI design. In > humans, and in a more pragmatic AI design like Novamente, one has a > situation where the system's goals adapt and change along with the rest of > the system, beginning from (and sometimes but not always straying far from) > a set of initial goals. This seems to be a non-sequitor. The weakness of AIXI is not that it's goals don't change, but that it has no goals other than to maximize an externally given reward. So it's going to do whatever it predicts will most efficiently produce that reward, which is to coerce or subvert the evaluator. If you start with such a goal, I don't see how allowing the system to change its goals is going to help. But I think Eliezer's real point, which I'm not sure has come across, is that if you didn't spot such an obvious flaw right away, maybe you shouldn't trust your intuitions about what is safe and what is not. ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
