Hi Eliezer, > An intuitively fair, physically realizable challenge, with important > real-world analogues, formalizable as a computation which can be fed > either a tl-bounded uploaded human or an AIXI-tl, for which the human > enjoys greater success measured strictly by total reward over time, due to > the superior strategy employed by that human as the result of rational > reasoning of a type not accessible to AIXI-tl. > > Roughly speaking: > > A (selfish) human upload can engage in complex cooperative strategies with > an exact (selfish) clone, and this ability is not accessible to AIXI-tl, > since AIXI-tl itself is not tl-bounded and therefore cannot be simulated > by AIXI-tl, nor does AIXI-tl have any means of abstractly representing the > concept "a copy of myself". Similarly, AIXI is not computable and > therefore cannot be simulated by AIXI. Thus both AIXI and AIXI-tl break > down in dealing with a physical environment that contains one or more > copies of them. You might say that AIXI and AIXI-tl can both do anything > except recognize themselves in a mirror.
Why do you require an AIXI or AIXI-tl to simulate itself, when humans cannot? A human cannot know that another human is an exact clone of itself. All humans or AIXIs can know is what they observe. They cannot know that another mind is identical. > The simplest case is the one-shot Prisoner's Dilemna against your own > exact clone. It's pretty easy to formalize this challenge as a > computation that accepts either a human upload or an AIXI-tl. This > obviously breaks the AIXI-tl formalism. Does it break AIXI-tl? This > question is more complex than you might think. For simple problems, > there's a nonobvious way for AIXI-tl to stumble onto incorrect hypotheses > which imply cooperative strategies, such that these hypotheses are stable > under the further evidence then received. I would expect there to be > classes of complex cooperative problems in which the chaotic attractor > AIXI-tl converges to is suboptimal, but I have not proved it. It is > definitely true that the physical problem breaks the AIXI formalism and > that a human upload can straightforwardly converge to optimal cooperative > strategies based on a model of reality which is more correct than any > AIXI-tl is capable of achieving. Given that humans can only know what they observe, and thus cannot know what is going on inside another mind, humans are on the same footing as AIXIs in Prisoner's Dilema. I suspect that two AIXIs or AIXI-tl's will do well at the game, since a strategy with betrayal probably needs a longer program than a startegy without betrayal, and the AIXI will weight more strongly a model of the other's behavior with a shorter program. > Ultimately AIXI's decision process breaks down in our physical universe > because AIXI models an environmental reality with which it interacts, > instead of modeling a naturalistic reality within which it is embedded. > It's one of two major formal differences between AIXI's foundations and > Novamente's. Unfortunately there is a third foundational difference > between AIXI and a Friendly AI. I will grant you one thing: that since an AIXI cannot exist and an AIXI-tl is too slow to be practical, using them as a basis for discussing safe AGIs is a bit futile. The other problem is that an AIXI's optimality is only as valid as its assumption about the probability distribution of universal Turing machine programs. Cheers, Bill ---------------------------------------------------------- Bill Hibbard, SSEC, 1225 W. Dayton St., Madison, WI 53706 [EMAIL PROTECTED] 608-263-4427 fax: 608-263-6738 http://www.ssec.wisc.edu/~billh/vis.html ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
