Bill Hibbard wrote:
The halting problem establishes that you can never perfectly understand your own decision process well enough to predict its decision in advance, because you'd have to take into account the decision process including the prediction, et cetera, establishing an infinite regress.On Fri, 14 Feb 2003, Eliezer S. Yudkowsky wrote:It *could* do this but it *doesn't* do this. Its control process is such that it follows an iterative trajectory through chaos which is forbidden to arrive at a truthful solution, though it may converge to a stable attractor.This is the heart of the fallacy. Neither a human nor an AIXI can know "that his synchronized other self - whichever one he is - is doing the same". All a human or an AIXI can know is its observations. They can estimate but not know the intentions of other minds.
However, Corbin doesn't need to know absolutely that his other self is synchronized, nor does he need to know his other self's decision in advance. Corbin only needs to establish a probabilistic estimate, good enough to guide his actions, that his other self's decision is correlated with his *after* the fact. (I.e., it's not a halting problem where you need to predict yourself in advance; you only need to know your own decision after the fact.)
AIXI-tl is incapable of doing this for complex cooperative problems because its decision process only models tl-bounded things and AIXI-tl is not *remotely close* to being tl-bounded. Humans can model minds much closer to their own size than AIXI-tl can. Humans can recognize when their policies, not just their actions, are reproduced. We can put ourselves in another human's shoes imperfectly; AIXI-tl can't put itself in another AIXI-tl's shoes to the extent of being able to recognize the actions of an AIXI-tl computed using a process that is inherently 2t^l large. Humans can't recognize their other selves perfectly but the gap in the case of AIXI-tl is enormously greater. (Humans also have a reflective control process on which they can perform inductive and deductive generalizations and jump over a limited class of infinite regresses in decision processes, but that's a separate issue. Suffice it to say that a subprocess which generalizes over its own infinite regress does not obviously suffice for AIXI-tl to generalize over the top-level infinite regress in AIXI-tl's control process.)
Let's say that AIXI-tl takes action A in round 1, action B in round 2, and action C in round 3, and so on up to action Z in round 26. There's no obvious reason for the sequence {A...Z} to be predictable *even approximately* by any of the tl-bounded processes AIXI-tl uses for prediction. Any given action is the result of a tl-bounded policy but the *sequence* of *different* tl-bounded policies was chosen by a t2^l process.
A human in the same situation has a mnemonic record of the sequence of policies used to compute their strategies, and can recognize correlations between the sequence of policies and the other agent's sequence of actions, which can then be confirmed by directing O(other-agent) strategic processing power at the challenge of seeing the problem from the opposite perspective. AIXI-tl is physically incapable of doing this directly and computationally incapable of doing it indirectly. This is not an attack on the computability of intelligence; the human is doing something perfectly computable which AIXI-tl does not do.
--
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
-------
To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
