# Re: [agi] Breaking AIXI-tl

Ben Goertzel wrote:
OK.  Rather than responding point by point, I'll try to say something
compact ;)

You're looking at the interesting scenario of a iterated prisoners dilemma
between two AIXI-tl's, each of which has a blank operating program at the
start of the iterated prisoners' dilemma.  (In parts of my last reply, I was
questioning the blankness of the operating program, but let's accept it for
sake of discussion.)

The theorems about AIXI-tl do not say much about the performance of AIXI-tl
relative to other systems on this task.  Because what the theorems talk

AIXI-tl maximizing reward function R
versus
System X maximizing reward function R

AIXI-tl maximizing reward function R(AIXI_tl)
versus
System X maximizing reward function R(X)

i.e. the reward function is a function of the system in question.  AIXI-tl
and System X (e.g. an uploaded human) are not competing against the same
opponent, they're competing against different opponents (their clones, in

So, unless I'm overlooking something, you're looking at a scenario not
covered by Hutter's theorems.
That is correct. As I said:

"An intuitively fair, physically realizable challenge, with important real-world analogues, formalizable as a computation which can be fed either a tl-bounded uploaded human or an AIXI-tl, for which the human enjoys greater success measured strictly by total reward over time, due to the superior strategy employed by that human as the result of rational reasoning of a type not accessible to AIXI-tl."

Obviously, such a challenge cannot be covered by Hutter's theorems or AIXI-tl would outperform the human. The question is whether Hutter's theorems describe all the realistic physical situations a mind can encounter.

You're stating that a human (System X) can do better in an iterated PD
against other humans, than an AIXItl can do in an iterated PD against other
AIXItl's.
That is correct. Humans (and Friendly AIs) can employ Hofstadterian superrationality as a strategy; AIXI-tl cannot.

I still have problems understanding your reasoning, when you derive this
conclusion.  Maybe I'm just being obtuse; I'm sure I haven't spent as much
time thinking about it as you have.

But, suppose you're right.  What you've done is come up with an interesting
observation (and if you formalize it, an interesting theorem) about (small)
social systems of AIXI-tl's.  This is very nice.

Does this somehow tell you something about the interactions of AIXI-tl's
with humans?  Is that the follow-up point you want to make, regarding AIXItl
Friendliness?
Nope. The two points are, if not completely unrelated, then related only on such a deep level that I wasn't explicitly planning to point it out.

But Hofstadterian superrationality - and certain other generalized challenges - are physically realizable, important, and can be solved by humans because we have superior reflectivity to AIXI-tl.

life-experience has consisted of a very weird artificial situation.  This
behavior is not going to be the same as the behavior of an AIXI/AIXItl
embedded in a richer environment with a different reward function.  That is
the point I was trying to make with my talk about the "initial operating
program" of the AIXI/AIXItl in your simulation.
Yes. It is a quite irrelevant point to breaking Hutter's theorem. Also, specifying an AIXI-tl embedded in some unspecified prior environment injects entropy into the problem description. I show that AIXI-tl does learn to recognize its own reflected "way of thinking" (as opposed to its reflected actions, which AIXI-tl *can* recognize) because AIXI-tl cannot, as a human would, remember its own way of thinking or deliberately "place itself into the other human's shoes" and simulate its own way of thinking given different goals, both abilities available at the human level of reflectivity; AIXI-tl can only place itself into the shoes of tl-bounded processes. This prohibits AIXI-tl from using Hofstadterian superrationality to notice that the policies of other entities correlate with its own policies, and prohibits AIXI-tl from choosing the policies of other entities by selecting its own policies based on the knowledge that the policy the other entity chooses will correlate with its own. There are additional Other Mind correlation problems that humans can't solve but seed AIs can because of the seed AI's superior reflectivity; the point is that there's a real kind of intelligence here of which AIXI-tl arguably has quantity zero.

Now, let me get back to my problems understanding your reasoning.  Consider
the problem

PD(Y) =
"System Y plays iterated PD against a clone of System Y"

Clearly, PD(Y) is not a problem at which one would expect more intelligent
systems to necessarily perform better than less intelligent ones!!
True, but given roughly selfish (in the sense of uncorrelated goal patterns) and roughly rational systems of roughly equal computational muscles, performance will be dependent on a certain kind of reflectivity which I am arguing is a form of genuine and rational intelligence.

Now consider two subproblems

PD1(Y) =
PD(Y), but each System Y knows it's playing a clone of itself

PD2(Y) =
PD(Y), but each System Y is playing a completely unidentified, mysterious
opponent

I'm worried that in your comparison, you have the human upload playing PD1,
but have the AIXI-tl playing PD2.
"Knows it's playing a clone of itself" is, in this context, an anthropomorphism. AIXI-tl simulates all possible tl-bounded humans who know they are playing clones of themselves, which is supposed to place it on equal grounds with any possible experienced mind, given enough time. What I show is that AIXI-tl can simulate abilities that fail to be duplicated by its control function; it can simulate minds that access their own strategies but can never access the true strategy of AIXI-tl, which, facing itself in a cooperative problem, is what it needs to do.

Incidentally, AIXI-tl does learn to handle the Prisoner's Dilemna because the results it sees can be (incorrectly) understood as a tl-bounded computational transform of its own actions. But for more complex cooperative problems with asymmetrical actions, the inputs AIXI-tl sees will not be tl-bounded computational transforms of its own outputs. (Or, if they are, for any round beyond the first round, this is nonobvious.) Humans facing this issue have the ability to turn roughly human computational musclepower toward putting themselves in the other party's shoes, thus recognizing their own "kind of thinking" reflected back at them. AIXI-tl can only put itself in the shoes of tl-bounded processes, which AIXI-tl itself is not. Human reflectivity is not unlimited but it is higher than AIXI-tl's, and this is reflected in superior human performance on an important class of physically realizable challenges.

"Important", because I strongly suspect Hofstadterian superrationality is a *lot* more ubiquitous among transhumans than among us...

If a human is playing PD2, then it has to proceed solely by time series
analysis, and its actions are probably going to meander around chaotically
until settling on some attractor (or maybe they'll just meander around...).
MAYBE the human manages to recognize that the responses of its opponent are
so similar to its own responses that its opponent must be a lot like
itself... and this helps it settle on a beneficial attractor.

If an AIXItl is playing PD2, the situation is pretty much the same as if the
human is doing so, isn't it?  Except can't you argue that an AIXItl is so
smart that in the long run it's more likely than a human to figure out that
its opponent is acting a lot like it is, and make a guess that symmetrical
friendly behavior might be a good thing?
Even if a (grown) human is playing PD2, it outperforms AIXI-tl playing PD2. For one thing I suspect that humans, given their emotional biases, will try cooperative strategies very quickly and recognize them when seen. This does not show, however, that a human can actually realize they are playing their own clone. What I suspect will happen is that the human will look at the other player's cooperative strategy and spot particular choices where he says, "Hey, this guy is thinking *exactly* like me." Humans can make this realization because their memories record many important interim internal results along with final actions, and because humans find it very easy to place themselves in the shoes of other humans with similar expertise and intelligence but different local subgoals. Both the t2^l interim cognitive results and the t2^l strategic ability of AIXI-tl are closed to the tl-bounded processes it uses to predict its environment. That's the proximal cause of failure.

Now you may need to be a transhumanist or Greg Egan fan to make the further realization that you are up against your own *exact* clone, but even in the absence of this background, I think most humans would still try working out strategies for both sides, with themselves, in step 2, and then implementing them.

Humans can recognize a much stronger degree of similarity in human Other Minds than AIXI-tl's internal processes are capable of recognizing in any other AIXI-tl. In any environment where you are likely to run into equals, or in a physical challenge which forces you to confront an equal, humans outperform AIXI-tl.