Re: [agi] unFriendly AIXI

[Eliezer S. Yudkowsky]

Ben Goertzel wrote:

Yeah, you're right, I mis-spoke.  The theorems assume the goal function is
known in advance -- but not known to the system, just known to the entity
defining and estimating the system's intelligence and giving the rewards.

I was implicitly assuming the case in which the goal was encapsulated in a
"goal-definition program" of some sort, which was hooked up to AIXI in
advance; but that is not the only case.

Actually, there's no obvious way you could ever include V in AIXI, at all. V would have to operate as a predicate on internal representations of reality that have no fixed format or pattern. At most you might be able to define a V that operates as a predicate on AIXI's inputs, in which case you can dispense with the separate reward channel. In fact this is formally equivalent to AIXI, since it equates to an AIXI with an input channel I and a reward channel that is deterministically V(I).

It's a very different sort of setup than Novamente, because

1) a Novamente will be allowed to modify its own goals based on its
experience.

Depending on the pattern of inputs and rewards, AIXI will modify its
internal representation of the algorithm which it expects to determine
future rewards.  Would you say that this is roughly analogous to
Novamente's learning of goals based on experience, or is there in your
view a fundamental difference?  And if so, is AIXI formally
superior or in some way inferior to Novamente?

Well, AIXI is superior to any computable algorithm, in a sense.  If you had
the infinite-computing-power hardware that it requires, it would be pretty
damn powerful ;-p  But so would a lot of other approaches!!  Infinite
computing power provides AI's with a lot of axle grease!!

Obviously it is not AIXI's purpose to be implemented. What AIXI defines rather is an abstraction that lets us talk more easily about certain kinds of intelligence. If any AI program we could conceivably want to build is an imperfect approximation of AIXI, that is an interesting property of AIXI. If an AI program we want to build is *superior* to AIXI then that is an *extremely* interesting property.

The reason I asked the question was not to ask whether AIXI is pragmatically better as a design strategy than Novamente. What I was asking you rather is if, looking at AIXI, you see something *missing* that would be present in Novamente. In other words, *if* you had an infinitely powerful computer processor, is there a reason why you would *not* implement AIXI on it, and would instead prefer Novamente, even if it had to run on a plain old cluster?

If the purpose of spontaneous behavior is to provoke learning
experiences,
this behavior is implicit in AIXI as well, though not obviously so.  I'm
actually not sure about this because Hutter doesn't explicitly
discuss it.

Well, you could argue that if Novamente is so good, AIXI will eventually
figure out how to emulate Novamente, since Novamente is just one of the many
programs in the space it searches!!

I am really not very interested in comparing AIXI to Novamente, because they
are not comparable: AIXI assumes infinite computing power and Novamente does
not.

We aren't comparing AIXI's design to Novamente's design so much as we're comparing AIXI's *kind of intelligence* to Novamente's *kind of intelligence*. Does Novamente have something AIXI is missing? Or does AIXI have strictly more intelligence than Novamente?

Actually, given the context of Friendliness, what we're interested in is not so much "intelligence" as "interaction with humans"; under this view, for example, giving humans a superintelligently deduced cancer cure is just one way of interacting with humans. Looking at AIXI and Novamente, do you see any way that Novamente interacts with humans in a way that AIXI cannot?

AIXItl, on the other hand, is a finite-computing-power program.  In
principle it can demonstrate spontaneous behaviors, but in practice, I think
it will not demonstrate many interesting spontaneous behaviors.  Because it
will spend all its time dumbly searching through a huge space of useless
programs!!

Also, not all of Novamente's spontaneous behaviors are even implicitly
goal-directed.  Novamente is a goal-oriented but not 100% goal-directed
system, which is one major difference from AIXI and AIXItl.

I agree that it is a major difference; does it mean that Novamente can interact with humans in useful or morally relevant ways of which AIXI is incapable?

 But it looks to me like AIXI, under its formal definition, emergently
exhibits "curiosity" wherever there are, for example, two equiprobable
models of reality which determine different rewards and can be
distinguished by some test.  What we interpret as "spontaneous" behavior
would then emerge from a horrendously uncomputable exploration of all
possible realities to find tests which are ultimately likely to result in
distinguishing data, but in ways which are not at all obvious to
any human
observer.  Would it be fair to say that AIXI's "spontaneous behavior" is
formally superior to Novamente's spontaneous behavior?

Yeah, AIXI is formally superior if one distinguishes any fixed goal and asks
whether Novamente or AIXI can better achieve that goal.  But so what?  AIXI
assumes you have infinite computing power!!  If I assumed infinite computing
power, I would have designed Novamente rather differently... and much  more
simply...

As a side point, I'm not sure the best way to compare systems is to assume a
fixed formal goal and ask who can achieve it better.  This is the way
Hutter's theorems do the comparison, but...

>

But no matter HOW you want to compare systems, if you let me assume infinite
computing power, I can design a system that will outperform a Novamente ...

I'm not trying to compare amounts of computing power but fundamental *kinds* of intelligence, as is AIXI's purpose as a formal definition.

Again, AIXI as a formal system has no goal definition.  [Note:  I may be
wrong about this; Ben Goertzel and I seem to have acquired different
models of AIXI and it is very possible that mine is the wrong
one.]

Well, the purpose of AIXI and AIXItl is to have theorems proved about them.

These theorems are of the form: Given a fixed reward function (a fixed
goal),

That's an interesting paraphrase.

* AIXI is  maximally intelligent at achieving the goal
* AIXItl is as intelligent as any other finite-resource program at achieving
the goal, so long as AIXItl is given C more computing power than the other
program, where C is very big

But you are right that AIXI and AIXItl could also be run without a fixed
reward function /goal.

No, that's not what I said. I said AIXI could be run with only a reward channel and no separate input channel, or with a reward predicate that is a deterministic function of the input channel. You cannot run AIXI in the absence of a reward channel.

> In that case you cannot prove any of Hutter's

theorems about them.  And if you can't prove theorems about them then they
are nothing more than useless abstractions.  Since AIXI can never be
implemented and AIXItl is so inefficient it could never do anything useful
in practice.

But they are very useful tools for talking about fundamental kinds of intelligence.

If the humans see that AIXI seems to be dangerously inclined toward just
proving math theorems, they might decide to press the reward button when
AIXI provides cures for cancer, or otherwise helps people.  AIXI would
then modify its combined reality-and-reward representation accordingly to
embrace the new simplest explanation that accounted for *all* the data,
i.e., its reward function would then have to account for mathematical
theorems *and* cancer cures *and* any other kind of help that humans had,
in the past, pressed the reward button for.

Would you say this is roughly analogous to the kind of learning
you intend
Novamente to perform?  Or perhaps even an ideal form of such learning?

Well, sure ... it's *roughly analogous*, in the sense that it's experiential
reinforcement learning, sure.

Is it roughly analogous, but not really analogous, in the sense that Novamente can do something AIXI can't?

Self-modification in any form completely breaks Hutter's definition, and
you no longer have an AIXI any more.  The question is whether Hutter's
adaptive reality-and-reward algorithm encapsulates the behaviors you
want... do you think it does?

Not really.  There is certainly a significant similarity between Hutter's
stuff and the foundations of Novamente, but there are significant
differences too.  To sort out the exact relationship would take me more than
a few minutes' thought.

There are indeed major differences in the foundations. Is there something useful or important that Novamente does, given its foundations, that you could not do if you had a physically realized infinitely powerful computer running Hutter's stuff?

One  major difference, as I mentioned above, is that Hutter's systems are
purely concerned with goal-satisfaction, whereas Novamente is not entirely
driven by goal-satisfaction.

Is this reflected in a useful or important behavior of Novamente, in its intelligence or the way it interacts with humans, that is not possible to AIXI?

--
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence

-------
To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]