Hi, > > The harmfulness or benevolence of an AIXI system is therefore > closely tied > > to the definition of the goal that is given to the system in advance. > > Under AIXI the goal is not given to the system in advance; rather, the > system learns the humans' goal pattern through Solomonoff > induction on the > reward inputs.
Yeah, you're right, I mis-spoke. The theorems assume the goal function is known in advance -- but not known to the system, just known to the entity defining and estimating the system's intelligence and giving the rewards. I was implicitly assuming the case in which the goal was encapsulated in a "goal-definition program" of some sort, which was hooked up to AIXI in advance; but that is not the only case. > Technically, in fact, it would be entirely feasible to > give AIXI *only* reward inputs, although in this case it might require a > long time for AIXI to accumulate enough data to constrain the > Solomonoff-induced representation to a sufficiently detailed model of > reality that it could successfully initiate complex actions. The utility > of the non-reward input is that it provides additional data, causally > related to the mechanisms producing the reward input, upon which > Solomonoff induction can also be performed. Agreed? Yep > > It's a very different sort of setup than Novamente, because > > > > 1) a Novamente will be allowed to modify its own goals based on its > > experience. > > Depending on the pattern of inputs and rewards, AIXI will modify its > internal representation of the algorithm which it expects to determine > future rewards. Would you say that this is roughly analogous to > Novamente's learning of goals based on experience, or is there in your > view a fundamental difference? And if so, is AIXI formally > superior or in > some way inferior to Novamente? Well, AIXI is superior to any computable algorithm, in a sense. If you had the infinite-computing-power hardware that it requires, it would be pretty damn powerful ;-p But so would a lot of other approaches!! Infinite computing power provides AI's with a lot of axle grease!! AIXItl is a different story. It's computable, and is vastly less useful than Novamente. It's a ridiculous algorithm really, since at each time step it searches an infeasibly large space of possible programs. It's useful purely for theoretical purposes. One big difference is that AIXItl modifies its whole operating program at each time step, whereas Novamente modifies itself incrementally. But of course it's also possible for AIXItl to modify itself incrementally, if this is what its program search comes up with... Another big difference is that AIXItl has a rigid separation between its operating program and its "program learning" component, whereas in Novamente these two are more blended together. > > 2) a Novamente will be capable of spontaneous behavior as well > as explicitly > > goal-directed behavior > > If the purpose of spontaneous behavior is to provoke learning > experiences, > this behavior is implicit in AIXI as well, though not obviously so. I'm > actually not sure about this because Hutter doesn't explicitly > discuss it. Well, you could argue that if Novamente is so good, AIXI will eventually figure out how to emulate Novamente, since Novamente is just one of the many programs in the space it searches!! I am really not very interested in comparing AIXI to Novamente, because they are not comparable: AIXI assumes infinite computing power and Novamente does not. AIXItl, on the other hand, is a finite-computing-power program. In principle it can demonstrate spontaneous behaviors, but in practice, I think it will not demonstrate many interesting spontaneous behaviors. Because it will spend all its time dumbly searching through a huge space of useless programs!! Also, not all of Novamente's spontaneous behaviors are even implicitly goal-directed. Novamente is a goal-oriented but not 100% goal-directed system, which is one major difference from AIXI and AIXItl. > But it looks to me like AIXI, under its formal definition, emergently > exhibits "curiosity" wherever there are, for example, two equiprobable > models of reality which determine different rewards and can be > distinguished by some test. What we interpret as "spontaneous" behavior > would then emerge from a horrendously uncomputable exploration of all > possible realities to find tests which are ultimately likely to result in > distinguishing data, but in ways which are not at all obvious to > any human > observer. Would it be fair to say that AIXI's "spontaneous behavior" is > formally superior to Novamente's spontaneous behavior? Yeah, AIXI is formally superior if one distinguishes any fixed goal and asks whether Novamente or AIXI can better achieve that goal. But so what? AIXI assumes you have infinite computing power!! If I assumed infinite computing power, I would have designed Novamente rather differently... and much more simply... As a side point, I'm not sure the best way to compare systems is to assume a fixed formal goal and ask who can achieve it better. This is the way Hutter's theorems do the comparison, but... But no matter HOW you want to compare systems, if you let me assume infinite computing power, I can design a system that will outperform a Novamente ... > > The Friendliness and other qualities of such a system seem to > me to depend > > heavily on the goal chosen. > > Again, AIXI as a formal system has no goal definition. [Note: I may be > wrong about this; Ben Goertzel and I seem to have acquired different > models of AIXI and it is very possible that mine is the wrong > one.] Well, the purpose of AIXI and AIXItl is to have theorems proved about them. These theorems are of the form: Given a fixed reward function (a fixed goal), * AIXI is maximally intelligent at achieving the goal * AIXItl is as intelligent as any other finite-resource program at achieving the goal, so long as AIXItl is given C more computing power than the other program, where C is very big But you are right that AIXI and AIXItl could also be run without a fixed reward function /goal. In that case you cannot prove any of Hutter's theorems about them. And if you can't prove theorems about them then they are nothing more than useless abstractions. Since AIXI can never be implemented and AIXItl is so inefficient it could never do anything useful in practice. > If the humans see that AIXI seems to be dangerously inclined toward just > proving math theorems, they might decide to press the reward button when > AIXI provides cures for cancer, or otherwise helps people. AIXI would > then modify its combined reality-and-reward representation accordingly to > embrace the new simplest explanation that accounted for *all* the data, > i.e., its reward function would then have to account for mathematical > theorems *and* cancer cures *and* any other kind of help that humans had, > in the past, pressed the reward button for. > > Would you say this is roughly analogous to the kind of learning > you intend > Novamente to perform? Or perhaps even an ideal form of such learning? Well, sure ... it's *roughly analogous*, in the sense that it's experiential reinforcement learning, sure. > > What if one supplied AIXI with a goal that explicitly involved > modifying its > > own goal, though? > > Self-modification in any form completely breaks Hutter's definition, and > you no longer have an AIXI any more. The question is whether Hutter's > adaptive reality-and-reward algorithm encapsulates the behaviors you > want... do you think it does? Not really. There is certainly a significant similarity between Hutter's stuff and the foundations of Novamente, but there are significant differences too. To sort out the exact relationship would take me more than a few minutes' thought. One major difference, as I mentioned above, is that Hutter's systems are purely concerned with goal-satisfaction, whereas Novamente is not entirely driven by goal-satisfaction. -- Ben ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
