> Now, it is certainly conceivable that the laws of physics just > happen to be > such that a sufficiently good technology can create a provably > impenetrable > defense in a short time span, using very modest resources.
Agreed, except for the "very modest resources" part. AIXI could potentially accumulate pretty significant resources pretty quickly. > If that happens > to be the case, the runaway AI isn't a problem. But in just about > any other > case we all end up dead, either because wiping out humanity now is far > easier that creating a defense against our distant descendants, or because > the best defensive measures the AI can think of require > engineering projects > that would wipe us out as a side effect. > > Billy Brown Yes, I agree that an AIXI could be very dangerous. I was really just arguing against the statement that it would *definitely* lead to the end of the human race. I can see plausible alternatives, that's all... An interesting related question is: What if AIXI were implemented, not as a standalone AI system hooked up to a reward button, but as a component of another AI system (such as Novamente)? Novamente has a procedure/predicate learning component. The function of this component would be much more effectively served by a module that was able to rapidly search through an infinite space of computer programs and run any one of them rapidly ;_) [Hey, even a fast-running AIXItl would do nicely, we don't even need a real AIXI.] In this case, the same learning algorithm (AIXI) would not lead to the same behaviors. I wonder what would happen though? If the system were not allowed to modify its basic architecture, perhaps it would just act like a very smart Novamente... If it were allowed to modify its basic architecture, then it would quickly become something other than Novamente, but, there's no reason to assume it would create itself to have an AIXI-like goal structure... This exploration makes the (fairly obvious, I guess) point that the problem with AIXI Friendliness-wise is its simplistic goal architecture (the reward function) rather than its learning mechanism. This is what *meant to be saying* when Eliezer first brought up the AIXI/Friendliness issue. But what I actually said was that it wasn't good to have a system with a single fixed reward function ... and this wasn't quite the right way to say it. -- Ben G ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
