On Wed, Feb 19, 2003 at 11:02:31AM -0500, Ben Goertzel wrote: > I'm not sure why an AIXI, rewarded for pleasing humans, would learn an > operating program leading it to hurt or annihilate humans, though. > > It might learn a program involving actually doing beneficial acts for humans > > Or, it might learn a program that just tells humans what they want to hear, > using its superhuman intelligence to trick humans into thinking that hearing > its soothing words is better than having actual beneficial acts done. > > I'm not sure why you think the latter is more likely than the former. My > guess is that the former is more likely. It may require a simpler program > to please humans by benefiting them, than to please them by tricking them > into thinking they're being benefited....
The AIXI would just contruct some nano-bots to modify the reward-button so that it's stuck in the down position, plus some defenses to prevent the reward mechanism from being further modified. It might need to trick humans initially into allowing it the ability to construct such nano-bots, but it's certainly a lot easier in the long run to do this than to benefit humans for all eternity. And not only is it easier, but this way he gets the maximum rewards per time unit, which he would not be able to get any other way. No real evaluator will ever give maximum rewards since it will always want to leave room for improvement. > Furthermore, my stated intention is NOT to rely on my prior intuitions to > assess the safety of my AGI system. I don't think that anyone's prior > intuitions about AI safety are worth all that much, where a complex system > like Novamente is concerned. Rather, I think that once Novamente is a bit > further along -- at the "learning baby" rather than "partly implemented > baby" stage -- we will do experimentation that will give us the empirical > knowledge needed to form serious opinions about safety (Friendliness). What kinds of experimentations do you plan to do? Please give some specific examples. ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
