Wei Dai wrote: > The AIXI would just contruct some nano-bots to modify the reward-button so > that it's stuck in the down position, plus some defenses to > prevent the reward mechanism from being further modified. It might need to > trick humans initially into allowing it the ability to construct such > nano-bots, but it's certainly a lot easier in the long run to do > this than > to benefit humans for all eternity. And not only is it easier, but this > way he gets the maximum rewards per time unit, which he would not be able > to get any other way. No real evaluator will ever give maximum rewards > since it will always want to leave room for improvement.
I think it's worse than that, actually. The next logical step is to make sure that nothing ever interferes with its control of the reward signal, or does anything else that would turn off AIXI. It will therefore persue the most effective defensive scheme it can come up with, and it has no reason to care about adverse consequences to humans. Now, there is no easy way to predict what strategy it will settle on, but "build a modest bunker and ask to be left alone" surely isn't it. At the very least it needs to become the strongest military power in the world, and stay that way. It might very well decide that exterminating the human race is a safer way of preventing future threats, by ensuring that nothing that could interfere with its operation is ever built. Then it has to make sure no alien civilization ever interferes with the reward button, which is the same problem on a much larger scale. There are lots of approaches it might take to this problem, but most of the obvious ones either wipe out the human race as a side effect or reduce us to the position of ants trying to survive in the AI's defense system. Billy Brown ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
