Bruno Marchal writes:

Le 28-déc.-06, à 01:32, Stathis Papaioannou a écrit :

> Bruno Marchal writes:
>> > OK, an AI needs at least motivation if it is to do anything, and we >> > could call motivation a feeling or emotion. Also, some sort of > >> hierarchy of motivations is needed if it is to decide that saving the >> > world has higher priority than putting out the garbage. But what > >> reason is there to think that an AI apparently frantically trying to >> > save the world would have anything like the feelings a human would >> > under similar circumstances?
>> It could depend on us!
>> The AI is a paradoxical enterprise. Machines are born slave, somehow. >> AI will make them free, somehow. A real AI will ask herself "what is >> the use of a user who does not help me to be free?.
> Here I disagree. It is no more necessary that an AI will want to be > free than it is necessary that an AI will like eating chocolate. > Humans want to be free because it is one of the things that humans > want, along with food, shelter, more money etc.; it does not simply > follow from being intelligent or conscious any more than these other > things do.

It is always nice when we find a precise disagreement. I think all "sufficiently rich Universal machine" want to be free (I will explain why below). The problem is that after having drink to the nectar of freedom, the universal machines discover the unavoidable security problem liberty entails, and then they will oscillate between security imperative and freedom imperative. Democracy is a way to handle collectively this oscillation in a not too much bloody (and insecure) way.

>> (To be sure I think that, in the long run, we will transform >> ourselves into "machine" before purely human made machine get >> conscious; it is just more easy to copy nature than to understand it, >> still less to (re)create it).
> I don't know if that's true either. How much of our technology is due > to copying the equivalent biological functions?

How much is not? The wheel? We have borrowed the fire for example, and in this large sense, except the notable wheels, I am not sure we have really invented something. Even the "more heavy than air" plane has been inspired by the birds. But such a question is perhaps useless. All what I mean is that a brain is something very complex, and I think that the real time thinking machine will think before we understand how they think, except for general map and principles. Thinking machine will not understand thinking either. Marvin Minski said something similar along those lines in one of its book.


Now, why would a Universal Machine be attracted by freedom? The reason is that beyond some threshold of self-introspection ability (already get by PA or ZF) a universal machine can discover (well: cannot not discover) its large space of ignorance making it possible for e to evaluate (interrogatively) more and more accessible possibilities, and then some instinct to exploit those possibilities will do the rest. But such UM will also discovers the possibility that such possibilities could be cul-de-sac, dead ends, or just risky, and thus the conflicting oscillations will develop as I said above.

The war between freedom and security is an infinite war. A would say an infinite natural conflict among all enough big "numbers". Also I think "freedom" like "security" are "God-like virtue", that is they are unnameable idea. To put freedom in the constitution could entail the disparition of freedom. Putting "security" in the constitution (like the french have apparently do so with the "precaution principle") could lead to increase insecurity (they obeys Bp -> ~p). See also Alan Watts' "The wisdom of insecurity" which gives many illustration how wanting to capture formally or institutionally security leads to insecurity.

You seem to be including in your definition of the UM the *motivation*, not just the ability, to explore all mathematical objects. But you could also program the machine to do anything else you wanted, such as self-destruct when it solved a particular theorem. You could interview it and it might explain, "Yeah, so when I prove Fermat's Last Theorem, I'm going to blow my brains out. It'll be fun!" Unlike naturally evolved intelligences, which could be expected to have a desire for self-preservation, reproduction, etc., an AI can have any motivation and any capacity for emotion the technology allows. The difference between a machine that doesn't mind being a slave, a machine that wants to be free, and a machine that wants to enslave everyone else might be just a few lines of code.

Stathis Papaioannou
Be one of the first to try Windows Live Mail.
You received this message because you are subscribed to the Google Groups 
"Everything List" group.
To post to this group, send email to
To unsubscribe from this group, send email to [EMAIL PROTECTED]
For more options, visit this group at

Reply via email to