Harshad RJ wrote:
On Feb 3, 2008 10:22 PM, Richard Loosemore <[EMAIL PROTECTED]
<mailto:[EMAIL PROTECTED]>> wrote:
Harshad RJ wrote:
> I read the conversation from the start and believe that Matt's
> argument is correct.
Did you mean to send this only to me? It looks as though you mean it
for the list. I will send this reply back to you personally, but let me
know if you prefer it to be copied to the AGI list.
Richard, thanks for replying. I did want to send it to the list.. and
your email address (as it turns out) was listed on the forum for
replying to the list.
> There is a difference between intelligence and motive which Richard
> seems to be ignoring. A brilliant instance of intelligence could
still
> be subservient to a malicious or ignorant motive, and I think that is
> the crux of Matt's argument.
With respect, I was not at all ignoring this point: this is a
misunderstanding that occurs very frequently, and I thought that I
covered it on this occasion (my apologies if I forgot to do so..... I
have had to combat this point on so many previous occasions that I may
have overlooked yet another repeat).
The crucial words are "... could still be subservient to a malicious or
ignorant motive."
The implication behind these words is that, somehow, the "motive" of
this intelligence could arise after the intelligence, as a completely
independent thing over which we had no control. We are so used to this
pattern in the human case (we can make babies, but we cannot stop the
babies from growing up to be dictators, if that is the way they happen
to go).
This implication is just plain wrong.
I don't believe so, though your next statement..
If you build an artificial
intelligence, you MUST choose how it is motivated before you can even
switch it on.
... might be true. Yes, a motivation of some form could be coded into
the system, but the paucity of expression in the level at which it is
coded, may still allow for "unintended" motivations to emerge out.
Say, for example, the motivation is coded in a form similar to current
biological systems. The AGI system is motivated to keep itself happy,
and it is happy when it has sufficient electrical energy at its disposal
AND when the pheromones from nearby humans are all screaming "positive".
It is easy to see how this kind of motivation could cause unintended
results. The AGI system could do dramatic things like taking over a
nuclear power station and manufacturing its own pheromone supply from a
chemical plant. Or it could do more subtle things like, manipulating
government policies to ensure that the above happens!
Even allowing for a higher level of coding for motivation, like those
Asimov's Robot rules (#1 : Though shall not harm any human), it is very
easy for the system to go out of hand, since such codings are ambiguous.
Should "stem cell research" be allowed for example? It might harm some
embryos but help more number of adults. "Should prostitution be
legalised?" It might harm the human gene pool in some vague way, or
might even harm some specific individuals, but it also allows the
victims themselves to earn some money and survive longer.
So, yes, motivation might be coded, but an AGI system would eventually
need to have the *capability* to deduce its own motivation, and that
emergent motivation could be malicious/ignorant.
I quote the rest of the message, only for the benefit of the list.
Otherwise, my case rests here.
Stepping back for a moment, I think the problem that tends to occur in
discussions of AGI motivation is that the technical aspects get
overlooked when we go looking for nightmare scenarios. What this means,
for me, is that when I reply to a suggestion such as the one you give
above, my response is not "That kind of AGI, and AGI behavioral problem,
is completely unimaginable", but instead what I have to say is "That
kind of AGI would not actually BE an AGI at all, because, for technical
reasons, you would never be able to get such a thing to be intelligent
in the first place".
There is a subtle difference between these two, but what I find is that
most people mistakenly believe that I am making the first kind of
reponse instead of the second.
So, to deal with your suggestion in detail.
When I say that some kind of motivation MUST be built into the system, I
am pretty much uttering a truism: an AGI without any kind of
motivational system is like a swimmer with no muscles. It has to be
driven to do something, so no drives mean no activity.
Putting that to one side, then, what you propose is an AGI with an
extremely simple motivational system: seek electricity and high human
pheromonal output.
I don't suggest that this is unimaginable (it is!), but what I suggest
is that you implicitly assume a lot of stuff that, almost certainly,
will never happen.
You assume that the system does not go through a learning phase
(childhood) during which it acquires its knowledge by itself. Why do
you assume this? Because an AGI that was motivated only to seek
electricity and pheromones is going to be as curious, as active, as
knowledge seeking, as exploratory (etc etc etc) as a moth that has been
preprogrammed to go towards bright lights. It will never learn aything
by itself because you left out the [curiosity] motivation (and a lot
else besides!). All you can hope for is that human programmers insert
all of its knowledge by hand .... and I mean all of its knowledge,
because it will have no motivation to acquire more (by your assumption
that it only seeks two very narrow goals).
Most AI people agree that a full AGI cannot be built if all its
knowledge has to be compiled by hand. It must be able to learn by itself.
But when we try to get an AGI to have the kind of structured behavior
necessary to learn by itself, we discover ..... what? That you cannot
have that kind of structured exploratory behavior without also having an
extremely sophisticated motivation system.
In other words you cannot have your cake and eat it too: you cannot
assume that this hypothetical AGI is (a) completely able to build its
own understanding of the world, right up to the human level and beyond,
while also being (b) driven by an extremely dumb motivation system that
makes the AGI seek only a couple of simple goals.
If an AGI could ever be built with only such simplistic goals then,
sure, that kind of motivational system would be ridiculously unstable,
and liable to blow up in your face. But you simply cannot do that: try
it and you will see. The thing may have isolated bursts of rationality
within some limited domain, if you go to the trouble hand coding it in
that domain, but if you try to persuade such a simply-motivated system
to actually grow and develop into an adult that would be your equal, you
will spend your whole life in absolute frustration as it just sits there
and looks up at you with all the intellectual depth of a Sphex Wasp or a
stick insect.
I believe that if you go and analyze every single science-fiction robot
nightmare scenario that was ever conceived, you will see the same pair
of unworkable assumptions The hypthetical AGI is supposed to be so
smart that it can understand all the incredible depths of complexity
that are involved in building a motivational system (because, by
assumption, it knows enough about its own design to be able to make
coherent modifications), and yet at the same time it is supposed to be
so simple underneath that it cannot understand anything except for a
primal desire to get electricity and a pheromone hit!
So, these science fiction AGIs would be able to take part in this
discussion that you and I are having right now, and go into such
incredible technical detail about the mechanics of building a
motivational system that neither of us could follow ....... but this
same AGI is also supposed to be following a simple explicit-rule-based
motivational system in which rule #1 is "Though shall not harm any
human", where this statement has to be interpreted?!
Now, the exact details of the line of argument I propose here are not
set in stone (there is much work to be done to see if the argument
really does play out, the way it seems to), but right now this
particular approach to AGI motivation is the only one that does not have
fundamental contradictions built into it.
There is more to say, but not in an already long post.
Richard Loosemore
Nature does this in our case (and nature is very
insistent that it wants its creations to have plenty of selfishness and
aggressiveness built into them, because selfish and aggressive species
survive), but nature does it so quietly that we sometimes think that all
she does is build an intelligence, then leave the motivations to grow
however they will. But what nature does quietly, we have to do
explicitly.
My argument was (at the beginning of the debate with Matt, I believe)
that, for a variety of reasons, the first AGI will be built with
peaceful motivations. Seems hard to believe, but for various technical
reasons I think we can make a very powerful case that this is exactly
what will happen. After that, every other AGI will be the same way
(again, there is an argument behind that). Furthermore, there will not
be any "evolutionary" pressures going on, so we will not find that (say)
the first few million AGIs are built with perfect motivations, and then
some rogue ones start to develop.
So, when you say that "A brilliant instance of intelligence could still
be subservient to a malicious or ignorant motive" you are saying
something equivalent to "Toyota could build a car with a big red button
on the roof, and whenever anyone slapped the button a nuclear weapon
would go off in the car's trunk." Technically, yes, I am sure Toyota
could find a way to do this! But oing this kind of thing is not an
automatic consequence (or even a remotely probably consequence) of a
company becoming a car manufacturer with enough resources to do such a
thing. Similarly, having malevolent motives is not an automatic
consequence (or even a remotely probably consequence) of a system being
an intelligence with enough resources to do such a thing.
> There are two possibilities:
> 1. The AGI in question could have been programmed to choose it's own
> motive, in which case, the AGI may very well choose a motive that is
> malicious to humanity.
As I just explained, it must already have a motive before it can do any
choosing. That first motive will determine whether it even considers
the possibility of being malevolent.
Also, to be smart enough to redesign its own motivations, it has to be
very smart indeed, having a profound understanding of different motives
and the consequences of tampering with its own motivations. Under those
circumstances I believe it will simply take steps to reinforce its
bening motives to make sure there is no chance of accidental deviation.
>
> 2. The AGI could be programmed to satisfy a motive specified by the
> creator, in which case the maliciousness (or ignorance) of the
creator
> is what should be considered in this discussion. And since the
> creators (that is we, humans) are known to be both ignorant and
> capable of malice, the system is highly susceptible to "an AGI
> singularity that leads to humanity being rendered redundant".
Again, this argument is similar to the "Since Toyota is staffed by
humans, who are are known to be both ignorant and capable of malice, the
Toyota company is highly susceptible to a scenario in which they create
a car with a nuclear weapon in the trunk". Being capable of doing it is
not the same as actually doing it.
All that needs to happen is for the first AGI to be built with benign
motives, and then all the possibilities for malicious systems go down to
(virtually) zero within a few hours. (There is a long list of
supporting arguments for this, but that will have to be a separate
discussion).
To argue that there is any possibility of a malevolent AGI emerging, you
have to attack the hinge point of this argument - you have to give a
convincing reason why the first AGI (a) is going to be created by
someone who has genocidal motives, (b) is going to be stable AND
malevolent: something that is probably very hard to achieve.
I believe, overall, that when you look into the details of these
scenarios, what you find is that all the Bad Outcome scenarios involve
assumptions that, when you examine them carefully, are wildly
improbable, or based on pure supposition about technical topics that we
have not resolved yet.
I think we should definitely have structured debate about this.
Richard Loosemore.
> -HRJ
>
> ------------------------------
>
>
> Richard Loosemore wrote:
>
> Your comments below are unfounded, and all the worse for being so
> poisonously phrased. If you read the conversation from the beginning
> you will discover why: Matt initially suggested the idea that an AGI
> might be asked to develop a virus of maximum potential, for purposes
> of testing a security system, and that it might respond by inserting
> an entire AGI system into the virus, since this would give the virus
> its maximum potential. The thrust of my reply was that his entire
idea
> of Matt's made no sense, since the AGI could not be a "general"
> intelligence if it could not see the full implications of the
request.
>
>
> Please feel free to accuse me of gross breaches of rhetorical
> etiquette, but if you do, please make sure first that I really have
> committed the crimes. ;-)
>
>
>
> Richard Loosemore
>
>
------------------------------------------------------------------------
*agi* | Archives <http://www.listbox.com/member/archive/303/=now>
<http://www.listbox.com/member/archive/rss/303/> | Modify
<http://www.listbox.com/member/?&;>
Your Subscription [Powered by Listbox] <http://www.listbox.com>
-------------------------------------------
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription:
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com
