Hank Conn wrote:
> Yes, you are exactly right. The question is which of my
assumption are
> unrealistic?
Well, you could start with the idea that the AI has "... a strong goal
that directs its behavior to aggressively take advantage of these
means...". It depends what you mean by "goal" (an item on the task
stack or a motivational drive? They are different things) and this
begs
a question about who the idiot was that designed it so that it pursue
this kind of aggressive behavior rather than some other!
A goal is a problem you want to solve in some environment. The "idiot"
who designed it may program its goal to be, say, making paperclips.
Then, after some thought and RSI, the AI decides converting the entire
planet into a computronium in order to figure out how to maximize the
number of paper clips in the Universe will satisfy this goal quite
optimally. Anybody could program it with any goal in mind, and RSI
happens to be a very useful process for accomplishing many complex goals.
There is *so* much packed into your statement that it is difficult to go
into it in detail.
Just to start with, you would need to cross compare the above statement
with the account I gave recently of how a system should be built with a
motivational system based on large numbers of diffuse constraints. Your
description is one particular, rather dangerous, design for an AI - it
is not an inevitable design.
I'm not asserting any specific AI design. And I don't see how
a motivational system based on "large numbers of diffuse constrains"
inherently prohibits RSI, or really has any relevance to this. "A
motivation system based on large numbers of diffuse constraints" does
not, by itself, solve the problem- if the particular constraints do not
form a congruent mapping to the concerns of humanity, regardless of
their number or level of diffuseness, then we are likely facing an
Unfriendly outcome of the Singularity, at some point in the future.
The point I am heading towards, in all of this, is that we need to
unpack some of these ideas in great detail in order to come to sensible
conclusions.
I think the best way would be in a full length paper, although I did
talk about some of that detail in my recent lengthy post on motivational
systems.
Let me try to bring out just one point, so you can see where I am going
when I suggest it needs much more detail. In the above, you really are
asserting one specific AI design, because you talk about the goal stack
as if this could be so simple that the programmer would be able to
insert the "make paperclips" goal and the machine would go right ahead
and do that. That type of AI design is very, very different from the
Motivational System AI that I discussed before (the one with the diffuse
set of constraints driving it).
Here is one of many differences between the two approaches.
The goal-stack AI might very well turn out simply not to be a workable
design at all! I really do mean that: it won't become intelligent
enough to be a threat. Specifically, we may find that the kind of
system that drives itself using only a goal stack never makes it up to
full human level intelligence because it simply cannot do the kind of
general, broad-spectrum learning that a Motivational System AI would do.
Why? Many reasons, but one is that the system could never learn
autonomously from a low level of knowledge *because* it is using goals
that are articulated using the system's own knowledge base. Put simply,
when the system is in its child phase it cannot have the goal "acquire
new knowledge" because it cannot understand the meaning of the words
"acquire" or "new" or "knowledge"! It isn't due to learn those words
until it becomes more mature (develops more mature concepts), so how can
it put "acquire new knowledge" on its goal stack and then unpack that
goal into subgoals, etc?
Try the same question with any goal that the system might have when it
is in its infancy, and you'll see what I mean. The whole concept of a
system driven only by a goal stack with statements that resolve on its
knowledge base is that it needs to be already very intelligent before it
can use them.
I have never seen this idea discussed by anyone except me, but it is
extremely powerful and potentially a complete showstopper for the kind
of design inherent in the goal stack approach. I have certainly never
seen anything like a reasonable rebuttal of it: even if it turns out
not to be as serious as I claim it is, it still needs to be addressed in
a serious way before anyone can make assertions about what goal stack
systems can do.
What is the significance of just this one idea? That all the goal stack
approaches might be facing a serious a problem if they want to get
autonomous, powerful learning mechanisms that build themselves from a
low level. So what are AI researchers doing about this problem? Trying
to address the issue and figure out how to build goal/motivational
systems that can bootstrap themselves? Not likely! They just sweep it
under the carpet and avoid getting their systems to be autonomous at
all!! Amazing really: if there is a problem, just have a tacit
agreement to postpone it until later, and pretend it will go away.
The only conceivable way to get around it is to hand-build an AI all the
way up to the point where it can be generally intelligent enough to
understand goals written in terms of its (hand-built) knowledge base.
But this is just the Cyc approach, and Cyc would have to be so smart
that it could discuss with you the concepts on its goal stack before it
would be smart enough to carry on by itself. Anyone want to put bets on
how soon *that* is going to happen? Does hand-building an AI all the
way up to that level sound like something a lone hacker is going to do
in their basement when nobody is looking? I don't think so.
"But," you might say, "I could imagine a reasonably large, hand-built
goal-stack AI still being smart enough to understand the <Build
Paperclips!> goal and then go around destroying the world. It might not
be generally intelligent, but it surely could be intelligent enough to
do that?"
To this reply I would say: what makes you so sure that it would have
the power to be dangerous? That might just as easily be an illusion
brought on by watching too many Mad Robot movies .... if it is not up to
full general intelligence (i.e. if it does not have the ability to
bootstrap itself using autonomous knowledge acquisition techniques),
then the thing is going to be a pushover. It will have spent its life
doing none of the exploratory, intelligence-building activity needed to
get on in the real world. Instead, it is vulnerable to all the mistakes
and stupidities put into it by a weary, error-prone gang of Ai
programmers who tried their best to anticipate all the intelligence it
would have acquired if it had gained its knowledge autonomously.
I would claim, therefore, that when you assume that a goal stack AI
system would actually make it to general intelligence (never mind
sperintelligence!), you are quietly, unwittingly, making an enormous
assumption about what kind of AI design would actually function
correctly. In that sense, any statement about how easy it would be for
a paperclip maximizer to go on the rampage are just not believable.
Conclusion: the dangerousness of an AI depends crucially on very
detailed arguments about the viability of the design. Without those
detailed arguments, no sensible conclusion are reachable.
I can't resist the temptation to close on humorous note, with an
(edited) excerpt from Marvin's encounter with the Frogstar Scout robot
class D. THIS, I think, is the most likely character of an conversation
between a real intelligence and a paperclip maximizing AI. :-)
***************************************************
[From Hitchhiker's Guide to the Galaxy, book 2 (Restaurant at the End of
the Universe) chapter 7.
Marvin looked pitifully small as the gigantic black tank rolled to a
halt in front of him.
"Out of my way little robot," growled the tank.
"I'm afraid," said Marvin, "that I've been left here to stop you."
"You? Stop me?" roared the tank. "Go on!"
"No, really I have," said Marvin simply.
"What are you armed with?" roared the tank in disbelief.
"Guess," said Marvin.
"Errmmm ..." said the machine, vibrating with unaccustomed thought,
"laser beams?"
Marvin shook his head solemnly.
"No," muttered the machine in its deep guttural rumble, "Too obvious.
Anti-matter ray?" it hazarded.
"Far too obvious," admonished Marvin.
"Yes," grumbled the machine, somewhat abashed, "Er ... how about an
electron ram?"
This was new to Marvin. "What's that?" he said.
"One of these," said the machine with enthusiasm. From its turret
emerged a sharp prong which spat a single lethal blaze of light. Behind
Marvin a wall roared and collapsed as a heap of dust. The dust billowed
briefly, then settled.
"No," said Marvin, "not one of those."
"Good though, isn't it?"
"Very good," agreed Marvin.
"I know," said the Frogstar battle machine, after another moment's
consideration, "you must have one of those new Xanthic Re-Structron
Destabilized Zenon Emitters!"
"Nice, aren't they?" said Marvin.
"That's what you've got?" said the machine in considerable awe.
"No," said Marvin.
"Oh," said the machine, disappointed, "then it must be ..."
"You're thinking along the wrong lines," said Marvin, "You're failing to
take into account something fairly basic in the relationship between men
and robots."
"Er, I know," said the battle machine, "is it ..." it tailed off into
thought again.
"Just think," urged Marvin, "they left me, an ordinary, menial robot, to
stop you, a gigantic heavy-duty battle machine, whilst they ran off to
save themselves. What do you think they would leave me with?"
"Oooh, er," muttered the machine in alarm, "something pretty damn
devastating I should expect."
"Expect!" said Marvin, "oh yes, expect. I'll tell you what they gave me
to protect myself with shall I?"
"Yes, alright," said the battle machine, bracing itself.
"Nothing," said Marvin.
There was a dangerous pause. "Nothing?" roared the battle machine.
"Nothing at all," intoned Marvin dismally, "not an electronic sausage."
The machine heaved about with fury. "Well, doesn't that just take the
biscuit!" it roared, "Nothing, eh? Just don't think, do they?"
"And me," said Marvin in a soft low voice, "with this terrible pain in
all the diodes down my left side."
"Makes you spit, doesn't it?"
"Yes," agreed Marvin with feeling.
"Hell that makes me angry," bellowed the machine, "think I'll smash that
wall down!" The electron ram stabbed out another searing blaze of light
and took out the wall next to the machine.
"How do you think I feel?" said Marvin bitterly.
"Just ran off and left you, did they?" the machine thundered.
"Yes," said Marvin.
"I think I'll shoot down their bloody ceiling as well!" raged the tank.
It took out the ceiling of the bridge.
"That's very impressive," murmured Marvin.
"You ain't seeing nothing yet," promised the machine, "I can take out
this floor too, no trouble!" It took out the floor, too. "Hell's bells!"
the machine roared as it plummeted fifteen storeys and smashed itself to
bits on the ground below.
"What a depressingly stupid machine," said Marvin and trudged away.
******************************************************
Richard Loosemore
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?list_id=303
