Edward W. Porter wrote:
Richard in your November 02, 2007 11:15 AM post you stated:
“If AI systems are built with motivation systems that are stable, then
we could predict that they will remain synchronized with the goals of
the human race until the end of history.”
and
“I can think of many, many types of non-goal-stack motivational systems
for which [Matt’s statement about the inherent instability of goal
systems of recursively self improving AGIs] is a complete falsehood.”
In your 11/3/2007 1:17 PM post you described what I assume to be such a
suppostedly stable “non-goal-stack motivational system.” as follows:
“ consider the motivational system of the
best kind of AGI: it is motivated by a balanced set of desires that
include the desire to explore and learn, and empathy for the human
species. By definition, I would think, this simple cluster of desires
and empathic motivations *are* the things that "give it pleasure".
and
“I think that in general, making the AGI as similar to us as possible
(but without the aggressive and dangerous motivations that we are
victims of) would be a good idea simply because we want them to start
out with a strong empathy for us, and we want them to stay that way.”
I think this type of motivational system makes a lot of sense, but for
all the reasons stated in my Fri 11/2/2007 2:07 PM post (arguments you
have not responded to) as well as many other reasons, it does appear at
all certain such a motivational system would reliably remain stable and
“synchronized with the goals of the human race until the end of
history,” as you claim.
For example, humans might for short sighted personal gain (such as when
using them in weapon systems)
Whoaa! You assume that it would be possible to "use" an AGI for
personal gain, or in a weapon system. If it starts out with the
supposed empathic motivational system, it would not allow this.
or accidentally alter such a motivational
system.
Again, under the assumption, it would not allow such 'accidental"
alteration.
Or over time the inherent biases that were designed to make
AGI’s have empathy for humans, might cause it to have empathy for some
humans more than others
As part of its initial (assumed) empathy, it will set up mechanisms to
monitor such things. It could not possibly start having "more empathy
for some humans more than others" without *also* being aware of the fact
that, by being so biassed, it was in conflict with its general
motivation. So it would not do such a thing. (We have to remember not
to assume it would be both superintelligent, and also suscetible to such
easily-caught problems).
, or might cause them to make decisions that they
think are in our best interest, but would not.
Again, this assumes (implicitly) that it would both be generally and
broadly empathic -- which means sensitive to our wishes -- and at the
same time, for some inexplicable reason, decide to do something that it
thinks is in our best interests, without consulting us. Effectively
assumed that it would spontaneously *stop* being empathic, without
explaining how it could happen.
Or perhaps AGI robots
would begin to embody the "human features" that they have been taught to
be empathetic to better than people. Etc.
Does this mean things like beginning to get aggressive, or jealous, etc?
this is where the technical characteristics of a motivational system
become important: this kind of drift would be impossible unless the
motivational system already had aggressiveness modules built in (which
is not the case, by assumption).
The world is too complicated and is going to change too rapidly in the
next one hundred, one thousand, or ten thousand years for any goal
system designed circa 2015 to remain appropriate until the end of
history – unless history ends pretty soon.
Not true: the statement was that it would stay empathic to our motivations.
Only if the goal system were particularly rigid would this be a problem,
and by assumption I am talking about motivational systems that are
stable (diffuse systems, along the lines of my previously mentioned post).
If I am wrong I would appreciate the enlightenment and increased hope
that would come with being shown how I am wrong.
I apologize for giving too brief answers to these questions. I have too
much stuff that is not written out in long form.
Richard Loosemore
Ed Porter
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?member_id=8660244&id_secret=60958578-e87007
