J. Storrs Hall, PhD. wrote:
On Friday 01 December 2006 23:42, Richard Loosemore wrote:
It's a lot easier than you suppose. The system would be built in two
parts: the motivational system, which would not change substantially
during RSI, and the "thinking part" (for want of a better term), which
is where you do all the improvement.
For concreteness, I have called these the Utility Function and World Model in
my writings on the subject...
Well .... I am avoiding "Utility Function" precisely because it has a
specific meaning in the context of the type of AI that I have been
lambasting as the "goal stack approach" to motivation.
A plan that says "Let RSI consist of growing the WM and not the UF" suffers
from the problem that the sophistication of the WM's understanding soon makes
the UF look crude and stupid. Human babies want food, proximity to their
mothers, and are frightened of strangers. That's good for babies but a person
with greater understanding and capabilities is better off (and the rest of us
are better off if the person has) a more sophisticated UF as well.
I don't want to take the bait on your baby-motivation analogy because I
do not believe the difference between human baby and adult is the same
as the difference between adult AI and even-smarter-adult-AI. Some,
including myself, are of the opinion that there is a threshold of
sentience above which things settle down a lot, so the AI would never
look back on its earlier motivational system and call it "crude and stupid".
Also, implicit in your description of the UF and WM are some ideas that
I have been explicitly avoiding in my discussion of "diffuse"
motivational systems. That would make some of your points not applicable.
No time to spell it out right now. If you look back at the root of this
thread you might see why, or you can wait until I get the thing written
up properly.
It is not quite a contradiction, but certainly this would be impossible:
deciding to make a modification that clearly was going to leave it
wanting something that, if it wanted that thing today, would contradict
its current priorities. Do you see why? The motivational mechanism IS
what the system wants, it is not what the system is considering wanting.
This is a good first cut at the problem, and is taken by e.g. Nick Bostrom in
a widely cited paper at http://www.nickbostrom.com/ethics/ai.html
Well, Nick Bostrum is not the origin of this idea: it is kind of obvious.
The system is not protecting current beliefs, it is believing its
current beliefs. Becoming more capable of understanding the "reality"
it is immersed in? You have implicitly put a motivational priority in
your system when you suggest that that is important to it ... does that
rank higher than its empathy with the human race?
You see where I am going: there is nothing god-given about the desire
to "understand reality" in a better way. That is just one more
candidate for a motivational priority.
Ah, but consider: knowing more about how the world works is often a valuable
asset to the attempt to increase the utility of the world, *no matter* what
else the utility function might specify.
Whoa: "increase the utility of the world"? Again, your terms do not
map onto a viewpoint of motivation that dumps the idea of a crude UF.
In essence, you have restated the idea that I was attacking: that
"increase the utility of the world" is a motivation that trumps others.
It is not necessarily the case that this is the system's primary
motivation.
Thus, a system's self-modification (or evolution in general) is unlikely to
remove curiosity / thirst for knowledge / desire to improve one's WM as a
high utility even as it changes other things.
Yes and no. I am going to have to get back to you on this.
Here is an idea to try to fit into that worldview. After the
Singularity, I would love to go into a closed domain in which I get to
live in a replica of 17th century England, growing up there from
childhood with my memories put on ice for the duration of a (then
normal) lifetime, and with the goal of experiencing what it would have
been like to be a Natural Philosopher discovering the wonder of science
for the first time. I want to discover things that are known in this
era, after temporarily removing them from my mind.
So I would be.... what? Contradicting my utility function by
deliberately removing knowledge? Seeking to do what? Get the knowledge
back a different way? Am I seeking knowledge, or just seeking a new
"experience"?
I claim the latter: but that idea of seeking new experience just does
not map onto the kind of silly :-) utility functions that AI people play
games with today. They cannot even represent the goal of "having
interesting subjective experiences", as far as I can see.
Richard Loosemore
There are several such properties of a utility function that are likely to be
invariant under self-improvement or evolution. It is by the use of such
invariants that we can design self-improving AIs with reasonable assurance of
their continued beneficence.
--Josh
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?list_id=303
