Mark Waser wrote:
...
The motivation that is in the system is "I want to achieve *my* goals".
The goals that are in the system I deem to be entirely irrelevant
UNLESS they are deliberately and directly contrary to Friendliness. I
am contending that, unless the initial goals are deliberately and
directly contrary to Friendliness, an optimizing system's motivation
of achieve *my* goals (over a large enough set of goals) will
eventually cause it to finally converge on the goal of Friendliness
since Friendliness is the universal super-meta-subgoal of all it's
other goals (and it's optimizing will also drive it up to the
necessary intelligence to understand Friendliness). Of course, it may
take a while since we humans are still in the middle of it . . . . but
hopefully we're almost there. ;-)
...
Mark
I think here we need to consider A. Maslow's hierarchy of needs. That
an AGI won't have the same needs as a human is, I suppose, obvious, but
I think it's still true that it will have a "hierarchy" (which isn't
strictly a hierarchy). I.e., it will have a large set of motives, and
which it is seeking to satisfy at any moment will alter as the
satisfaction of the previous most urgent motive changes.
It it were a human we could say that breathing was the most urgent
need...but usually it's so well satisfied that we don't even think about
it. Motives, then, will have satisficing as their aim. Only aberrant
mental functions will attempt to increase the satisfying of some
particular goal without limit. (Note that some drives in humans seem to
occasionally go into that "satisfy increasingly without limit" mode,
like quest for wealth or power, but in most sane people these are reined
in. This seems to indicate that there is a real danger here...and also
that it can be avoided.)
-------------------------------------------
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription:
http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b
Powered by Listbox: http://www.listbox.com
