So if Omuhundro's claim rests on that fact that "being self improving" is
part of the AGI's makeup, and that this will cause the AGI to do certain
things, develop certain subgoals etc. I say that he has quietly inserted a
*motivation* (or rather assumed it: does he ever say how this is supposed
to work?) into the system, and then imagined some consequences.
I think that I'm missing something here . . . . Omohundro is *explicitly*
assuming self-improving and yes, self-improving is a goal/motivation. What
do you believe that this proves/disproves? I'm not getting your point.
Further, I do not buy the supposed consequences. Me, I have the
"self-improving" motivation too. But it is pretty modest, and also it is
just one among many, so it does not have the consequences that he
attributes to the general existence of the self-improvement motivation.
AS I said in my previous e-mail, I don't buy his consequences either.
My point is that since he did not understand that he was making the
assumption,
Excuse me? What makes you believe that he didn't understand that he was
making the self-improvement assumption or that it was a goal/motivation? It
looked pretty deliberate to me.
and did not realize the role that it could play in a Motivational
Emotional system (as opposed to a Goal Stack system),
OK. So could you describe what role it would play in an MES system as
opposed to a Goal Stack System? I don't see a difference in terms of
effects.
he made a complete dog's dinner of claiming how a future AGI would
*necessarily* behave.
This I agree with -- but not because of any sort of differences between GS
and MES systems. I don't believe that his conclusions apply to an
intelligent GS system either.
Only in a Goal Stack system is there a danger of a self-improvement
supergoal going awol.
Why? An MES system requires more failures to have a problem, but certain
types of environment could (and should) cause such a problem.
As far as i can see, his arguments simply do not apply to MES systems: the
arguments depend too heavily on the assumption that the architecture is a
Goal Stack. It is simply that none of what he says *follows* if an MES is
used. Just a lot of non-sequiteurs.
I *STILL* don't get this. His arguments depend heavily upon the system
having goals/motivations. Yes, his arguments do not apply to an MES system
without motivations. But they do apply to MES systems with motivations
(although, again, I don't agree with his conclusions).
When an MES system is set up with motivations (instead of being blank)
what happens next depends on the mechanics of the system, and the
particular motivations.
YES! But his argument is that to fulfill *any* motivation, there are
generic submotivations (protect myself, accumulate power, don't let my
motivation get perverted) that will further the search to fulfill your
motivation.
= = = = =
As a relevant aside, you never answered my question regarding how you
believed an MES system was different from a system with a *large* number of
goal stacks.
----- Original Message -----
From: "Richard Loosemore" <[EMAIL PROTECTED]>
To: <agi@v2.listbox.com>
Sent: Friday, May 23, 2008 9:22 PM
Subject: Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity
Outcomes...]
Mark Waser wrote:
he makes a direct reference to goal driven systems, but even more
important he declares that these bad behaviors will *not* be the result
of us programming the behaviors in at the start .... but in an MES
system nothing at all will happen unless the designer makes an explicit
decision to put some motivations into the system, so I can be pretty
sure that he has not considered that type of motivational system when he
makes these comments.
Richard, I think that you are incorrect here.
When Omohundro says that the bad behaviors will *not* be the result of us
programming the behaviors in at the start, what he means is that the very
fact of having goals or motivations and being self-improving will
naturally lead (**regardless of architecture**) to certain (what I call
generic) sub-goals (like the acquisition of power/money,
self-preservation, etc.) and that the fulfillment of those subgoals,
without other considerations (like ethics or common-sense), will result
in what we would consider bad behavior.
This I do not buy, for the following reason.
What is this thing called "being self improving"? Complex concept, that.
How are we going to get an AGI to do that? This is a motivation, pure and
simple.
So if Omuhundro's claim rests on that fact that "being self improving" is
part of the AGI's makeup, and that this will cause the AGI to do certain
things, develop certain subgoals etc. I say that he has quietly inserted a
*motivation* (or rather assumed it: does he ever say how this is supposed
to work?) into the system, and then imagined some consequences.
Further, I do not buy the supposed consequences. Me, I have the
"self-improving" motivation too. But it is pretty modest, and also it is
just one among many, so it does not have the consequences that he
attributes to the general existence of the self-improvement motivation. My
point is that since he did not understand that he was making the
assumption, and did not realize the role that it could play in a
Motivational Emotional system (as opposed to a Goal Stack system), he made
a complete dog's dinner of claiming how a future AGI would *necessarily*
behave.
Could an intelligent system be built without a rampaging desire for
self-improvement (or, as Omuhundro would have it, rampaging power hunger)?
Sure: a system could just modestly want to do interesting things and have
new and pleasureful experiences. At the very least, I don't think that
you could claim that such an unassuming, hedonistic and unambitious type
of AGI is *obviously* impossible.
I believe that he is correct in that goals or motivations and
self-improvement will lead to generic subgoals regardless of
architecture. Do you believe that your MES will not derive generic
subgoals under self-improvement?
See above: if self-improvement is just one motivation among many, then
the answer depends on exactly how it is implemented.
Only in a Goal Stack system is there a danger of a self-improvement
supergoal going awol.
Omohundro's arguments aren't *meant* to apply to an MES system without
motivations -- because such a system can't be considered to have goals.
His arguments will start to apply as soon as the MES system does have
motivations/goals. (Though, I hasten to add that I believe that his
logical reasoning is flawed in that there are some drives that he missed
that will prevent such bad behavior in any sufficiently advanced system).
As far as i can see, his arguments simply do not apply to MES systems: the
arguments depend too heavily on the assumption that the architecture is a
Goal Stack. It is simply that none of what he says *follows* if an MES is
used. Just a lot of non-sequiteurs.
When an MES system is set up with motivations (instead of being blank)
what happens next depends on the mechanics of the system, and the
particular motivations.
Richard Loosemore
-------------------------------------------
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription:
http://www.listbox.com/member/?&;
Powered by Listbox: http://www.listbox.com
-------------------------------------------
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription:
http://www.listbox.com/member/?member_id=8660244&id_secret=103754539-40ed26
Powered by Listbox: http://www.listbox.com
