Hi Aleksei,
In this thread [see the end of this email for quote], I feel you are
attacking a peripheral part of Shane's argument. The problems with
the feasibility of defining or implementing "Friendly AI" that Shane
is presenting in his main blog entry, are not really dependent on his
prefatory side-comment about self-preservation.
FWIW, I don't agree that AGI's with self-preservation as the top-level
supergoal are extremely likely to reign supreme in future history.
I'm not sure whether Shane meant to imply this or not, but I suspect
not.
One might argue that such AGI's would be likely to rise to the top in
some sort of fair, long-term AGI tournament -- because the other AGI's
would be wasting some of their energy on goals aside from
self-preservation.
But, there are a couple arguments against this. First, reality is not
a fair, long-term tournament. So, if someone initially creates an AGI
with
-- self-preservation as a powerful subgoal
-- a different top-level supergoal
then this AGI may quickly accumulate enough power to have a permanent
"lead" over other AGI's even if these others are more selfish.
Second, the universe itself may have properties more sympathetic to
some kinds of AGI's than others. A simple case of this would be if
there were non-human-origin AGI's out there that are more amenable to
non-totally-selfish AGI's than to totally-selfish ones.
-- Ben G
On 9/10/06, Aleksei Riikonen <[EMAIL PROTECTED]> wrote:
On 9/10/06, Shane Legg <[EMAIL PROTECTED]> wrote:
> I did not claim that a primary interest in self preservation was a
> necessary feature when designing an AGI. I only claimed that
> the greater an AGI's emphasis on self preservation, the more
> likely it is that it will survive.
You claimed that an AGI is "critically unstable due to evolutionary
pressure" unless it "is primarily interested in its own self
preservation".
To me, "I did not claim that a primary interest in self preservation
was a necessary feature" seems to directly contradict this.
But, ok, maybe you don't actually mean to criticize systems, whose
interest in self-preservation arises only as a derived value. Thanks
for the clarification. In that case I would like to point out, that
the "Friendly AI" systems which SIAI-folks consider, certainly value
self-preservation as a derived value, and hence this observation of
yours isn't really criticism of Friendly AI, even though you seemed to
present it as such.
-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
