Isn't your Nirvana trap exactly equivalent to Pascal's Wager? Or am I
missing something?
----- Original Message -----
From: "J Storrs Hall, PhD" <[EMAIL PROTECTED]>
To: <[email protected]>
Sent: Wednesday, June 11, 2008 10:54 PM
Subject: Re: [agi] Nirvana
On Wednesday 11 June 2008 06:18:03 pm, Vladimir Nesov wrote:
On Wed, Jun 11, 2008 at 6:33 PM, J Storrs Hall, PhD <[EMAIL PROTECTED]>
wrote:
> I claim that there's plenty of historical evidence that people fall
> into
this
> kind of attractor, as the word nirvana indicates (and you'll find
> similar
> attractors at the core of many religions).
Yes, some people get addicted to a point of self-destruction. But it
is not a catastrophic problem on the scale of humanity. And it follows
from humans not being nearly stable under reflection -- we embody many
drives which are not integrated in a whole. Which would be a bad
design choice for a Friendly AI, if it needs to stay rational about
Freindliness content.
This is quite true but not exactly what I was talking about. I would claim
that the Nirvana attractors that AIs are vulnerable to are the ones that
are
NOT generally considered self-destructive in humans -- such as religions
that
teach Nirvana!
Let's look at it another way: You're going to improve yourself. You will
be
able to do more than you can now, so you can afford to expand the range of
things you will expend effort achieving. How do you pick them? It's the
frame
problem, amplified by recursion. So it's not easy nor has it a simple
solution.
But it does have this hidden trap: If you use stochastic search, say, and
use
an evaluation of (probability of success * value if successful), then
Nirvana
will win every time. You HAVE to do something more sophisticated.
-------------------------------------------
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription:
http://www.listbox.com/member/?member_id=8660244&id_secret=103754539-40ed26
Powered by Listbox: http://www.listbox.com
