Ben,
Thanks for that. Your explanation makes the whole thing a lot clearer.
I'll come back to this thread again after Eliezer's discussion on AGI
friendliness has progressed a bit further.
Cheers, Philip
From: "Ben Goertzel" <[EMAIL PROTECTED]>
To: <[EMAIL PROTECTED]>
Subject: RE: [agi] Novamente: how crtical is self-improvement to
getting human parity?
Date sent: Sun, 16 Feb 2003 12:13:16 -0500
Send reply to: [EMAIL PROTECTED]
Hi,
As we're thinking about it now, Novamente Version 1 will not have
feature 4. It will involve Novamente learning a lot of small programs to
use within its overall architecture, but not modifying its overall
architecture.
Technically speaking: Novamente Version 1 will be C++ code, and
within this C++ code, there will small programs running in a language
called Sasha. Novamente will write its own Sasha code to run in its
C++ "Mind OS", but will not modify its C++ source.
The plan for Novamente Version 2 is still sketchy, because we're
focusing on Version 1, which still has a long way to go. One possible
path is to write a fast, scalable Sasha compiler and write the whole
thing in Sasha. Then the Sasha-programming skills of Novamente
Version 1 will fairly easily translate into skills at deeper-level self-
modification. (Of course, the Sasha compiler will be in C++ ... so
eventually you can't escape teaching Novamente C++ ;-).
How intelligent Novamente Version 1 will be -- well ... hmmm ... who
knows!!
Among the less sexy benefits of the Novamente Version 2 architecture,
I really like the idea of having Novamente correct bugs in its own
source code. It is really hard to get a complex system like this truly
bug-free..... An AGI should be a lot better at debugging very complex
code than humans are!
So the real answer to your question is, I'm not sure. My hope, and my
guess, is that Novamente Version 1 will --- with ample program learning
and self-modification on the Sasha level -- be able to achieve levels of
intelligence that seem huge by human standards.
Of course, a lot of sci-fi scenarios suggest themselves: What happens
when we have a super-smart Version 1system and it codes Version 2
and finds a security hole in Linux and installs Version 2 in place of half
of itself, then all of itself... etc.
-- Ben G
-----Original Message-----
From: [EMAIL PROTECTED] [mailto:owner-
[EMAIL PROTECTED]]On Behalf Of Philip Sutton
Sent: Sunday, February 16, 2003 10:55 AM
To: [EMAIL PROTECTED]
Subject: [agi] Novamente: how crtical is self-improvement to
getting human parity?
Hi Ben,
As far as I can work out, there are four things that could conceivably
contribute to a Novamente reaching human intelligence parity:
1 the cleverness/power of the original architecture
2 the intensity, length and effectiveness of the Novamente learning
after being booted up
3 the upgrading of the achitecture/code base by humans as a result
of
learning by anyone (including Novamentes).
4 the self-improvement of the achitecture/code base by the
Novamente
as a result of learning by anyone (humans and Novamentes).
To what extend is the learning system of the current Novamente
system (current or planned for the first switched on version) dependent
on or intertwined with the capacity for a Novamente to alter its own
fundamental architecture?
It seems to me that the risk of getting to the sigularity (or even a
dangerous earlier stage) without the human plus AGI community being
adequately prepared and sufficiently ethically mature lies in the
possiblity of AGIs self-improving on an unhalted exponential trajectory.
If you could get Novamentes to human parity using strategies 1-3 only
then you might be able to control the process of moving beyond human
parity sufficiently to make it safe.
If getting to human parity relies on strategy 4 then the safety strategy
could well be very problematic - Eliezer's full Friendly AI program might
need to be applied in full (ie. developing the theory of friendlieness first
and then applying "Supersaturated Friendliness" (as Eliezer calls it).
What do you reckon?
Cheers, Philip
-------
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
