Re: Self-improvement is not a special case (was Re: [agi] Religion-free technical content)
On 10/12/07, Eliezer S. Yudkowsky <[EMAIL PROTECTED]> wrote: > some of us are much impressed by it. Anyone with even a surface grasp > of the basic concept on a math level will realize that there's no > difference between self-modifying and writing an outside copy of > yourself, but *either one* involves the sort of issues I've been > calling "reflective". > Well, this could be at least a definition of "self-modifying" ;-) - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=53163374-01d6ba
Re: Self-improvement is not a special case (was Re: [agi] Religion-free technical content)
Tim Freeman wrote: My point is that if one is worried about a self-improving Seed AI exploding, one should also be worried about any AI that competently writes software exploding. There *is* a slight gap between competently writing software and competently writing minds. Large by human standards, not much by interspecies standards. It does involve new math issues, which is why some of us are much impressed by it. Anyone with even a surface grasp of the basic concept on a math level will realize that there's no difference between self-modifying and writing an outside copy of yourself, but *either one* involves the sort of issues I've been calling "reflective". -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=53161697-a947ab
RE: Self-improvement is not a special case (was Re: [agi] Religion-free technical content)
Linas Vepstas:> > >Let's take Novamente as an example. ... It cannot improve itself> > >until the following things happen:> > >> > >1) It acquires the knowledge and skills to become a competent> > > programmer, a task that takes a human many years of directed> > > training and practical experience.> > Wrong. This was hashed to death in previous emails; and then again > probably several more times before I joined the list. > > Anyone care to assemble a position paper on "self improvement"> that reviews the situation? I'm slightly irritated by the > recurring speculation and misunderstanding. Ok, the conversation was about how Novamente could recursively self-improve itself into a runaway hard takeoff scenario. You're claiming that it can do so without the knowledge or skills of a competent programmer, with the very convincing argument "Wrong". Care to elaborate at all? Or is your only purpose to communicate your slight irritation? - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=53154037-90851f
Re: Self-improvement is not a special case (was Re: [agi] Religion-free technical content)
From: Derek Zahn <[EMAIL PROTECTED]> >You seem to think that self-reference buys you nothing at all since it >is a simple matter for the first AGI projects to reinvent their own >equivalent from scratch, but I'm not sure that's true. The "from scratch" part is a straw-man argument. The AGI project will have lots of resources to draw on; it could read Hutter's papers or license HTM or incorporate lots of other packages or existing AI projects that it might find useful. My point is that if one is worried about a self-improving Seed AI exploding, one should also be worried about any AI that competently writes software exploding. Keeping its source code secret from itself doesn't help much. Hmm, I suppose an AI that does mechanical engineering could explode too, perhaps by doing nanotech, so AI's competently doing engineering is a risk in general. -- Tim Freeman http://www.fungible.com [EMAIL PROTECTED] - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=53137773-858c34
Re: Self-improvement is not a special case (was Re: [agi] Religion-free technical content)
> >Let's take Novamente as an example. ... It cannot improve itself > >until the following things happen: > > > >1) It acquires the knowledge and skills to become a competent > > programmer, a task that takes a human many years of directed > > training and practical experience. Wrong. This was hashed to death in previous emails; and then again probably several more times before I joined the list. Anyone care to assemble a position paper on "self improvement" that reviews the situation? I'm slightly irritated by the recurring speculation and misunderstanding. --linas - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=53103798-a02f8e
Re: Self-improvement is not a special case (was Re: [agi] Religion-free technical content)
Derek, Tim, There is no oversight: self-improvement doesn't necessarily refer to actual instance of self that is to be improved, but to AGI's design. Next thing must be better than previous one for runaway progress to happen, and one way of doing it is for next thing to be a refinement of previous thing. Self-improvement 'in place' may depending on nature of improvement be preferable, if it provides a way to efficiently transfer acquired knowledge from previous version to the next one (probably even without any modification). On 10/12/07, Derek Zahn <[EMAIL PROTECTED]> wrote: > > Tim Freeman: > > > No value is > > added by introducing considerations about self-reference into > > conversations about the consequences of AI engineering. > > > > Junior geeks do find it impressive, though. > > The point of that conversation was to illustrate that if people are worried > about Seed AI exploding, then one option is to not build Seed AI (since that > is only one approach to developing AGI, and in fact I do not know of any > actual project that includes it at present). Quoting Yudkowsky: > > > The task is not to build an AI with some astronomical level > > of intelligence; the task is building an AI which is capable > > of improving itself, of understanding and rewriting its own > > source code. > > Perhaps only "junior geeks" like him find the concept relevant. You seem > to think that self-reference buys you nothing at all since it is a simple > matter for the first AGI projects to reinvent their own equivalent from > scratch, but I'm not sure that's true. > > > This list is sponsored by AGIRI: http://www.agiri.org/email > To unsubscribe or change your options, please go to: > http://v2.listbox.com/member/?&; -- Vladimir Nesovmailto:[EMAIL PROTECTED] - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=53025431-7e3757
RE: Self-improvement is not a special case (was Re: [agi] Religion-free technical content)
Tim Freeman:> No value is> added by introducing considerations about self-reference into> conversations about the consequences of AI engineering.> > Junior geeks do find it impressive, though. The point of that conversation was to illustrate that if people are worried about Seed AI exploding, then one option is to not build Seed AI (since that is only one approach to developing AGI, and in fact I do not know of any actual project that includes it at present). Quoting Yudkowsky: > The task is not to build an AI with some astronomical level > of intelligence; the task is building an AI which is capable > of improving itself, of understanding and rewriting its own > source code. Perhaps only "junior geeks" like him find the concept relevant. You seem to think that self-reference buys you nothing at all since it is a simple matter for the first AGI projects to reinvent their own equivalent from scratch, but I'm not sure that's true. - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=53018769-30e88d
RE: Self-improvement is not a special case (was Re: [agi] Religion-free technical content)
Tim Freeman writes:> >Let's take Novamente as an example. ... It cannot improve itself> >until the following things happen:> >> >1) It acquires the knowledge and skills to become a competent> > programmer, a task that takes a human many years of directed> > training and practical experience.> > > >2) It is given access to its own implementation and permission to alter it.> > > >3) It understands its own implementation well enough to make a helpful change.> >...> > I agree that resource #1, competent programming, is essential for any> interesting takeoff scenario. I don't think the other two matter,> though. Ok, this alternative scenario -- where Novamente secretly reinvents the theoretical foundations needed for AGI development, designs its successor from those first principles, and somehow hijacks an equivalent or superior supercomputer to receive the de novo design and surreptitiously trains it to superhuman capacity -- should also be protected against. It's a fairly ridiculous scenario, but for completeness should be mentioned. - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=53003223-9d4579
