There will always be some hostile, virus or Trojan spreading elements. I can envision AI wars.
AI can do a lot for mankind, yet the development may get bogged down in more security versus productive development. Interesting, very interesting... Dan Goe ---------------------------------------------------- >From : Mark Waser <[EMAIL PROTECTED]> To : <agi@v2.listbox.com>, <[EMAIL PROTECTED]> Subject : Re: [agi] Two draft papers: AI and existential risk; heuristics and biases Date : Wed, 7 Jun 2006 11:32:36 -0400 > Wow! Reading this, I must say that I was struck by the Geddes-like proportion of *claims* to *reasonable proofs* (or even true discussions where you've deigned to share even the beginnings of a proof). > > Claims like "AGI understanding will always run ahead of FAI understanding" are fundamentally misleading and unhelpful -- particularly if you had a clearly designed system of morality. Yes, you can "prove" that *if* FAI encompasses AGI *then* AGI must always run ahead BUT that's as useful a statement as saying that an ocean plus a drop is larger than an ocean. > > You have not only opted to go down what appears to many of us to be a dead-end (that of ABSOLUTE surety of friendliness) but you insist upon being extremely condescending of other people's understanding and efforts. I wonder if you've noticed that it is the opinion of many posters here that the Pentagon will succeed at AGI long before you succeed at your efforts of "mathematically ensuring" friendliness (assuming that it is even possible). > > It is my contention that there is not only one way to truth (or friendliness) and that a blind insistence on such will greatly increase the odds of AGI without any attempt at friendliness built in (not to mention the fact that I would also contend that such a claim is, in and of itself, unfriendly :-). > > It is also my contention that CEV "sweeps all the interesting parts of the problem under the rug, forgives your own ignorance, hopes for good results without proof, and generally holds yours to much too low a standard to come up with an interesting theory" since CEV fundamentally ASSUMES that you're going to be able to extrapolate to something (friendliness) that many of us are not even headed for (and we all should know how accurate extrapolation is in general, particularly across the phase changes that we all say are inevitable with greater intelligence). > > I think that we as a community need to get off our butts and start building consensus as to what even the barest framework of friendliness is. I think that we've seen more than enough proof that no one here can go on for more than twenty lines without numerous people objecting vociferously to their idea of friendliness (and just wait til you start trying to include Leon Kass, your average fundamentalist Christian or your average fundamentalist Muslim). But you've gone off and invented a magical system which will solve all of these problems by determining what we would define as friendly if we were "better" (and are now looking for a way to mathematically guarantee that such a system will work correctly). > > I do applaud your work on existential risk and heuristics and biases. I'm just appalled by your approach to the FAI community. > > Mark > > ----- Original Message ----- > From: "Eliezer S. Yudkowsky" <[EMAIL PROTECTED]> > To: <[EMAIL PROTECTED]>; <agi@v2.listbox.com> > Sent: Wednesday, June 07, 2006 1:20 AM > Subject: Re: [agi] Two draft papers: AI and existential risk; heuristics and biases > > > > Ben Goertzel wrote: > >> > >> CFAI tried, and ultimately didn't succeed, to articulate an approach > >> to solving the problem of Friendly AI. Or at least, that is the > >> impression it made on me.... > >> > >> On the other hand, AIGR basically just outlines the problem of > >> Friendly AI and explains why it's important and why it's hard. > >> > >> In this sense, it seems to be a retreat.... > >> > >> I suppose the subtext is that your attempts to take the intuitions > >> underlying CFAI and turn them into a more rigorous and defensible > >> theory did not succeed. > > > > The subtext is: > > > > 1) Do not propose any solutions before discussing the problem as > > thoroughly as possible without proposing any. This is most important > > when dealing with extremely difficult problems, as that is when people > > are most apt to propose solutions immediately. See the associated book > > chapter on heuristics and biases. > > > > 2) This is a book chapter for general academic readers interested in > > how AI fits into the big picture of global catastrophic risks. It was > > sharply constrained by space, as you may have noticed, and there simply > > wasn't time to go into any AI-design details. That would have been a > > book not a chapter. > > > > 3) I am still working on a rigorous theory and have made what I count > > as progress. Over the next year or so, I hope to work on this nearly > > full-time, and am refusing to take on other commitments (such as book > > chapters) to make sure my time stays free. > > > > CFAI proposed a solution too quickly, and worse, claimed it was a > > workable approach in itself. Before a complete solution necessarily > > comes a partial solution, where you know how to solve M out of N > > problems with M < N. This is where I am now, but at least I know it. > > At the time of CFAI, I (Eliezer-2001) had difficulty admitting I didn't > > have a workable solution in hand, because that would have meant that I'd > > have to work more on FAI theory instead of doing what I wanted to do, > > what I thought would make me look more respectable, and plunging > > straight into AI as soon as I had the funding to hire more > > programmers... Actually, it would be more accurate to say the reason I > > wanted to believe I had a workable solution in hand, was because this > > let me preserve all the existing plans for AGI development that I had > > made before I realized that FAI was an issue. People try to preserve as > > much of their existing plans as possible, when unexpected news arrives; > > in this case, what I needed to do, and did not do, was rethink all my > > plans from scratch. But that was a much younger Eliezer... Needless to > > say, I think that you, Ben, are now making my old mistake. > > > >> I also note that your Coherent Extrapolated Volition ideas were not > >> focused on in AIGR, which I think is corrrect because I consider CEV a > >> fascinating science-fictional speculation without much likelihood of > >> ever being practically relevant. > > > > That is because CEV is merely my proposed *solution*, and AIGR doesn't > > even get far enough into discussing the problem; it is nowhere near the > > point where it would become wise to propose a solution. Did you read > > the chapter on heuristics and biases? If not, please stop here, and > > read that chapter. > > > >> I agree with you that taking a more rigorous mathematical approach is > >> going to be the way -- if any -- to a theory of FAI. However, I am > >> more optimistic that this approach will lead to a theory of FAI > >> **assuming monstrously great computational resources** than to a > >> theory of pragmatic FAI. This would be expected since thanks to > >> Schmidhuber, Hutter and crew we now have the beginnings of a theory of > >> AGI itself assuming monstrously great computational resources, but > >> nothing approaching a theory of AGI assuming realistic computational > >> resources... > > > > As previously disscussed on AGI, I think that Schmidhuber, Hutter et. > > al. left key dimensions out of their AI, such as its ability to conceive > > of what happens when it drops an anvil on its own head. That is, what > > happens when an environmental process penetrates the intrinsic Cartesian > > boundary on which their formalism is based. > > > >> It would seem to me that FIRST we should try to create a theoretical > >> framework useful for analyzing and describing AGIs that operate with > >> realistic computational resources. > > > > This is more or less what I'm doing right now. > > > > And lo, I only started making progress on the problem by holding it to > > Friendly AI standards of determinism and knowability. Otherwise, you > > end up sweeping all the interesting parts of the problem under the rug, > > forgiving your own ignorance, hoping for good results without proof, and > > generally holding yourself to much too low a standard to come up with an > > interesting theory. I've made a lot more progress on AGI than in the > > CFAI/LOGI era, and the difference was holding myself to the standard of > > proof-of-Friendliness. > > > > AGI understanding will always run ahead of FAI understanding; I have > > previously remarked on this point - it is what makes the problem of > > Earth's survival *difficult*. Surely it is not possible to be able to > > build FAI, and not be able to build AGI. But you can develop > > sophisticated AGI techniques that are not even theoretically usable for > > AGI, that *cannot* be reshaped to safety. Thinking about AGI doesn't > > put you on an incremental path. I've *been* there, Ben. I wrote LOGI > > while thinking about AGI, and then I had to throw LOGI away and start > > over from scratch because it wasn't even on an incremental path to FAI. > > Neither is CFAI, for that matter. > > > > > Then Friendly AI should be > > > approached, theoretically, within this framework. > > > > You won't be able to approach FAI "within" an AGI framework that was > > designed without thinking about FAI. You *will*, always, be able to > > approach a certain type of AGI within a framework that was designed for > > thinking about FAI. *That* is where the inequality comes from - that's > > why you'll always know more about AGI than FAI at any given point. > > > > What you need is a frame of mind in which there are no "AGI" problems. > > There is, simply, the goal of building a Friendly AI, and you do what is > > required for that in whatever order seems best, including devising a > > theory of optimization with limited computational resources. That > > someone else could call that "AGI" is of no consequence, except insofar > > as there is existential risk from incomplete theories. > > > >> I can see the viability of also proceeding in a more specialized way, > >> and trying to get a theory of FAI under limited resources in the > >> absence of an understanding of other sorts of AGIs under limited > >> resources. But my intuition is that the best way to approach "FAI > >> under limited resources" is to first get an understanding of "AGI > >> under limited resources." > > > > The vast majority of AGI techniques are intrinsically unsuited to FAI > > and are not on an incremental pathway to FAI. So why am I, right now, > > working mostly on "AGI"-ish questions, rather than CEV-ish questions? > > Because, in the course of solving those problems which are naturally > > encountered on the road to an FAI theory, one finds that the simplest > > questions of FAI, which must be answered first before moving on to more > > complex questions, happen to be questions *about a certain type of AGI*. > > This does *not* mean you can answer them if you conceive of what you > > are doing as "trying to build AGI" rather than "trying to build FAI". > > > > -- > > Eliezer S. Yudkowsky http://singinst.org/ > > Research Fellow, Singularity Institute for Artificial Intelligence > > > > ------- > > To unsubscribe, change your address, or temporarily deactivate your subscription, > > please go to http://v2.listbox.com/member/[EMAIL PROTECTED] > > > > ------- > To unsubscribe, change your address, or temporarily deactivate your subscription, > please go to http://v2.listbox.com/member/[EMAIL PROTECTED] ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
