Thanks to Peter for starting this discussion and to Ben for following up. This seems to me a more constructive way to talk about "Friendly AI".
Now it's my turn to comment on the 8 Guidelines, according to my NARS design (for people who have no idea what I'm talking about, see http://www.cogsci.indiana.edu/farg/peiwang/papers.html). 1. Friendliness-topped goal system Though in NARS initial goals can be specified by the designer, they are not the "supergoals" described. Due to insufficient knowledge and resources, there is no way to guarantee the consistency (or even the relevance) of the subgoals and initial goals. Therefore, to have friendliness as the only initial goal does not make the system friendly. 2. Cleanly causal goal system Since all causal knowledge is empirical, and therefore maybe biased, the above problem is also here. Though NARS indeed has a “cleanly causal goal system”, where all goals are derived from the initial goals according to the current causal knowledge, this mechanism cannot guarantee friendliness. 3. Probabilistic supergoal content In NARS all empirical knowledge has uncertainty, so this is not an issue for me. 4. Acquisition of Friendliness sources As an adaptive system, NARS will learn various types of knowledge from human beings. I don’t see anything special about ethical knowledge on this issue. 5. Causal validity semantics. Again, this is just causal inference, which can be applied to moral issues, but is not limited there. 6. Injunctions. With insufficient knowledge and resources, it is impossible “to prevent possible negative outcomes”. Of course, the system will try to avoid them as much as possible. 7. Self-modeling of fallibility as above. 8. Controlled ascent I don’t think it is possible to implement an "improvements counter". An adaptive system is changing all the time. I cannot tell when NARS is changing faster than usual, and if it is, whether this is an alarm signal. Furthermore, whether a change is an improvement or not is usually a subjective judgment based on feedback over a certain period. Something may look like an improvement now, but lead to a big trouble in the future. In summary, I agree with some of the “Guidelines”, though I think they are actually about AI, and are neutral to the friendliness issue. Of course, high intelligence can make a system friendlier, but it can also make the system more dangeours. To select proper initial goals and to provide proper feedback will reduce the danger, but cannot eliminate it. I post this reply just to show that I fully agree that AI can be dangerous, and we should be ready to deal with the problems when designing an AI system. However, at the current time, moral discussions contribute little to the real work. I haven’t seen how the design of NARS should be changed to make it friendlier. Pei -------Original Message------- From: Ben Goertzel <[EMAIL PROTECTED]> Sent: 02/20/03 12:01 PM To: [EMAIL PROTECTED] Subject: RE: [agi] Building a safe AI > > Peter, At the end of the page you reference, you list "The Guidelines' eight design recommendations in the light of my theory of mind/ intelligence: 1. ... 2. ... ... " All of your comments in that section apply to Novamente without significant modification. Although in detail your design is of course quite different from Novamente. -- Ben > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On > Behalf Of Peter Voss > Sent: Thursday, February 20, 2003 11:48 AM > To: [EMAIL PROTECTED] > Subject: [agi] Building a safe AI > > > > > http://www.optimal.org/peter/siai_guidelines.htm > > Peter > > > > > -----Original Message----- > From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On > Behalf Of Ben Goertzel > > I would recommend Eliezer's excellent writings on this topic if you don't > know them, chiefly www.singinst.org/CFAI.html . Also, I have a brief > informal essay on the topic, > www.goertzel.org/dynapsyc/2002/AIMorality.htm , > although my thoughts on the topic have progressed a fair bit since I wrote > that. Note that I don't fully agree with Eliezer on this stuff, but I do > think he's thought about it more thoroughly than anyone else > (including me). > > It's a matter of creating an initial condition so that the > trajectory of the > evolving AI system (with a potentially evolving goal system) will have a > very high probability of staying in a favorable region of state space ;-) > > ------- > To unsubscribe, change your address, or temporarily deactivate > your subscription, > please go to http://v2.listbox.com/member/?[EMAIL PROTECTED] > ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED] > ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
