> 1. How will the AI determine what is in the set of "horrible nasty > thing[s] that would make *us* unhappy"? I guess this is related to how you > will define the attractor precisely. > > 2. Preventing the extinction of the human race is pretty clear today, but > *human race* will become increasingly fuzzy and hard to define, as will > *extinction* when there are more options for existence than existence as > meat. In the long term, how will the AI decide who is "*us*" in the above > quote?
Excellent questions. The answer to the second question is that the value of *us* is actually irrelevant. Thinking that it is relevant is one of the fatal flaws of Eli's vision. The method of determination of what is in the set of "horrible nasty thing[s]" is (necessarily) coming as an integral part of the paper. So, to continue . . . . Part 2. Stephen Omohundro presented a paper at the AGI-08 post-conference workshop on "The Basic AI Drives" which is available at http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf. The paper claims to "identify a number of “drives” that will appear in sufficiently advanced AI systems of any design" and identifies these drives as "tendencies which will be present unless explicitly counteracted." It is my contention that these "drives" will appear not only in sufficiently advanced AI systems, but in *any* goal-directed system of sufficient intelligence (most particularly including human beings). The six drives that Omohundro identifies are 1.. self-improvement, 2.. rationality, 3.. utility function preservation, 4.. counterfeit utility prevention, 5.. self-protection, and 6.. acquisition and efficient use of resources. My take on these drives is that they are universally applicable sub-goals (and/or goal maintenance operations) for any goal which they do not directly conflict. Thus, *any* goal-driven intelligence (of sufficient intelligence) will display these drives/sub-goals (with the exception, of course, of those that directly contradict their goal) as part of their goal-seeking behavior. And thus, we get back to a specific answer to jk's second question. "*US*" should be assumed to apply to any sufficiently intelligent goal-driven intelligence. We don't need to define "*us*" because I DECLARE that it should be assumed to include current day humanity and all of our potential descendants (specifically *including* our Friendly AIs and any/all other "mind children" and even hybrids). If we discover alien intelligences, it should apply to them as well. I contend that Eli's vision of Friendly AI is specifically wrong because it does *NOT* include our Friendly AIs in "*us*". In later e-mails, I will show how this intentional, explicit lack of inclusion is provably Unfriendly on the part of humans and a direct obstacle to achieving a Friendly attractor space. TAKE-AWAY: All goal-driven intelligences have drives that will be the tools that will allow us to create a self-correcting Friendly/CEV attractor space. PART 3 will answer "what is in the set of "horrible nasty thing[s]". ------------------------------------------- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
