As of now, we are aware of no non-human friendlies, so the set of excluded beings will in all likelihood be the empty set.
Eliezer's current vision of Friendliness puts AGIs (who are non-human friendlies) in the role of excluded beings. That is why I keep hammering this point.
To answer your question, I don't see the "people are evil and will screw it all up" scenario as being even remotely likely, for reasons of self-interest among others. And I think it very likely that if it turns out that including non-human friendlies is the right thing to do, that the system will do as designed and renormalize accordingly.
People are *currently* screwing it all up in the sense that our society is *seriously* sub-optimal and far, FAR less than it could be. Will we screw it up to the point of self-destruction? That's too early to tell. The Cuban Missile Crisis was an awfully near miss. Grey Goo would be *really* bad (though I think that it is a bit further off than most people on this list). It's scary to even consider what I *know* that I could do if I were a whack-job terrorist but with my knowledge.
The only reason why I am as optimistic as I am currently is because I truly do believe that Friendliness is an attractor that we are solidly on the approach path to and I hope that I can speed the process by pointing that fact out.
As for the other option, my question was not about the dangers relating to *who is or is not protected*, but rather *whose volition is taken into account* in calculating the CEV, since your approach considers only the volition of friendly humanity (and non-human friendlies but not non-friendly humanity), while Eliezer's includes all of humanity.
Actually, I *will* be showing that basically Friendly behavior *IS* extended to everyone except in so far as non-Friendlies insist upon being non-Friendly. I just didn't see a way to successfully introduce that idea early *AND* forestall Vladimir's obvious "so why don't I just kill them all" argument. I need to figure out a better way to express that earlier.
------------------------------------------- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
