Argh! I hate premature e-mailing . . . . :-) Interlude 1 . . . . continued
One of the first things that we have to realize and fully internalize is that we (and by we I continue to mean "all sufficiently intelligent entities/systems") are emphatically not single-goal systems. Further, the means/path that we use to achieve a particular goal has a very high probability of affecting the path/means that we must use to accomplish subsequent goals -- as well as the likely success rate of those goals. Unintelligent systems/entities simply do not recognize this fact -- particularly since it probably interferes with their immediate goal-seeking behavior. Insufficiently intelligent systems/entities (or systems/entities under sufficient duress) are not going to have the foresight (or the time for foresight) to recognize all the implications of this fact and will therefore deviate from unseen optimal goal-seeking behavior in favor of faster/more obvious (though ultimately less optimal) paths. Borderline intelligent systems/entities under good conditions are going to try to tend in the directions suggested by this fact -- it is, after all, the ultimate in goal-seeking behavior -- but finding the optimal path/direction becomes increasingly difficult as the horizon expands. And this is, in fact, the situation that we are all in and debating about. As a collection of multi-goal systems/entities, how do the individual "we"s optimize our likelihood of achieving our goals? Clearly, we do not want some Unfriendly AGI coming along and preventing our goals by wiping us out or perverting our internal goal structure. = = = = = Now, I've just attempted to sneak a critical part of the answer right past everyone with my plea . . . . so let's go back and review it in slow-motion. :-) Part of our environment is that we have peers. And peers become resources towards our goals when we have common or compatible goals. Any unimaginably intelligent system/entity surrounded by peers is certainly going to work with it's peers wherever possible. Society/community is a feature that is critically important to Friendliness -- and this shows up in *many* places in evolution (if you're intelligent enough and can see beyond the "red in tooth and claw"). Note also that this can also (obviously) be easily and profitably extended to sub-peers (entities below a peer status) as long as the sub-peer can be convinced to interact in manner such that they are a net positive to the super-intelligences goals. Now, one of the assumptions of the Friendliness debate is that current-day humans are going to be sub-peers to the coming mind-children -- possibly/probably sub-sub-sub-...-peers. That leaves us in the situation of probably needing to interact in a manner such that we are a net positive to the super-intelligence's goals. Fortunately, it is my contention (which should be obvious by the end of the paper) that a Friendly sub-peer is *always* a resource and that Friendly behavior towards that sub-peer (our goal) is optimal for the super-intelligence. Thus, if we can get both ourselves and our mind-children to a Friendly state -- it should be reassuringly self-reinforcing from there on out. Of course, the big bugaboo to this whole theory is whether it will be too onerous for humans to be Friendly. Eliezer's vision of a Friendly future is that humans don't have to be Friendly -- only the AGIs do. My contention is that you don't get the Friendly attractor without all of the parties involved being Friendly -- which is why I'm so down on Eliezer's vision. Under my vision of Friendliness, entities that aren't Friendly generally don't receive behavior that would be regarded as Friendly. Now, note a critical feature of my arguments -- I am *NOT* trying to constrain the goals of my goal-seeking entity/entities (as in Eliezer's vision of Friendliness). I am trying to prove that *any* sufficiently intelligent multi-goal entity will find Friendliness an attractor because it promotes it's own goal-seeking behavior. Friendliness, in effect and assuming that it can be made coherent and consistent, is an optimal subgoal for all non-conflicting goals (and thus, in the aggregate of a large number of varying goals). So, as I said, if we can get both ourselves and our mind-children to a Friendly state -- it should be reassuringly self-reinforcing from there on out. TAKE-AWAY: Friendliness is an attractor because it IS equivalent to "enlightened self-interest" -- but it only works where all entities involved are Friendly. PART 3 will answer part of "What is Friendly behavior?" by answering "What is in the set of "horrible nasty thing[s]?". ----- Original Message ----- From: Mark Waser To: [email protected] Sent: Thursday, March 06, 2008 10:01 AM Subject: Re: [agi] What should we do to be prepared? Hmm. Bummer. No new feedback. I wonder if a) I'm still in "Well duh" land, b) I'm so totally off the mark that I'm not even worth replying to, or c) <I hope> being given enough rope to hang myself. :-) Since I haven't seen any feedback, I think I'm going to divert to a section that I'm not quite sure where it goes but I think that it might belong here . . . . Interlude 1 Since I'm describing Friendliness as an attractor in state space, I probably should describe the state space some and answer why we haven't fallen into the attractor already. The answer to latter is a combination of the facts that a.. Friendliness is only an attractor for a certain class of beings (the sufficiently intelligent). b.. It does take time/effort for the borderline sufficiently intelligent (i.e. us) to sense/figure out exactly where the attractor is (much less move to it). c.. We already are heading in the direction of Friendliness (or alternatively, Friendliness is in the direction of our most enlightened thinkers). and most importantly a.. In the vast, VAST majority of cases, Friendliness is *NOT* on the shortest path to any single goal. ------------------------------------------------------------------------------ agi | Archives | Modify Your Subscription ------------------------------------------- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
