Re: [agi] What should we do to be prepared?
On Mon, Mar 10, 2008 at 3:04 AM, Mark Waser [EMAIL PROTECTED] wrote: 1) If I physically destroy every other intelligent thing, what is going to threaten me? Given the size of the universe, how can you possibly destroy every other intelligent thing (and be sure that no others ever successfully arise without you crushing them too)? I can destroy all Earth-originated life if I start early enough. If there is something else out there, it can similarly be hostile and try destroy me if it can, without listening to any friendliness prayer. Plus, it seems like an awfully lonely universe. I don't want to live there even if I could somehow do it. I can upload what I can and/or initiate new intelligent entities inside controlled virtual environments. Also, if you crush them all, you can't have them later for allies, friends, and co-workers. It just doesn't seem like a bright move unless you truly can't avoid it. See my above arguments about why comparative advantage doesn't work in this case. I can produce ideal slaves that are no less able than potential allies, but don't have agenda of their own. -- Vladimir Nesov [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] What should we do to be prepared?
Mark Waser wrote: Part 4. ... Eventually, you're going to get down to Don't mess with anyone's goals, be forced to add the clause unless absolutely necessary, and then have to fight over what when absolutely necessary means. But what we've got here is what I would call the goal of a Friendly society -- */Don't mess with anyone's goals unless absolutely necessary/* and I would call this a huge amount of progress. Along with a fight over when absolutely necessary there could easily be a fight over mess with. Note how often we mess with others goals. Example 1: driving down the road encountering a person who appears to be lost. If you stop to help them, you are messing with their goal of the moment which is probably to figure out where they are. Is it absolutely necessary to help them? probably not since they likely have a cell phone or two... Example 2: You ask a child what they are frustrated about. If they explain the problem they are trying to solve - their goal - and then you offer an opinion, you might easily be messing. One could speculate that the messing was welcome, but it is risky if the law of the land is don't mess unless necessary. Example 3: You decide to carry a sign in public showing either that you are pro choice or pro life. Evidently you are there to mess with the goals and intents that others might have. Taboo? An expressed opinion about someones goal could be considered messing with it. Lawyers are about the only thing sure about the future! --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Recap/Summary/Thesis Statement
It *might* get stuck in bad territory, but can you make an argument why there is a *significant* chance of that happening? Not off the top of my head. I'm just playing it better safe than sorry since, as far as I can tell, there *may* be a significant chance of it happening. Also, I'm not concerned about it getting *stuck* in bad territory, I am more concerned about just transiting bad territory and destroying humanity on the way through. One thing that I think most of will agree on is that if things did work as Eliezer intended, things certainly could go very wrong if it turns out that the vast majority of people -- when smarter, more the people they wish they could be, as if they grew up more together ... -- are extremely unfriendly in approximately the same way (so that their extrapolated volition is coherent and may be acted upon). Our meanderings through state space would then head into very undesirable territory. (This is the people turn out to be evil and screw it all up scenario.) Your approach suffers from a similar weakness though, since it would suffer under the seeming friendly people turn out to be evil and screw it all up before there are non-human intelligent friendlies to save us scenario. But my approach has the advantage that it proves that Friendliness is in those evil people's self-interest so *maybe* we can convert them before they do us in. I'm not claiming that my approach is perfect or fool-proof. I'm just claiming that it's better than anything else thus far proposed. Which, if either, of 'including all of humanity' rather than just 'friendly humanity', or 'excluding non-human friendlies (initially)' do you see as the greater risk? I see 'excluding non-human friendlies (initially)' as a tremendously greater risk. I think that the proportionality aspect of Friendliness will keep the non-Friendly portion of humanity safe as we move towards Friendliness. Actually, let me rephrase your question and turn it around -- Which, if either, of 'not protecting all of humanity from Friendlies rather than just friendly humanity' or 'being actively unfriendly' do you see as a greater risk? Or is there some other aspect of Eliezer's approach that especially concerns you and motivates your alternative approach? The lack of self-reinforcing stability under errors and/or outside forces is also especially concerning and was my initially motivation for my vision. Thanks for continuing to answer my barrage of questions. No. Thank you for the continued intelligent feedback. I'm disappointed by all the people who aren't interested in participating until they can get a link to the final paper without any effort. This is still very much a work in progress with respect to the best way to present it and the only way I can improve it is with decent feedback -- which is therefore *much* appreciated. --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
[agi] Some thoughts of an AGI designer
I find myself totally bemused by the recent discussion of AGI friendliness. I am in sympathy with some aspects of Mark's position, but I also see a serious problem running through the whole debate: everyone is making statements based on unstated assumptions about the motivations of AGI systems. EVERYTHING depends on what assumptions you make, and yet each voice in this debate is talking as if their own assumption can be taken for granted. The three most common of these assumptions are: 1) That it will have the same motivations as humans, but with a tendency toward the worst that we show. 2) That it will have some kind of Gotta Optimize My Utility Function motivation. 3) That it will have an intrinsic urge to increase the power of its own computational machinery. There are other assumptions, but these seem to be the big three. So what I hear is a series of statements that are analogous to: Well, since the AGI will be bright yellow, it will clearly do this and this and this.. Well, since the AGI will be a dull sort of Cambridge blue, it will clearly do this and this and this.. Well, since the AGI will be orange, it will clearly do this and this and this.. (Except, of course, that nobody is actually coming right out and saying what color of AGI they assume.) In the past I have argued strenuously that (a) you cannot divorce a discussion of friendliness from a discussion of what design of AGI you are talking about, and (b) some assumptions about AGI motivation are extremely incoherent. And yet in spite of all my efforts that I have made, there seems to be no acknowledgement of the importance of these two points. Richard Loosemore --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Some thoughts of an AGI designer
The three most common of these assumptions are: 1) That it will have the same motivations as humans, but with a tendency toward the worst that we show. 2) That it will have some kind of Gotta Optimize My Utility Function motivation. 3) That it will have an intrinsic urge to increase the power of its own computational machinery. There are other assumptions, but these seem to be the big three. And IMO, the truth is likely to be more complex... For instance, a Novamente-based AGI will have an explicit utility function, but only a percentage of the system's activity will be directly oriented toward fulfilling this utility function Some of the system's activity will be spontaneous ... i.e. only implicitly goal-oriented .. and as such may involve some imitation of human motivation, and plenty of radically non-human stuff... ben g --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Some thoughts of an AGI designer
On Mon, Mar 10, 2008 at 5:47 PM, Richard Loosemore [EMAIL PROTECTED] wrote: In the past I have argued strenuously that (a) you cannot divorce a discussion of friendliness from a discussion of what design of AGI you are talking about, and (b) some assumptions about AGI motivation are extremely incoherent. And yet in spite of all my efforts that I have made, there seems to be no acknowledgement of the importance of these two points. Would it be so easy to reliably transfer semitechnical understanding, and better yet to check its rationality... -- Vladimir Nesov [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Some thoughts of an AGI designer
I am in sympathy with some aspects of Mark's position, but I also see a serious problem running through the whole debate: everyone is making statements based on unstated assumptions about the motivations of AGI systems. Bummer. I thought that I had been clearer about my assumptions. Let me try to concisely point them out again and see if you can show me where I have additional assumptions that I'm not aware that I'm making (which I would appreciate very much). Assumption - The AGI will be a goal-seeking entity. And I think that is it.:-) EVERYTHING depends on what assumptions you make, and yet each voice in this debate is talking as if their own assumption can be taken for granted. I agree with you and am really trying to avoid this. I will address your specific examples below and would appreciate any others that you can point out. The three most common of these assumptions are: 1) That it will have the same motivations as humans, but with a tendency toward the worst that we show. I don't believe that I'm doing this. I believe that all goal-seeking generally tends to be optimized by certain behaviors (the Omohundro drives). I believe that humans show many of these behaviors because these behaviors are relatively optimal in relation to the alternatives (and because humans are relatively optimal). But I also believe that the AGI will also have dramatically different motivations from humans where the human motivations were evolved stepping stones that were on the necessary and optimal path for one environment but haven't been eliminated now that they are unnecessary and sub-optimal in the current environment/society (Richard's the worst that we show). 2) That it will have some kind of Gotta Optimize My Utility Function motivation. I agree with the statement but I believe that it is a logical follow-on to my assumption that the AGI is a goal-seeking entity (i.e. it's an Omohundro drive). Would you agree, Richard? 3) That it will have an intrinsic urge to increase the power of its own computational machinery. Again, I agree with the statement but I believe that it is a logical follow-on to my single initial assumption (i.e. it's another Omohundro drive). Wouldn't you agree? There are other assumptions, but these seem to be the big three. And I would love to go through all of them, actually (or debate one of my answers above). So what I hear is a series of statements snip (Except, of course, that nobody is actually coming right out and saying what color of AGI they assume.) I thought that I pretty explicitly was . . . . :-( In the past I have argued strenuously that (a) you cannot divorce a discussion of friendliness from a discussion of what design of AGI you are talking about, And I have reached the conclusion that you are somewhat incorrect. I believe that goal-seeking entities OF ANY DESIGN of sufficient intelligence (goal-achieving ability) will see an attractor in my particular vision of Friendliness (which I'm deriving by *assuming* the attractor and working backwards from there -- which I guess you could call a second assumption if you *really* had to ;-). and (b) some assumptions about AGI motivation are extremely incoherent. If you perceive me as incoherent, please point out where. My primary AGI motivation is self-interest (defined as achievement of *MY* goals -- which directly derives from my assumption that the AGI will be a goal-seeking entity). All other motivations are clearly logically derived from that primary motivation. If you see an example where this doesn't appear to be the case, *please* flag it for me (since I need to fix it :-). And yet in spite of all my efforts that I have made, there seems to be no acknowledgement of the importance of these two points. I think that I've acknowledged both in the past and will continue to do so (despite the fact that I am now somewhat debating the first point -- more the letter than the spirit :-). --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] What should we do to be prepared?
On Mon, Mar 10, 2008 at 6:13 PM, Mark Waser [EMAIL PROTECTED] wrote: I can destroy all Earth-originated life if I start early enough. If there is something else out there, it can similarly be hostile and try destroy me if it can, without listening to any friendliness prayer. All definitely true. The only advantage to my approach is that you *have* a friendliness prayer that *might* convince them to leave you alone. Do you have any better alternative to stop a vastly superior power? I'll bet not. What if they are secular deities and send believers to Hell? I can upload what I can and/or initiate new intelligent entities inside controlled virtual environments. You can but doing so requires effort and you're tremendously unlikely to get the richness and variety that you would get if you just allowed evolution to do the work throughout the universe. Why are you voluntarily impoverishing yourself? That's *not* in your self-interest. Virtual environment is almost as powerful as physical. Simply converting enough matter to appropriate variety of computronium shouldn't require too much effort. See my above arguments about why comparative advantage doesn't work in this case. I can produce ideal slaves that are no less able than potential allies, but don't have agenda of their own. Producing slaves takes resources/effort. I feel that you underestimate the power of generally intelligent tools. -- Vladimir Nesov [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Some thoughts of an AGI designer
For instance, a Novamente-based AGI will have an explicit utility function, but only a percentage of the system's activity will be directly oriented toward fulfilling this utility function Some of the system's activity will be spontaneous ... i.e. only implicitly goal-oriented .. and as such may involve some imitation of human motivation, and plenty of radically non-human stuff... Which, as Eliezer has pointed out, sounds dangerous as all hell unless you have some reason to assume that it wouldn't be (like being sure that the AGI sees and believes that Friendliness is in it's own self-interest). --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Some thoughts of an AGI designer
Mark Waser wrote: I am in sympathy with some aspects of Mark's position, but I also see a serious problem running through the whole debate: everyone is making statements based on unstated assumptions about the motivations of AGI systems. Bummer. I thought that I had been clearer about my assumptions. Let me try to concisely point them out again and see if you can show me where I have additional assumptions that I'm not aware that I'm making (which I would appreciate very much). Assumption - The AGI will be a goal-seeking entity. And I think that is it.:-) Okay, I can use that as an illustration of what I am getting at. There are two main things. One is that the statement The AGI will be a goal-seeking entity has many different interpretations, ad I am arguing that these different interpretations have a massive impact on what kind of behavior you can expect to see. It is almost impossible to list all the different interpretations, but two of the more extreme variants are the two that I have described before: a Goal-Stack system in which the goals are represented in the same form as the knowledge that the system stores, and a Motivational Emotional System which biasses the functioning of the system and is intimately connected with the development of its knowledge. The GS system has the dangerous feature that any old fool could go in and rewrite the top level goal so it reads make as much computronium as possible or cultivate dandelions or learn how to do crochet. The MES system, on the other hand, can be set up to have values such as ours and to feel empathy with human beings, and once set up that way you would have to re-grow the system before you could get it to have some other set of values. Clearly, these two interpretations of The AGI will be a goal-seeking entity have such different properties that, unless there is detailed clarification of what the meaning is, we cannot continue to discuss what they would do. My second point is that some possible choices of the meaning of The AGI will be a goal-seeking entity will actually not cash out into a coherent machine design, so we would be wasting our time if we considered how that kind of AGI would behave. In particular, there are severe doubts about whether the Goal-Stack type of system can ever make it up to the level of a full intelligence. I'll go one further on that: I think that one of the main reasons we have trouble getting AI systems to be AGI is precisely because we have not yet realised that they need to be driven by something more than a Goal Stack. It is not the only reason, but its a big one. So the message is: we need to know exactly details of the AGI's motivation system (The AGI will be a goal-seeking entity is not specific enough), and we need to then be sure that the details we give are going to lead to a type of AGI that can actually be an AGI. These questions, I think, are the real battleground. BTW, this is not a direct attack on what you were saying, because I believe that there is a version of what you are saying (about an intrinsic tendency toward a Friendliness attractor) that I agree with. My problem is that so much of the current discussion is tangled up with hidden assumptions that I think that the interesting part of your message is getting lost. EVERYTHING depends on what assumptions you make, and yet each voice in this debate is talking as if their own assumption can be taken for granted. I agree with you and am really trying to avoid this. I will address your specific examples below and would appreciate any others that you can point out. The three most common of these assumptions are: 1) That it will have the same motivations as humans, but with a tendency toward the worst that we show. I don't believe that I'm doing this. I believe that all goal-seeking generally tends to be optimized by certain behaviors (the Omohundro drives). I believe that humans show many of these behaviors because these behaviors are relatively optimal in relation to the alternatives (and because humans are relatively optimal). But I also believe that the AGI will also have dramatically different motivations from humans where the human motivations were evolved stepping stones that were on the necessary and optimal path for one environment but haven't been eliminated now that they are unnecessary and sub-optimal in the current environment/society (Richard's the worst that we show). I am in complete disagreement with Omuhundro's idea that there are a canonical set of drives. This is like saying that there is a canonical set of colors that AGIs will come in: Cambridge Blue, Lemon Yellow and True Black. What color the thing is will be what color you decide to paint it! Ditto for its goals and motivations: what you decide to put into it is what it does, so I cannot make any sense of statements like I also believe that the AGI will also
Re: [agi] What should we do to be prepared?
On Mon, Mar 10, 2008 at 8:10 PM, Mark Waser [EMAIL PROTECTED] wrote: Information Theory is generally accepted as correct and clearly indicates that you are wrong. Note that you are trying to use a technical term in a non-technical way to fight a non-technical argument. Do you really think that I'm asserting that virtual environment can be *exactly* as capable as physical environment? All interesting stuff is going to be computational anyway. My requirement only limits potentially invasive control over physical matter (in other words, influencing other computational processes to which access is denied). In most cases, computation should be implementable on universal substrate without too much overhead, and if it needs something completely different, captive system can order custom physical devices verified to be unable to do anything but computation. We are doing it already, by trashing old PCs and running Windows 98 in virtual machines, in those rare circumstances where killing them altogether still isn't optimal. -- Vladimir Nesov [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Some thoughts of an AGI designer
First off -- yours was a really helpful post. Thank you! I think that I need to add a word to my initial assumption . . . . Assumption - The AGI will be an optimizing goal-seeking entity. There are two main things. One is that the statement The AGI will be a goal-seeking entity has many different interpretations, ad I am arguing that these different interpretations have a massive impact on what kind of behavior you can expect to see. I disagree that it has many interpretations. I am willing to agree that my original assumption phrase didn't sufficiently circumscribe the available space of entities to justify some of my further reasoning (most particularly because Omohundro drives *ASSUME* an optimizing entity -- my bad for not picking that up before :-). The MES system, on the other hand, can be set up to have values such as ours and to feel empathy with human beings, and once set up that way you would have to re-grow the system before you could get it to have some other set of values. As a system that (arguably) finds itself less able to massively (and possibly dangerously) optimize itself, the MES system is indeed less subject to my reasoning to the extent that it is not able to optimize itself (or, to the extent that it is constrained in optimizing itself). On the other hand, to the extent that the MES system *IS* able to optimize itself, I would contend that my Omohundro-drive-based reasoning is valid and correct. Clearly, these two interpretations of The AGI will be a goal-seeking entity have such different properties that, unless there is detailed clarification of what the meaning is, we cannot continue to discuss what they would do. Hopefully my statement just above will convince you that we can continue since we really aren't arguing different properties -- merely the degree to which a system can self-optimize. That should not prevent a useful discussion. My second point is that some possible choices of the meaning of The AGI will be a goal-seeking entity will actually not cash out into a coherent machine design, so we would be wasting our time if we considered how that kind of AGI would behave. I disagree. Even if 50% of the possible choices can't be implemented, then I still don't believe that we shouldn't investigate the class as a whole. It has interesting characteristics that lead me to believe that the remaining 50% of implementable choices may hit the jackpot. In particular, there are severe doubts about whether the Goal-Stack type of system can ever make it up to the level of a full intelligence. Ah. But this is an intelligence argument rather than a Friendliness argument and doubly irrelevant because I am not proposing or nor assuming a goal-stack. I prefer your system of a large, diffuse set of (often but not always simple) goals and constraints and don't believe it to be at all contrary to what I am envisioning. I particularly like it because *I BELIEVE* that such an approach is much more likely to produce a safe, orderly/smooth transition into my Friendliness attractor that a relatively easily breakable Goal-Stack system. I'll go one further on that: I think that one of the main reasons we have trouble getting AI systems to be AGI is precisely because we have not yet realised that they need to be driven by something more than a Goal Stack. It is not the only reason, but its a big one. I agree with you (but it's still not relevant to my argument:-). So the message is: we need to know exactly details of the AGI's motivation system (The AGI will be a goal-seeking entity is not specific enough), and we need to then be sure that the details we give are going to lead to a type of AGI that can actually be an AGI. No, we don't need to know the details. I'm contending that my vision/theory applies regardless of the details. If you don't believe so, please supply contrary details and I'll do whatever necessary to handle them.:-) These questions, I think, are the real battleground. We'll see . . . . :-) BTW, this is not a direct attack on what you were saying, Actually, I prefer a direct attack:-). I should have declared Crocker's rules with the Waste of my time exception (i.e. I reserve the right to be rude to anyone who both is rude *and* wastes my time :-). My problem is that so much of the current discussion is tangled up with hidden assumptions that I think that the interesting part of your message is getting lost. So let's drag those puppies into the light! This is not an easy message. It touches on (and, I believe, revises) one helluva lot. That's why I laugh when someone just wants a link to the completed paper. Trust me -- the wording on the completed paper changes virtually every time there is an e-mail on the subject. And I *don't* want people skipping ahead to the punch line if I'm not explaining it well enough at the beginning --
Re: [agi] What should we do to be prepared?
Note that you are trying to use a technical term in a non-technical way to fight a non-technical argument. Do you really think that I'm asserting that virtual environment can be *exactly* as capable as physical environment? No, I think that you're asserting that the virtual environment is close enough to as capable as the physical environment without spending significant resources that the difference doesn't matter. And I'm having problems with the without spending significant resources part, not the that the difference doesn't matter part. All interesting stuff is going to be computational anyway. So, since the physical world can perform interesting computation automatically without any resources, why are you throwing the computational aspect of the physical world away? In most cases, computation should be implementable on universal substrate without too much overhead How do we get from here to there? Without a provable path, it's all just magical hand-waving to me. (I like it but it's ultimately an unsatifying illusion) --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] What should we do to be prepared?
On Mon, Mar 10, 2008 at 11:36 PM, Mark Waser [EMAIL PROTECTED] wrote: Note that you are trying to use a technical term in a non-technical way to fight a non-technical argument. Do you really think that I'm asserting that virtual environment can be *exactly* as capable as physical environment? No, I think that you're asserting that the virtual environment is close enough to as capable as the physical environment without spending significant resources that the difference doesn't matter. And I'm having problems with the without spending significant resources part, not the that the difference doesn't matter part. I use significant in about the same sense as something that matters, so it's merely a terminological mismatch. All interesting stuff is going to be computational anyway. So, since the physical world can perform interesting computation automatically without any resources, why are you throwing the computational aspect of the physical world away? I only add one restriction on allowed physical structures to be constructed for captive systems: they must be verifiably unable to affect other computations that they are not allowed to. I'm sure that for computational efficiency it should be a very strict limitation. So any custom computers are allowed, as long as they can't morph into berserker probes and the like. In most cases, computation should be implementable on universal substrate without too much overhead How do we get from here to there? Without a provable path, it's all just magical hand-waving to me. (I like it but it's ultimately an unsatifying illusion) It's an independent statement. -- Vladimir Nesov [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] What should we do to be prepared?
errata: On Tue, Mar 11, 2008 at 12:13 AM, Vladimir Nesov [EMAIL PROTECTED] wrote: I'm sure that for computational efficiency it should be a very strict limitation. it *shouldn't* be a very strict limitation -- Vladimir Nesov [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] What should we do to be prepared?
On Tue, Mar 11, 2008 at 12:37 AM, Mark Waser [EMAIL PROTECTED] wrote: How do we get from here to there? Without a provable path, it's all just magical hand-waving to me. (I like it but it's ultimately an unsatifying illusion) It's an independent statement. No, it isn't an independent statement. If you can't get there (because it is totally unfeasible to do so) then it totally invalidates your argument. My second point that you omitted from this response doesn't need there to be universal substrate, which is what I mean. Ditto for significant resources. -- Vladimir Nesov [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Some thoughts of an AGI designer
Mark Waser wrote: ... The motivation that is in the system is I want to achieve *my* goals. The goals that are in the system I deem to be entirely irrelevant UNLESS they are deliberately and directly contrary to Friendliness. I am contending that, unless the initial goals are deliberately and directly contrary to Friendliness, an optimizing system's motivation of achieve *my* goals (over a large enough set of goals) will eventually cause it to finally converge on the goal of Friendliness since Friendliness is the universal super-meta-subgoal of all it's other goals (and it's optimizing will also drive it up to the necessary intelligence to understand Friendliness). Of course, it may take a while since we humans are still in the middle of it . . . . but hopefully we're almost there.;-) ... Mark I think here we need to consider A. Maslow's hierarchy of needs. That an AGI won't have the same needs as a human is, I suppose, obvious, but I think it's still true that it will have a hierarchy (which isn't strictly a hierarchy). I.e., it will have a large set of motives, and which it is seeking to satisfy at any moment will alter as the satisfaction of the previous most urgent motive changes. It it were a human we could say that breathing was the most urgent need...but usually it's so well satisfied that we don't even think about it. Motives, then, will have satisficing as their aim. Only aberrant mental functions will attempt to increase the satisfying of some particular goal without limit. (Note that some drives in humans seem to occasionally go into that satisfy increasingly without limit mode, like quest for wealth or power, but in most sane people these are reined in. This seems to indicate that there is a real danger here...and also that it can be avoided.) --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] What should we do to be prepared?
My second point that you omitted from this response doesn't need there to be universal substrate, which is what I mean. Ditto for significant resources. I didn't omit your second point, I covered it as part of the difference between our views. You believe that certain tasks/options are relatively easy that I believe to be infeasible without more resources than you can possibly imagine. I can't prove a negative but if you were more familiar with Information Theory, you might get a better handle on why your approach is ludicrously expensive. --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] What should we do to be prepared?
Part 5. The nature of evil or The good, the bad, and the evil Since we've got the (slightly revised :-) goal of a Friendly individual and the Friendly society -- Don't act contrary to anyone's goals unless absolutely necessary -- we now can evaluate actions as good or bad in relation to that goal. *Anything* that doesn't act contrary to someone's goals is GOOD. Anything that acts contrary to anyone's goals is BAD to the extent that it is not absolutely necessary. EVIL is the special case where an entity *knowingly and intentionally* acts contrary to someone's goals when it isn't absolutely necessary for one of the individual's own primary goals. This is the *intentional* direct opposite of the goal of Friendliness and it is in the Friendly society's best interest to make this as unappealing as possible. *Any* sufficiently effective Friendly society will *ENSURE* that the expected utility of EVIL is negative by raising the consequences of (sanctions for) EVIL to a level where it is clearly apparent that EVIL is not in an entity's self-interest. The reason why humans are frequently told Evil doesn't mean stupid is because many of us sense at a very deep level that, in a sufficiently efficient ethical/Friendly society, EVIL *is* stupid (in that it is not in an entity's self-interest). It's just a shame that our society is not sufficiently efficiently ethical/Friendly -- YET! Vladimir's crush-them-all is *very* bad. It is promoting that society's goal of safety (which is a valid, worthwhile goal) but it is refusing to recognize that it is *NOT* always necessary and that there are other, better ways to achieve that goal (not to mention the fact that the aggressor society would probably even benefit more by not destroying the lesser society's). My impression is that Vladimir is knowingly and intentionally acting contrary to someone else's goals when it isn't absolutely necessary because it is simply more convenient for him (because it certainly isn't safer since it invites sanctions like those following). This is EVIL. If I'm a large enough, effective enough Friendly society, Vladimir's best approach is going to be to immediately willingly convert to Friendliness and voluntarily undertake reparations that are rigorous enough that their negative utility is just greater than the total expected utility of the greater of either a) the expected utility of any destroyed civilizations or b) the utility that his society derived by destroying the civilization. If Vladimir doesn't immediately convert and undertake reparations, the cost and effort of making him do so will be added to the reparations. These reparations should be designed to assist every other Friendly *without* harming Vladimir's society EXCEPT for the cost and effort that are diverted from Vladimir's goals. Now, there is one escape hatch that immediately springs to the mind of the UnFriendly that I am now explicitly closing . . . . Generic sub-goals are *not* absolutely necessary. A Friendly entity does not act contrary to someone's goals simply because it is convenient, because it gives them more power, or because it feels good. In fact, it should be noted that allowing generic subgoals to override other's goals is probably the root of all evil (If you thought that it was money, you're partially correct. Money is Power is a generic sub-goal). Pleasure is a particularly pernicious sub-goal. Pleasure is evolutionarily adaptive when you feel good when you do something that is pro-survival. It is most frequently an indicator that you are doing something that is pro-survival -- but as such, seeking pleasure is merely a subgoal to the primary goal of survival. There's also a particular problem in that pleasure evolutionarily lags behind current circumstances and many things that are pleasurable because they were pro-survival in the past are now contrary to survival or most other goals(particularly when practiced to excess) in the present. Wire-heading is a particularly obvious example of this. Every other goal of the addicted wire-head is thrown away in search of a sub-goal that leads to no goal -- not even survival. I do want to be clear that there is nothing inherently wrong in seeking pleasure (as the Puritans would have it). Pleasure can rest, relax, and de-stress you so that you can achieve other goals even if it has no other purpose. The problem is when the search for pleasure overrides your own goals (addiction) or those of others (evil unless provably addiction). TAKE-AWAYs: a.. EVIL is knowingly and intentionally acting contrary to someone's goals when it isn't necessary (most frequently in the name of some generic sub-goal like pleasure, power, or convenience). b.. The sufficiently efficient ethical/Friendly society WILL ensure that the expected utility of EVIL is negative (i.e. not in an entity's self-interest and, therefore, stupid) Part 6 will move
Re: [agi] Some thoughts of an AGI designer
I think here we need to consider A. Maslow's hierarchy of needs. That an AGI won't have the same needs as a human is, I suppose, obvious, but I think it's still true that it will have a hierarchy (which isn't strictly a hierarchy). I.e., it will have a large set of motives, and which it is seeking to satisfy at any moment will alter as the satisfaction of the previous most urgent motive changes. I agree with all of this. It it were a human we could say that breathing was the most urgent need...but usually it's so well satisfied that we don't even think about it. Motives, then, will have satisficing as their aim. Only aberrant mental functions will attempt to increase the satisfying of some particular goal without limit. (Note that some drives in humans seem to occasionally go into that satisfy increasingly without limit mode, like quest for wealth or power, but in most sane people these are reined in. This seems to indicate that there is a real danger here...and also that it can be avoided.) I agree this except that I believe that humans *frequently* aim to optimize rather than satisfy (frequently to their detriment -- in terms of happiness as well as in the real costs of performing the search past a simple satisfaction point). Also, quest for pleasure (a.k.a. addiction) is also distressingly frequent in humans. Do you think that any of this contradicts what I've written thus far? I don't immediately see any contradictions. --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Artificial general intelligence
On 27/02/2008, a [EMAIL PROTECTED] wrote: This causes real controversy in this discussion list, which pressures me to build my own AGI. How about joining effort with one of the existing AGI projects? --linas --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
