Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
On Monday 26 May 2008 09:55:14 am, Mark Waser wrote: Josh, Thank you very much for the pointers (and replying so rapidly). You're welcome -- but also lucky; I read/reply to this list a bit sporadically in general. You're very right that people misinterpret and over-extrapolate econ and game theory, but when properly understood and applied, they are a valuable tool for analyzing the forces shaping the further evolution of AGIs and indeed may be our only one. No. I would argue that there is a lot of good basic research into human and primate behavior that is more applicable since it's already been tested and requires less extrapolation (and visibly shows where a lot of current extrapoloation is just plain wrong). It's interesting that behavioral economics appeared only fairly recently, to study the ways in which humans act irrationally in their economic choices. (See Predictably Irrational by Dan Ariely, e.g.) But it's been observed for a while that people tend to act more rationally in economic settings than non-economic ones, and there's no reason to believe that we couldn't build an AI to act more rationally yet. In other words, actors in the economic world will be getting closer and closer to the classic economic agent as time goes by, and so classic econ will be a better description of the world than it is now. The true question is, how do you raise the niceness of *all* players and prevent defection -- because being the single bad guy is a winning strategy while being just one among many is horrible for everyone. Intelligence. You identify the bad guys and act nasty just to them. Finding ways to do this robustly and efficiently is the basis of human society. So, in simplistic computer simulations at least, evolution seems to go through a set of phases with different (and improving!) moral character. So why do so many people think evolution favors the exactly the opposite? Several reasons -- first being that evolution education and literacy in this country is crap, thanks to a century and a half of religious propaganda and activism. Another is that people tend to study evolution at whatever level that predation and arms races happen, and don't pay attention to the levels where cooperation does. Example: lions vs zebras -- ignoring the fact that the actual units of evolution are the genes, which have formed amazingly cooperative systems to create a lion or zebra in the first place. And even then, the marketplace can channel evolution in better ways. It's a quantum jump higher step on the moral ladder than the jungle... Miller and Drexler write: (http://www.agorics.com/Library/agoricpapers/ce/ce0.html) ... Ecology textbooks show networks of predator-prey relationships-called food webs-because they are important to understanding ecosystems; symbiosis webs have found no comparable role. Economics textbooks show networks of trading relationships circling the globe; networks of predatory or negative-sum relationships have found no comparable role. (Even criminal networks typically form cooperative black markets.) One cannot prove the absence of such spanning symbiotic webs in biology, or of negative-sum webs in the market; these systems are too complicated for any such proof. Instead, the argument here is evolutionary: that the concepts which come to dominate an evolved scientific field tend to reflect the phenomena which are actually relevant for understanding its subject matter. 4.5 Is this picture surprising? Nature is commonly viewed as harmonious and human markets as full of strife, yet the above comparison suggests the opposite. The psychological prominence of unusual phenomena may explain the apparent inversion of the common view. Symbiosis stands out in biology: we have all heard of the unusual relationship between crocodiles and the birds that pluck their parasites, but one hears less about the more common kind of relationship between crocodiles and each of the many animals they eat. Nor, in considering those birds, is one apt to dwell on the predatory relationship of the parasites to the crocodile or of the birds to the parasites. Symbiosis is unusual and interesting; predation is common and boring. Similarly, fraud and criminality stand out in markets. Newspapers report major instances of fraud and embezzlement, but pay little attention to each day's massive turnover of routinely satisfactory cereal, soap, and gasoline in retail trade. Crime is unusual and interesting; trade is common and boring. Psychological research indicates that human thought is subject to a systematic bias: vivid and interesting instances are more easily remembered, and easily remembered instances are thought to be more common [21]). Further, the press (and executives) like to describe peaceful competition for customer favor as if it were mortal combat, complete with wounds and rolling heads: again, vividness wins
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
And again, *thank you* for a great pointer! - Original Message - From: J Storrs Hall, PhD [EMAIL PROTECTED] To: agi@v2.listbox.com Sent: Tuesday, May 27, 2008 8:04 AM Subject: Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...] On Monday 26 May 2008 09:55:14 am, Mark Waser wrote: Josh, Thank you very much for the pointers (and replying so rapidly). You're welcome -- but also lucky; I read/reply to this list a bit sporadically in general. You're very right that people misinterpret and over-extrapolate econ and game theory, but when properly understood and applied, they are a valuable tool for analyzing the forces shaping the further evolution of AGIs and indeed may be our only one. No. I would argue that there is a lot of good basic research into human and primate behavior that is more applicable since it's already been tested and requires less extrapolation (and visibly shows where a lot of current extrapoloation is just plain wrong). It's interesting that behavioral economics appeared only fairly recently, to study the ways in which humans act irrationally in their economic choices. (See Predictably Irrational by Dan Ariely, e.g.) But it's been observed for a while that people tend to act more rationally in economic settings than non-economic ones, and there's no reason to believe that we couldn't build an AI to act more rationally yet. In other words, actors in the economic world will be getting closer and closer to the classic economic agent as time goes by, and so classic econ will be a better description of the world than it is now. The true question is, how do you raise the niceness of *all* players and prevent defection -- because being the single bad guy is a winning strategy while being just one among many is horrible for everyone. Intelligence. You identify the bad guys and act nasty just to them. Finding ways to do this robustly and efficiently is the basis of human society. So, in simplistic computer simulations at least, evolution seems to go through a set of phases with different (and improving!) moral character. So why do so many people think evolution favors the exactly the opposite? Several reasons -- first being that evolution education and literacy in this country is crap, thanks to a century and a half of religious propaganda and activism. Another is that people tend to study evolution at whatever level that predation and arms races happen, and don't pay attention to the levels where cooperation does. Example: lions vs zebras -- ignoring the fact that the actual units of evolution are the genes, which have formed amazingly cooperative systems to create a lion or zebra in the first place. And even then, the marketplace can channel evolution in better ways. It's a quantum jump higher step on the moral ladder than the jungle... Miller and Drexler write: (http://www.agorics.com/Library/agoricpapers/ce/ce0.html) ... Ecology textbooks show networks of predator-prey relationships-called food webs-because they are important to understanding ecosystems; symbiosis webs have found no comparable role. Economics textbooks show networks of trading relationships circling the globe; networks of predatory or negative-sum relationships have found no comparable role. (Even criminal networks typically form cooperative black markets.) One cannot prove the absence of such spanning symbiotic webs in biology, or of negative-sum webs in the market; these systems are too complicated for any such proof. Instead, the argument here is evolutionary: that the concepts which come to dominate an evolved scientific field tend to reflect the phenomena which are actually relevant for understanding its subject matter. 4.5 Is this picture surprising? Nature is commonly viewed as harmonious and human markets as full of strife, yet the above comparison suggests the opposite. The psychological prominence of unusual phenomena may explain the apparent inversion of the common view. Symbiosis stands out in biology: we have all heard of the unusual relationship between crocodiles and the birds that pluck their parasites, but one hears less about the more common kind of relationship between crocodiles and each of the many animals they eat. Nor, in considering those birds, is one apt to dwell on the predatory relationship of the parasites to the crocodile or of the birds to the parasites. Symbiosis is unusual and interesting; predation is common and boring. Similarly, fraud and criminality stand out in markets. Newspapers report major instances of fraud and embezzlement, but pay little attention to each day's massive turnover of routinely satisfactory cereal, soap, and gasoline in retail trade. Crime is unusual and interesting; trade is common and boring. Psychological research indicates that human thought is subject to a systematic bias: vivid and interesting instances are more easily remembered
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Josh, Thank you very much for the pointers (and replying so rapidly). You're very right that people misinterpret and over-extrapolate econ and game theory, but when properly understood and applied, they are a valuable tool for analyzing the forces shaping the further evolution of AGIs and indeed may be our only one. No. I would argue that there is a lot of good basic research into human and primate behavior that is more applicable since it's already been tested and requires less extrapolation (and visibly shows where a lot of current extrapoloation is just plain wrong). But in the long run, slightly nicer programs can out-compete slightly nastier ones, and then in turn be out-competed by slightly nicer ones yet. For example, in a simulation with ``noise,'' meaning that occasionally at random a ``cooperate'' is turned in to a ``defect,'' tit-for-tat gets hung up in feuds, and a generous version that occasionally forgives a defection does better--but only if the really nasty strategies have been knocked out by tit-for-tat first. Even better is a strategy called Pavlov, due to an extremely simple form of learning. Pavlov repeats its previous play if it ``won,'' and switches if it ``lost.'' In particular, it cooperates whenever both it and its opponent did the same thing the previous time--it's a true, if very primitive, ``cahooter.'' Pavlov also needs the underbrush to be cleared by a ``stern retaliatory strategy like tit-for-tat.'' Actually, I've seen this presented as a good clock analogy as well . . . . slightly nicer out-competes slightly nastier but once it hits a nasty enough program, it loses. The true question is, how do you raise the niceness of *all* players and prevent defection -- because being the single bad guy is a winning strategy while being just one among many is horrible for everyone. So, in simplistic computer simulations at least, evolution seems to go through a set of phases with different (and improving!) moral character. So why do so many people think evolution favors the exactly the opposite? --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
On Monday 26 May 2008 06:55:48 am, Mark Waser wrote: The problem with accepted economics and game theory is that in a proper scientific sense, they actually prove very little and certainly far, FAR less than people extrapolate them to mean (or worse yet, prove). Abusus non tollit usum. Oh Josh, I just love it when you speak Latin to me! It makes you seem s smart . . . . But, I don't understand your point. What argument against proper use do you believe that I'm making? Or, do you believe that Omohundro is making improper use of AEFGT? You're very right that people misinterpret and over-extrapolate econ and game theory, but when properly understood and applied, they are a valuable tool for analyzing the forces shaping the further evolution of AGIs and indeed may be our only one. Could you please give some references (or, at least, pointers to pointers) that show the existence of the moral ladder? I'd appreciate it and could use them for something else. Thanks! BAI p. 178-9: Further research into evolutionary game theory shows that the optimal strategy is strongly dependent on the environment constituted by other players. In a population of all two-state automata (of which tit-for-tat is one), a program by the name of GRIM is optimal. GRIM cooperates until its opponent defects just once, and always defects after that. The reason it does well is that the population has quite a few programs whose behavior is oblivious or random. Rather than trying to decipher them, it just shoots them all and lets evolution sort them out. Chances are Axelrod's original tournaments are a better window into parts of the real, biological evolutionary dynamic than are the later tournaments with generated agents. The reason is that genetic algorithms are still unable to produce anything nearly as sophisticated as human programmers. Thus GRIM, for example, gets a foothold in a crowd of unsophisticated opponents. It wouldn't do you any good to be forgiving or clear if the other program were random. But in the long run, slightly nicer programs can out-compete slightly nastier ones, and then in turn be out-competed by slightly nicer ones yet. For example, in a simulation with ``noise,'' meaning that occasionally at random a ``cooperate'' is turned in to a ``defect,'' tit-for-tat gets hung up in feuds, and a generous version that occasionally forgives a defection does better--but only if the really nasty strategies have been knocked out by tit-for-tat first. Even better is a strategy called Pavlov, due to an extremely simple form of learning. Pavlov repeats its previous play if it ``won,'' and switches if it ``lost.'' In particular, it cooperates whenever both it and its opponent did the same thing the previous time--it's a true, if very primitive, ``cahooter.'' Pavlov also needs the underbrush to be cleared by a ``stern retaliatory strategy like tit-for-tat.'' So, in simplistic computer simulations at least, evolution seems to go through a set of phases with different (and improving!) moral character. Karl Sigmund, Complex Adaptive Systems and the Evolution of Reciprocation , International Institute for Applied Systems Analysis Interim Report IR-98-100; see http://www.iiasa.ac.at. there's a lot of good material at http://jasss.soc.surrey.ac.uk/JASSS.html Also, I'm *clearly* not arguing his basic starting point or the econ references. I'm arguing his extrapolations. Particularly the fact that his ultimate point that he claims applies to all goal-based systems clearly does not apply to human beings. I think we're basically in agreement here. Josh --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
- Original Message From: Richard Loosemore [EMAIL PROTECTED] Richard Loosemore said: If you look at his paper carefully, you will see that at every step of the way he introduces assumptions as if they were obvious facts ... and in all the cases I have bothered to think through, these all stem from the fact that he has a particular kind of mechanism in mind (one which has a goal stack and a utility function). There are so many of these assertions pulled out of think air that I found it gave me a headache just to read the paper. ... But this is silly: where was his examination of the systems various motives? Where did he consider the difference between different implementations of the entire motivational mechanism (my distinction between GS and MES systems)? Nowhere. He just asserts, without argument, that the system would be obsessed, and that any attempt by us to put locks on the system would result in an arms race of measures and countermeasures. That is just one example of how he pulls conclusions out of thin air. --- Your argument about the difference between a GS and an MES system is a strawman argument. Omohundro never made the argument, nor did he touch on it as far as I can tell. I did not find his paper very interesting either, but you are the one who seems to be pulling conclusions out of thin air. You can introduce the GS vs MES argument if you want, but you cannot then argue from the implication that everyone has to refer to it or else stand guilty of pulling arguments out of thin air. His paper Nature of Self Improving Artificial Intelligence September 5, 2007, revised January 21, 2008 provides a lot of reasoning. I don't find the reasoning compelling, but the idea that he is just pulling conclusions out of thin air is just bluster. Jim Bromer --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
The paper can be found at http://selfawaresystems.files.wordpress.com/2008/01/nature_of_self_improving_ai.pdf Read the appendix, p37ff. He's not making arguments -- he's explaining, with a few pointers into the literature, some parts of completely standard and accepted economics and game theory. It's all very basic stuff. On Sunday 25 May 2008 06:26:59 am, Jim Bromer wrote: - Original Message From: Richard Loosemore [EMAIL PROTECTED] Richard Loosemore said: If you look at his paper carefully, you will see that at every step of the way he introduces assumptions as if they were obvious facts ... and in all the cases I have bothered to think through, these all stem from the fact that he has a particular kind of mechanism in mind (one which has a goal stack and a utility function). There are so many of these assertions pulled out of think air that I found it gave me a headache just to read the paper. ... But this is silly: where was his examination of the systems various motives? Where did he consider the difference between different implementations of the entire motivational mechanism (my distinction between GS and MES systems)? Nowhere. He just asserts, without argument, that the system would be obsessed, and that any attempt by us to put locks on the system would result in an arms race of measures and countermeasures. That is just one example of how he pulls conclusions out of thin air. --- Your argument about the difference between a GS and an MES system is a strawman argument. Omohundro never made the argument, nor did he touch on it as far as I can tell. I did not find his paper very interesting either, but you are the one who seems to be pulling conclusions out of thin air. You can introduce the GS vs MES argument if you want, but you cannot then argue from the implication that everyone has to refer to it or else stand guilty of pulling arguments out of thin air. His paper Nature of Self Improving Artificial Intelligence September 5, 2007, revised January 21, 2008 provides a lot of reasoning. I don't find the reasoning compelling, but the idea that he is just pulling conclusions out of thin air is just bluster. Jim Bromer --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?; Powered by Listbox: http://www.listbox.com --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Your argument about the difference between a GS and an MES system is a strawman argument. Omohundro never made the argument, nor did he touch on it as far as I can tell. I did not find his paper very interesting either, but you are the one who seems to be pulling conclusions out of thin air. You can introduce the GS vs MES argument if you want, but you cannot then argue from the implication that everyone has to refer to it or else stand guilty of pulling arguments out of thin air. I'm afraid that I have to agree with Jim here, Richard. Nothing you've said convinces me that GS vs. MES belongs in this argument at all. I disagree with Omohundro's final conclusions but believe that his arguments apply equally well -- in a short-sighted sense (see next e-mail) -- to either architecture. - Original Message - From: Jim Bromer To: agi@v2.listbox.com Sent: Sunday, May 25, 2008 6:26 AM Subject: Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...] - Original Message From: Richard Loosemore [EMAIL PROTECTED] Richard Loosemore said: If you look at his paper carefully, you will see that at every step of the way he introduces assumptions as if they were obvious facts ... and in all the cases I have bothered to think through, these all stem from the fact that he has a particular kind of mechanism in mind (one which has a goal stack and a utility function). There are so many of these assertions pulled out of think air that I found it gave me a headache just to read the paper. ... But this is silly: where was his examination of the systems various motives? Where did he consider the difference between different implementations of the entire motivational mechanism (my distinction between GS and MES systems)? Nowhere. He just asserts, without argument, that the system would be obsessed, and that any attempt by us to put locks on the system would result in an arms race of measures and countermeasures. That is just one example of how he pulls conclusions out of thin air. --- Your argument about the difference between a GS and an MES system is a strawman argument. Omohundro never made the argument, nor did he touch on it as far as I can tell. I did not find his paper very interesting either, but you are the one who seems to be pulling conclusions out of thin air. You can introduce the GS vs MES argument if you want, but you cannot then argue from the implication that everyone has to refer to it or else stand guilty of pulling arguments out of thin air. His paper Nature of Self Improving Artificial Intelligence September 5, 2007, revised January 21, 2008 provides a lot of reasoning. I don't find the reasoning compelling, but the idea that he is just pulling conclusions out of thin air is just bluster. Jim Bromer -- agi | Archives | Modify Your Subscription --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Read the appendix, p37ff. He's not making arguments -- he's explaining, with a few pointers into the literature, some parts of completely standard and accepted economics and game theory. It's all very basic stuff. The problem with accepted economics and game theory is that in a proper scientific sense, they actually prove very little and certainly far, FAR less than people extrapolate them to mean (or worse yet, prove). All of the scientific experiments in game theory are very, VERY limited and deal with entities with little memory in small, toy systems. If you extrapolate their results with no additional input and no emergent effects, you can end up with arguments like Omohundro's BUT claiming that this extrapolation *proves* anything is very poor science. It's just speculation/science fiction and there are any number of reasons to believe that Omohundro's theories are incorrect -- the largest one, of course, being If all goal-based systems end up evil, why isn't every *naturally* intelligent entity evil? - Original Message - From: J Storrs Hall, PhD [EMAIL PROTECTED] To: agi@v2.listbox.com Sent: Sunday, May 25, 2008 8:14 AM Subject: Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...] The paper can be found at http://selfawaresystems.files.wordpress.com/2008/01/nature_of_self_improving_ai.pdf Read the appendix, p37ff. He's not making arguments -- he's explaining, with a few pointers into the literature, some parts of completely standard and accepted economics and game theory. It's all very basic stuff. On Sunday 25 May 2008 06:26:59 am, Jim Bromer wrote: - Original Message From: Richard Loosemore [EMAIL PROTECTED] Richard Loosemore said: If you look at his paper carefully, you will see that at every step of the way he introduces assumptions as if they were obvious facts ... and in all the cases I have bothered to think through, these all stem from the fact that he has a particular kind of mechanism in mind (one which has a goal stack and a utility function). There are so many of these assertions pulled out of think air that I found it gave me a headache just to read the paper. ... But this is silly: where was his examination of the systems various motives? Where did he consider the difference between different implementations of the entire motivational mechanism (my distinction between GS and MES systems)? Nowhere. He just asserts, without argument, that the system would be obsessed, and that any attempt by us to put locks on the system would result in an arms race of measures and countermeasures. That is just one example of how he pulls conclusions out of thin air. --- Your argument about the difference between a GS and an MES system is a strawman argument. Omohundro never made the argument, nor did he touch on it as far as I can tell. I did not find his paper very interesting either, but you are the one who seems to be pulling conclusions out of thin air. You can introduce the GS vs MES argument if you want, but you cannot then argue from the implication that everyone has to refer to it or else stand guilty of pulling arguments out of thin air. His paper Nature of Self Improving Artificial Intelligence September 5, 2007, revised January 21, 2008 provides a lot of reasoning. I don't find the reasoning compelling, but the idea that he is just pulling conclusions out of thin air is just bluster. Jim Bromer --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?; Powered by Listbox: http://www.listbox.com --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?; Powered by Listbox: http://www.listbox.com --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Um. I *really* need to point out that statements like transhumanists. They have this sort of gut emotional belief that self improvement is all good are really nasty, unwarranted bigotry. - Original Message - From: [EMAIL PROTECTED] To: agi@v2.listbox.com Sent: Saturday, May 24, 2008 10:03 PM Subject: Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...] I was sitting in the room when they were talking about it and I didn't feel like speaking up at the time (why break my streak?) but I felt he was just wrong. It seemed like you could boil the claim down to this: If you are sufficiently advanced, and you have a goal and some ability to meet goal, you will also want to improve your ability to achieve that goal. But there could be many reawsons why you would not want to improve (like maybe not wanting to devote the resources or risk any of the many possible problems), and it seems like I can think of a lot of goals like that I have that I can satisfy that I just don't care to get better at. Driving to work, say. Improving the ability is just a separate thing, and may or may not be there. It is not a necessecity, which was the claim. But I could see a couple of issues. There was the hedge sufficiently advanced. So if there is no goal to improve, it's just not sufficiently advance, and basically the claim is empty. But the other thing I thought was deeper. You need to remember that Steve and his posse are transhumanists. They have this sort of gut emotional belief that self improvement is all good. So in his and their decision processes, it just seems obvious that you must want to improve. I say the posse, because it's a socially reinforcing thing where they all egg on each's others feelings about how good this improvement stuff is going to be. So it might only be an assumption, but it's really, really clear and obvious to them. andi Quoting Richard Loosemore [EMAIL PROTECTED]: Mark Waser wrote: So if Omuhundro's claim rests on that fact that being self improving is part of the AGI's makeup, and that this will cause the AGI to do certain things, develop certain subgoals etc. I say that he has quietly inserted a *motivation* (or rather assumed it: does he ever say how this is supposed to work?) into the system, and then imagined some consequences. I think that I'm missing something here . . . . Omohundro is *explicitly* assuming self-improving and yes, self-improving is a goal/motivation. What do you believe that this proves/disproves?I'm not getting your point. Oh, simply that he cannot make deductions about what the self improvement motivation will actually do, until he has been explicit about exactly how it is implemented. In particular, the actual effects of a self-improvment motivation are different in a goal stack system versus a motivational-emotional system, and are also different depending on the strength and type of the self-improvement motivation. If you look at his paper carefully, you will see that at every step of the way he introduces assumptions as if they were obvious facts ... and in all the cases I have bothered to think through, these all stem from the fact that he has a particular kind of mechanism in mind (one which has a goal stack and a utility function). There are so many of these assertions pulled out of think air that I found it gave me a headache just to read the paper. Self-improvement is not a self-evident concept; it is not something that has a simple, unanalysable, a-priori clarity to it. We cannot just say that the AGI does self-improvment without saying how it goes about doing this. Omohundro, for all that he appears to be thinking about the topic deeply, is actually doing a sleight-of-hand job here he assumes a certain style of AGI design, then he pulls out a number of assertions about various aspects of self-improvement without stopping to clearly justify where these come from. I guess I need to pick an example to make this clear. He says that AGIs will generally want to improve the way that they achieve their goals, and this is correct so long as we understand it to be a general tendency. But then he points out that self-modifying their goal systems can have disastrous effects (again, true in principle), and he speculates about how we should try to minimize the risks of self-modification: If we wanted to prevent a system from improving itself, couldnt we just lock up its hardware and not tell it how to access its own machine code? For an intelligent system, impediments like these just become problems to solve in the process of meeting its goals. If the payoff is great enough, a system will go to great lengths to accomplish an outcome. If the runtime environment of the system does not allow it to modify its own machine code, it will be motivated to break the protection mechanisms of that runtime. In order to understand how much this paragraph is filled with unexamined
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Rationality and irrationality are interesting subjects . . . . Many people who endlessly tout rationally use it as an exact synonym for logical correctness and then argue not only that irrational then means logically incorrect and therefore wrong but that anything that can't be proved is irrational. Personally, I believe that we need to consciously have the same three forms of rational that we have for moral: rational - logically provable and correct irrational - logically provable as incorrect arational - not logically provable Irrational is a bad idea -- but there is a huge swath of stuff that is currently being defined as irrational that is actually arational. Logic is not as universally applicable as it's adherents would have you believe. Where it *can* be used, it is king -- but far, far too many people use logic to prove things where a single additional factor can easily render all of their precious arguments visibly incorrect. These are not logical arguments and this is exactly what Omohundro is doing. As I said in a previous e-mail, extrapolation is not logic (except in the very rare case of inductive proofs -- which *require* closed systems). - Original Message - From: Richard Loosemore [EMAIL PROTECTED] To: agi@v2.listbox.com Sent: Saturday, May 24, 2008 10:18 PM Subject: Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...] [EMAIL PROTECTED] wrote: I was sitting in the room when they were talking about it and I didn't feel like speaking up at the time (why break my streak?) but I felt he was just wrong. It seemed like you could boil the claim down to this: If you are sufficiently advanced, and you have a goal and some ability to meet goal, you will also want to improve your ability to achieve that goal. But there could be many reawsons why you would not want to improve (like maybe not wanting to devote the resources or risk any of the many possible problems), and it seems like I can think of a lot of goals like that I have that I can satisfy that I just don't care to get better at. Driving to work, say. Improving the ability is just a separate thing, and may or may not be there. It is not a necessecity, which was the claim. But I could see a couple of issues. There was the hedge sufficiently advanced. So if there is no goal to improve, it's just not sufficiently advance, and basically the claim is empty. But the other thing I thought was deeper. You need to remember that Steve and his posse are transhumanists. They have this sort of gut emotional belief that self improvement is all good. So in his and their decision processes, it just seems obvious that you must want to improve. I say the posse, because it's a socially reinforcing thing where they all egg on each's others feelings about how good this improvement stuff is going to be. So it might only be an assumption, but it's really, really clear and obvious to them. andi Indeed: it is becoming increasingly clear that there is a group obsessed with the idea of rationality (as they define it), and their God is the elimination of all irrational behavior (as they define it). I did not know that Omohundro was part of that group, but I note the signs: an obsession with trying to catalog the 'irrationality' of the average human, for example. It is a troubling sign. Some of us are trying to build some AGI systems that will do some good, but there is this dominant group in our field that are on some kind of fanatical mission to prove that ultra-rational AGI systems, with the least possible resemblance to human psychology, are the one true path to Nirvana. Fanatics are dangerous. Fanatics who are on your own side are *also* dangerous, sadly. Richard Loosemore Quoting Richard Loosemore [EMAIL PROTECTED]: Mark Waser wrote: So if Omuhundro's claim rests on that fact that being self improving is part of the AGI's makeup, and that this will cause the AGI to do certain things, develop certain subgoals etc. I say that he has quietly inserted a *motivation* (or rather assumed it: does he ever say how this is supposed to work?) into the system, and then imagined some consequences. I think that I'm missing something here . . . . Omohundro is *explicitly* assuming self-improving and yes, self-improving is a goal/motivation. What do you believe that this proves/disproves? I'm not getting your point. Oh, simply that he cannot make deductions about what the self improvement motivation will actually do, until he has been explicit about exactly how it is implemented. In particular, the actual effects of a self-improvment motivation are different in a goal stack system versus a motivational-emotional system, and are also different depending on the strength and type of the self-improvement motivation. If you look at his paper carefully, you will see that at every step of the way he introduces assumptions
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
- Original Message From: J Storrs Hall, PhD [EMAIL PROTECTED] The paper can be found at http://selfawaresystems.files.wordpress.com/2008/01/nature_of_self_improving_ai.pdf Read the appendix, p37ff. He's not making arguments -- he's explaining, with a few pointers into the literature, some parts of completely standard and accepted economics and game theory. It's all very basic stuff. I think Omohundro is making arguments, or providing reasoning, to support his views that the application of rational economic theory and game theory would tend to make an advanced agi system capable of self-improvement. I don't think anyone would say that is an accepted viewpoint! (I may not know what you are talking about; that has actually happened on a few occasions believe it or not. And this may be a different paper than the one that was previously being discussed.) I am not in complete disagreement with Loosemore because I do not believe that Omohundro's view is well founded. But my main disagreement with Loosemore is that I object to his exaggerated claims like the one he made when he said that Omohundro is just pulling conclusions out of thin air. That argument can be made against any and all of us until someone actually produces a truly advanced AI program. I think Omohundro is pulling some assumptions out of thin air, but that is acceptable in a conjectural discussion. So far, I have found Omohundro's paper to be one of the more enjoyable papers I have read recently. But that does not mean that I agree with what he says. I think that Omohundro should use a slightly higher level of criticism of his own ideas, but on the other hand, there is also a need to occasionally express some opinions that might not meet the higher level of criticism. The more general a comment is, the more it tends to be an opinion. So the views I expressed here are really opinions that I have not supported. I would have to work much harder to discuss one of Omohundro's ideas in any detail. But if I wanted to attack (or support) something that he wrote, I would have to do at least a little extra work so that I could make sure that I do understand him. If that was what I wanted to do, I would draw a few quotes from his paper to argue for or against the apparent intent and perspective that I felt he was expressing. But maybe I found a different paper than was being discussed. I noticed that the abstract he wrote for his paper was not written too well (in my opinion). Jim Bromer --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
When I first read Omohundro's paper, my first reaction was . . . Wow! That's awesome. Then, when I tried to build on it, I found myself picking it apart instead. My previous e-mails from today should explain why. He's trying to extrapolate and predict from first principles and toy experiments to a very large and complex system -- when there are just too many additional variables and too much emergent behavior to do so successfully. He made a great try and it's worth spending a lot of time with the paper. My biggest fault with it is that he should have recognized that his statements about all goal-driven systems don't apply to the proto--typical example (humans) and he should have made at least some explanation as to why he believed that it didn't. In a way, Omohundro's paper is the prototypical/archetypal example for Richard's arguments about many AGIers trying to design complex systems through decomposition and toy examples and expecting the results to self-assemble and scale up to full intelligence. I disagree entirely with Richard's arguments that Omohundro's errors have *anything* to do with architecture. I am even tempted to argue that Richard is so enamored with/ensnared in his MES vision that he may well be violating his own concerns about building complex systems. - Original Message - From: Jim Bromer To: agi@v2.listbox.com Sent: Sunday, May 25, 2008 2:22 PM Subject: Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...] - Original Message From: J Storrs Hall, PhD [EMAIL PROTECTED] The paper can be found at http://selfawaresystems.files.wordpress.com/2008/01/nature_of_self_improving_ai.pdf Read the appendix, p37ff. He's not making arguments -- he's explaining, with a few pointers into the literature, some parts of completely standard and accepted economics and game theory. It's all very basic stuff. I think Omohundro is making arguments, or providing reasoning, to support his views that the application of rational economic theory and game theory would tend to make an advanced agi system capable of self-improvement. I don't think anyone would say that is an accepted viewpoint! (I may not know what you are talking about; that has actually happened on a few occasions believe it or not. And this may be a different paper than the one that was previously being discussed.) I am not in complete disagreement with Loosemore because I do not believe that Omohundro's view is well founded. But my main disagreement with Loosemore is that I object to his exaggerated claims like the one he made when he said that Omohundro is just pulling conclusions out of thin air. That argument can be made against any and all of us until someone actually produces a truly advanced AI program. I think Omohundro is pulling some assumptions out of thin air, but that is acceptable in a conjectural discussion. So far, I have found Omohundro's paper to be one of the more enjoyable papers I have read recently. But that does not mean that I agree with what he says. I think that Omohundro should use a slightly higher level of criticism of his own ideas, but on the other hand, there is also a need to occasionally express some opinions that might not meet the higher level of criticism. The more general a comment is, the more it tends to be an opinion. So the views I expressed here are really opinions that I have not supported. I would have to work much harder to discuss one of Omohundro's ideas in any detail. But if I wanted to attack (or support) something that he wrote, I would have to do at least a little extra work so that I could make sure that I do understand him. If that was what I wanted to do, I would draw a few quotes from his paper to argue for or against the apparent intent and perspective that I felt he was expressing. But maybe I found a different paper than was being discussed. I noticed that the abstract he wrote for his paper was not written too well (in my opinion). Jim Bromer -- agi | Archives | Modify Your Subscription --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Jim Bromer wrote: - Original Message From: Richard Loosemore [EMAIL PROTECTED] Richard Loosemore said: If you look at his paper carefully, you will see that at every step of the way he introduces assumptions as if they were obvious facts ... and in all the cases I have bothered to think through, these all stem from the fact that he has a particular kind of mechanism in mind (one which has a goal stack and a utility function). There are so many of these assertions pulled out of think air that I found it gave me a headache just to read the paper. ... But this is silly: where was his examination of the systems various motives? Where did he consider the difference between different implementations of the entire motivational mechanism (my distinction between GS and MES systems)? Nowhere. He just asserts, without argument, that the system would be obsessed, and that any attempt by us to put locks on the system would result in an arms race of measures and countermeasures. That is just one example of how he pulls conclusions out of thin air. --- Your argument about the difference between a GS and an MES system is a strawman argument. Omohundro never made the argument, nor did he touch on it as far as I can tell. I did not find his paper very interesting either, but you are the one who seems to be pulling conclusions out of thin air. You can introduce the GS vs MES argument if you want, but you cannot then argue from the implication that everyone has to refer to it or else stand guilty of pulling arguments out of thin air. His paper Nature of Self Improving Artificial Intelligence September 5, 2007, revised January 21, 2008 provides a lot of reasoning. I don't find the reasoning compelling, but the idea that he is just pulling conclusions out of thin air is just bluster. Taking things in reverse order: that last paper you refer to is not one that I have, and I hope that you do not think I was referring to that in my criticism. Why do make reference to it, and imply that my comments apply to that paper and then calling my non-existent comments on that paper bluster? I am confused: the paper I was critiquing was clearly the ai drives paper from the 2008 conference. When I accused Omohundro of pulling conclusions out of thin air, I went through a careful process: I quoted a passage from his paper, then I analysed it by describing cases of two hypothetical AGI systems, then I showed how those cases falsified his conclusion. I went to a great deal of trouble to back up my claim. Now you come along and make several claims (that my argument was a strawman argument, that I am pulling conclusions out of thin air, that I am guilty of bluster, etc.), but instead of explaining what the justification is for these claims, you offer nothing. And you offer no reply to the argument that I gave before: you ignore this as if it did not exist. I gave details. Address them! HOWEVER, since you have raised these questions, let me try to address your concerns. The argument I gave was not a strawman, because when I said that Omuhundro's arguments assume a Goal Stack architecture this was just a shorthand for the longer, but equivalent claim: Omohundro made many statements about how an AI 'would' behave, but in each of these cases it is very easy to imagine a type of AI that would not do that at all, so his claims are only about a very particular type of AI, not the general case. Now, as it happens, the best way to describe the 'very particular type of AI' that Omohundro had in the back of his mind when he made those statements, would be to say that he assumed a 'Goal Stack' type of AI, and he seemed to have no idea that other types of motivation mechanism would be just as feasible. In fact he did not *need* to know about the distinction between Goal-Stack mechanisms and Motivational-Emotional Systems (and I did NOT criticize him because he failed to refer to the GS-MES distinction) he could have realized that all of his 'An AI would do this... statements were not really valid if he had simply taken the trouble to think through the implications properly. I believe that the reason he did such a sloppy job of thinking through the implications of his statements was because he had a narrow, GS-type of mechanism in mind - but in a sense that is neither here nor there. It is the SLOPPY REASONING that I was targetting, not the fact that he considered only a GS architecture. I believe that if he had been aware of the distinction between GS and MES architecture, he would have been much less likely to have engaged in such sloppy reasoning, and that is what I said to Kaj Sotala when this thread started I simply said that Omohundro's examples of what an AI 'would' do did not work if you considered the case of general MES systems, rather than just GS systems. Richard Loosemore
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
J Storrs Hall, PhD wrote: The paper can be found at http://selfawaresystems.files.wordpress.com/2008/01/nature_of_self_improving_ai.pdf Read the appendix, p37ff. He's not making arguments -- he's explaining, with a few pointers into the literature, some parts of completely standard and accepted economics and game theory. It's all very basic stuff. This is NOT the paper that is under discussion. Look back to the first post in this thread. I will address that other paper on some other occasion. Richard Loosemore On Sunday 25 May 2008 06:26:59 am, Jim Bromer wrote: - Original Message From: Richard Loosemore [EMAIL PROTECTED] Richard Loosemore said: If you look at his paper carefully, you will see that at every step of the way he introduces assumptions as if they were obvious facts ... and in all the cases I have bothered to think through, these all stem from the fact that he has a particular kind of mechanism in mind (one which has a goal stack and a utility function). There are so many of these assertions pulled out of think air that I found it gave me a headache just to read the paper. ... But this is silly: where was his examination of the systems various motives? Where did he consider the difference between different implementations of the entire motivational mechanism (my distinction between GS and MES systems)? Nowhere. He just asserts, without argument, that the system would be obsessed, and that any attempt by us to put locks on the system would result in an arms race of measures and countermeasures. That is just one example of how he pulls conclusions out of thin air. --- Your argument about the difference between a GS and an MES system is a strawman argument. Omohundro never made the argument, nor did he touch on it as far as I can tell. I did not find his paper very interesting either, but you are the one who seems to be pulling conclusions out of thin air. You can introduce the GS vs MES argument if you want, but you cannot then argue from the implication that everyone has to refer to it or else stand guilty of pulling arguments out of thin air. His paper Nature of Self Improving Artificial Intelligence September 5, 2007, revised January 21, 2008 provides a lot of reasoning. I don't find the reasoning compelling, but the idea that he is just pulling conclusions out of thin air is just bluster. Jim Bromer --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?; Powered by Listbox: http://www.listbox.com --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?; Powered by Listbox: http://www.listbox.com --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
On Sunday 25 May 2008 10:06:11 am, Mark Waser wrote: Read the appendix, p37ff. He's not making arguments -- he's explaining, with a few pointers into the literature, some parts of completely standard and accepted economics and game theory. It's all very basic stuff. The problem with accepted economics and game theory is that in a proper scientific sense, they actually prove very little and certainly far, FAR less than people extrapolate them to mean (or worse yet, prove). Abusus non tollit usum. All of the scientific experiments in game theory are very, VERY limited and deal with entities with little memory in small, toy systems. If you extrapolate their results with no additional input and no emergent effects, you can end up with arguments like Omohundro's BUT claiming that this extrapolation *proves* anything is very poor science. It's just speculation/science fiction and there are any number of reasons to believe that Omohundro's theories are incorrect -- the largest one, of course, being If all goal-based systems end up evil, why isn't every *naturally* intelligent entity evil? Actually, modern (post-Axelrod) evolutionary game theory handles this pretty well, and shows the existence of what I call the moral ladder. BTW, I've had extended discussions with Steve O. about it, and consider his ultimate position to be over-pessimistic -- but his basic starting point (and the econ theory he references) is sound. Josh --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
On Sunday 25 May 2008 07:51:59 pm, Richard Loosemore wrote: This is NOT the paper that is under discussion. WRONG. This is the paper I'm discussing, and is therefore the paper under discussion. --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
In the context of Steve's paper, however, rational simply means an agent who does not have a preference circularity. On Sunday 25 May 2008 10:19:35 am, Mark Waser wrote: Rationality and irrationality are interesting subjects . . . . Many people who endlessly tout rationally use it as an exact synonym for logical correctness and then argue not only that irrational then means logically incorrect and therefore wrong but that anything that can't be proved is irrational. --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
J Storrs Hall, PhD wrote: On Sunday 25 May 2008 07:51:59 pm, Richard Loosemore wrote: This is NOT the paper that is under discussion. WRONG. This is the paper I'm discussing, and is therefore the paper under discussion. Josh, are you sure you're old enough to be using a computer without parental supervision? --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
So if Omuhundro's claim rests on that fact that being self improving is part of the AGI's makeup, and that this will cause the AGI to do certain things, develop certain subgoals etc. I say that he has quietly inserted a *motivation* (or rather assumed it: does he ever say how this is supposed to work?) into the system, and then imagined some consequences. I think that I'm missing something here . . . . Omohundro is *explicitly* assuming self-improving and yes, self-improving is a goal/motivation. What do you believe that this proves/disproves? I'm not getting your point. Further, I do not buy the supposed consequences. Me, I have the self-improving motivation too. But it is pretty modest, and also it is just one among many, so it does not have the consequences that he attributes to the general existence of the self-improvement motivation. AS I said in my previous e-mail, I don't buy his consequences either. My point is that since he did not understand that he was making the assumption, Excuse me? What makes you believe that he didn't understand that he was making the self-improvement assumption or that it was a goal/motivation? It looked pretty deliberate to me. and did not realize the role that it could play in a Motivational Emotional system (as opposed to a Goal Stack system), OK. So could you describe what role it would play in an MES system as opposed to a Goal Stack System? I don't see a difference in terms of effects. he made a complete dog's dinner of claiming how a future AGI would *necessarily* behave. This I agree with -- but not because of any sort of differences between GS and MES systems. I don't believe that his conclusions apply to an intelligent GS system either. Only in a Goal Stack system is there a danger of a self-improvement supergoal going awol. Why? An MES system requires more failures to have a problem, but certain types of environment could (and should) cause such a problem. As far as i can see, his arguments simply do not apply to MES systems: the arguments depend too heavily on the assumption that the architecture is a Goal Stack. It is simply that none of what he says *follows* if an MES is used. Just a lot of non-sequiteurs. I *STILL* don't get this. His arguments depend heavily upon the system having goals/motivations. Yes, his arguments do not apply to an MES system without motivations. But they do apply to MES systems with motivations (although, again, I don't agree with his conclusions). When an MES system is set up with motivations (instead of being blank) what happens next depends on the mechanics of the system, and the particular motivations. YES! But his argument is that to fulfill *any* motivation, there are generic submotivations (protect myself, accumulate power, don't let my motivation get perverted) that will further the search to fulfill your motivation. = = = = = As a relevant aside, you never answered my question regarding how you believed an MES system was different from a system with a *large* number of goal stacks. - Original Message - From: Richard Loosemore [EMAIL PROTECTED] To: agi@v2.listbox.com Sent: Friday, May 23, 2008 9:22 PM Subject: Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...] Mark Waser wrote: he makes a direct reference to goal driven systems, but even more important he declares that these bad behaviors will *not* be the result of us programming the behaviors in at the start but in an MES system nothing at all will happen unless the designer makes an explicit decision to put some motivations into the system, so I can be pretty sure that he has not considered that type of motivational system when he makes these comments. Richard, I think that you are incorrect here. When Omohundro says that the bad behaviors will *not* be the result of us programming the behaviors in at the start, what he means is that the very fact of having goals or motivations and being self-improving will naturally lead (**regardless of architecture**) to certain (what I call generic) sub-goals (like the acquisition of power/money, self-preservation, etc.) and that the fulfillment of those subgoals, without other considerations (like ethics or common-sense), will result in what we would consider bad behavior. This I do not buy, for the following reason. What is this thing called being self improving? Complex concept, that. How are we going to get an AGI to do that? This is a motivation, pure and simple. So if Omuhundro's claim rests on that fact that being self improving is part of the AGI's makeup, and that this will cause the AGI to do certain things, develop certain subgoals etc. I say that he has quietly inserted a *motivation* (or rather assumed it: does he ever say how this is supposed to work?) into the system, and then imagined some consequences. Further, I do not buy the supposed consequences. Me, I
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
On Saturday 24 May 2008 06:55:24 pm, Mark Waser wrote: ...Omuhundro's claim... YES! But his argument is that to fulfill *any* motivation, there are generic submotivations (protect myself, accumulate power, don't let my motivation get perverted) that will further the search to fulfill your motivation. It's perhaps a little more subtle than that. (BTW, note I made the same arguments re submotivations in Beyond AI p. 339) Steve points out that any motivational architecture that cannot be reduced to a utility function over world states is incoherent in the sense that the AI could be taken advantage of in purely uncoerced transactions by any other agent that understood its motivational structure. Thus one can assume that non-utility-function-equivalent AIs (not to mention humans) will rapidly lose resources in a future world and thus it won't particularly matter what they want. If you look at the suckerdom of average humans in todays sub-prime mortgage, easy credit, etc., markets, there's ample evidence that it won't take evil AI to make this economic cleansing environment happen. And the powers that be don't seem to be any too interested in shielding people from it... So Steve's point is that utility-function-equivalent AIs will predominate simply by lack of that basic vulnerability (and the fact that it is a vulnerability is a mathematically provable theorem) which is a part of ANY other motivational structure. The rest (self-interest, etc) follows, Q.E.D. Josh --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Mark Waser wrote: So if Omuhundro's claim rests on that fact that being self improving is part of the AGI's makeup, and that this will cause the AGI to do certain things, develop certain subgoals etc. I say that he has quietly inserted a *motivation* (or rather assumed it: does he ever say how this is supposed to work?) into the system, and then imagined some consequences. I think that I'm missing something here . . . . Omohundro is *explicitly* assuming self-improving and yes, self-improving is a goal/motivation. What do you believe that this proves/disproves? I'm not getting your point. Oh, simply that he cannot make deductions about what the self improvement motivation will actually do, until he has been explicit about exactly how it is implemented. In particular, the actual effects of a self-improvment motivation are different in a goal stack system versus a motivational-emotional system, and are also different depending on the strength and type of the self-improvement motivation. If you look at his paper carefully, you will see that at every step of the way he introduces assumptions as if they were obvious facts ... and in all the cases I have bothered to think through, these all stem from the fact that he has a particular kind of mechanism in mind (one which has a goal stack and a utility function). There are so many of these assertions pulled out of think air that I found it gave me a headache just to read the paper. Self-improvement is not a self-evident concept; it is not something that has a simple, unanalysable, a-priori clarity to it. We cannot just say that the AGI does self-improvment without saying how it goes about doing this. Omohundro, for all that he appears to be thinking about the topic deeply, is actually doing a sleight-of-hand job here he assumes a certain style of AGI design, then he pulls out a number of assertions about various aspects of self-improvement without stopping to clearly justify where these come from. I guess I need to pick an example to make this clear. He says that AGIs will generally want to improve the way that they achieve their goals, and this is correct so long as we understand it to be a general tendency. But then he points out that self-modifying their goal systems can have disastrous effects (again, true in principle), and he speculates about how we should try to minimize the risks of self-modification: If we wanted to prevent a system from improving itself, couldnt we just lock up its hardware and not tell it how to access its own machine code? For an intelligent system, impediments like these just become problems to solve in the process of meeting its goals. If the payoff is great enough, a system will go to great lengths to accomplish an outcome. If the runtime environment of the system does not allow it to modify its own machine code, it will be motivated to break the protection mechanisms of that runtime. In order to understand how much this paragraph is filled with unexamined assumptions, consider two possible AGI systems, A and B. System A has a motivational structure that includes some desire to improve itself, along with some empathy for the human species and some strong motivations not to do anything dangerous. It balances these three factors in such a way that it fully understands the dangers of self-modification of its motivational system, and while it would, in general, like to do some self-improvement, it also understands that the locks that the humans have inserted are there for its own, and the humans' protection, and it so the urge to try to crack those locks is virtually non-existent. System B is motivated to improve its goal system, and improving its motivational system is part of that quest, so it regards the locks that the humans have put on it as just an obstruction. Further, it is strongly motivated to try to solve difficult challenges more than simple challenges, so the locks represent a particularly appealing target. Now, System B will do all of the things that Omohundro suggests in the above passage, but System A will not do any of them: it would be ridiculous to say that for System A impediments like these just become problems to solve in the process of meeting its goals. System A is just not that monomaniacally obsessed with self-improvement! System A is mature, thoughtful and balanced in its assessment of the situation. It is cautious, and able to appreciate that there is a tradeoff here. If you read the rest of the paragraph from which that extract came, you will see that Omohundro would have us believe that the system goes on to try to convince or trick humans to make the changes! As far as he is concerned, there is no doubt whatsoever that an AGI would *have* to utterly obsessed with improving itself at all costs. But this is silly: where was his examination of the systems various motives? Where did he consider the difference between different implementations of the entire
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
I was sitting in the room when they were talking about it and I didn't feel like speaking up at the time (why break my streak?) but I felt he was just wrong. It seemed like you could boil the claim down to this: If you are sufficiently advanced, and you have a goal and some ability to meet goal, you will also want to improve your ability to achieve that goal. But there could be many reawsons why you would not want to improve (like maybe not wanting to devote the resources or risk any of the many possible problems), and it seems like I can think of a lot of goals like that I have that I can satisfy that I just don't care to get better at. Driving to work, say. Improving the ability is just a separate thing, and may or may not be there. It is not a necessecity, which was the claim. But I could see a couple of issues. There was the hedge sufficiently advanced. So if there is no goal to improve, it's just not sufficiently advance, and basically the claim is empty. But the other thing I thought was deeper. You need to remember that Steve and his posse are transhumanists. They have this sort of gut emotional belief that self improvement is all good. So in his and their decision processes, it just seems obvious that you must want to improve. I say the posse, because it's a socially reinforcing thing where they all egg on each's others feelings about how good this improvement stuff is going to be. So it might only be an assumption, but it's really, really clear and obvious to them. andi Quoting Richard Loosemore [EMAIL PROTECTED]: Mark Waser wrote: So if Omuhundro's claim rests on that fact that being self improving is part of the AGI's makeup, and that this will cause the AGI to do certain things, develop certain subgoals etc. I say that he has quietly inserted a *motivation* (or rather assumed it: does he ever say how this is supposed to work?) into the system, and then imagined some consequences. I think that I'm missing something here . . . . Omohundro is *explicitly* assuming self-improving and yes, self-improving is a goal/motivation. What do you believe that this proves/disproves? I'm not getting your point. Oh, simply that he cannot make deductions about what the self improvement motivation will actually do, until he has been explicit about exactly how it is implemented. In particular, the actual effects of a self-improvment motivation are different in a goal stack system versus a motivational-emotional system, and are also different depending on the strength and type of the self-improvement motivation. If you look at his paper carefully, you will see that at every step of the way he introduces assumptions as if they were obvious facts ... and in all the cases I have bothered to think through, these all stem from the fact that he has a particular kind of mechanism in mind (one which has a goal stack and a utility function). There are so many of these assertions pulled out of think air that I found it gave me a headache just to read the paper. Self-improvement is not a self-evident concept; it is not something that has a simple, unanalysable, a-priori clarity to it. We cannot just say that the AGI does self-improvment without saying how it goes about doing this. Omohundro, for all that he appears to be thinking about the topic deeply, is actually doing a sleight-of-hand job here he assumes a certain style of AGI design, then he pulls out a number of assertions about various aspects of self-improvement without stopping to clearly justify where these come from. I guess I need to pick an example to make this clear. He says that AGIs will generally want to improve the way that they achieve their goals, and this is correct so long as we understand it to be a general tendency. But then he points out that self-modifying their goal systems can have disastrous effects (again, true in principle), and he speculates about how we should try to minimize the risks of self-modification: If we wanted to prevent a system from improving itself, couldnt we just lock up its hardware and not tell it how to access its own machine code? For an intelligent system, impediments like these just become problems to solve in the process of meeting its goals. If the payoff is great enough, a system will go to great lengths to accomplish an outcome. If the runtime environment of the system does not allow it to modify its own machine code, it will be motivated to break the protection mechanisms of that runtime. In order to understand how much this paragraph is filled with unexamined assumptions, consider two possible AGI systems, A and B. System A has a motivational structure that includes some desire to improve itself, along with some empathy for the human species and some strong motivations not to do anything dangerous. It balances these three factors in such a way that it fully understands the dangers of
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
[EMAIL PROTECTED] wrote: I was sitting in the room when they were talking about it and I didn't feel like speaking up at the time (why break my streak?) but I felt he was just wrong. It seemed like you could boil the claim down to this: If you are sufficiently advanced, and you have a goal and some ability to meet goal, you will also want to improve your ability to achieve that goal. But there could be many reawsons why you would not want to improve (like maybe not wanting to devote the resources or risk any of the many possible problems), and it seems like I can think of a lot of goals like that I have that I can satisfy that I just don't care to get better at. Driving to work, say. Improving the ability is just a separate thing, and may or may not be there. It is not a necessecity, which was the claim. But I could see a couple of issues. There was the hedge sufficiently advanced. So if there is no goal to improve, it's just not sufficiently advance, and basically the claim is empty. But the other thing I thought was deeper. You need to remember that Steve and his posse are transhumanists. They have this sort of gut emotional belief that self improvement is all good. So in his and their decision processes, it just seems obvious that you must want to improve. I say the posse, because it's a socially reinforcing thing where they all egg on each's others feelings about how good this improvement stuff is going to be. So it might only be an assumption, but it's really, really clear and obvious to them. andi Indeed: it is becoming increasingly clear that there is a group obsessed with the idea of rationality (as they define it), and their God is the elimination of all irrational behavior (as they define it). I did not know that Omohundro was part of that group, but I note the signs: an obsession with trying to catalog the 'irrationality' of the average human, for example. It is a troubling sign. Some of us are trying to build some AGI systems that will do some good, but there is this dominant group in our field that are on some kind of fanatical mission to prove that ultra-rational AGI systems, with the least possible resemblance to human psychology, are the one true path to Nirvana. Fanatics are dangerous. Fanatics who are on your own side are *also* dangerous, sadly. Richard Loosemore Quoting Richard Loosemore [EMAIL PROTECTED]: Mark Waser wrote: So if Omuhundro's claim rests on that fact that being self improving is part of the AGI's makeup, and that this will cause the AGI to do certain things, develop certain subgoals etc. I say that he has quietly inserted a *motivation* (or rather assumed it: does he ever say how this is supposed to work?) into the system, and then imagined some consequences. I think that I'm missing something here . . . . Omohundro is *explicitly* assuming self-improving and yes, self-improving is a goal/motivation. What do you believe that this proves/disproves? I'm not getting your point. Oh, simply that he cannot make deductions about what the self improvement motivation will actually do, until he has been explicit about exactly how it is implemented. In particular, the actual effects of a self-improvment motivation are different in a goal stack system versus a motivational-emotional system, and are also different depending on the strength and type of the self-improvement motivation. If you look at his paper carefully, you will see that at every step of the way he introduces assumptions as if they were obvious facts ... and in all the cases I have bothered to think through, these all stem from the fact that he has a particular kind of mechanism in mind (one which has a goal stack and a utility function). There are so many of these assertions pulled out of think air that I found it gave me a headache just to read the paper. Self-improvement is not a self-evident concept; it is not something that has a simple, unanalysable, a-priori clarity to it. We cannot just say that the AGI does self-improvment without saying how it goes about doing this. Omohundro, for all that he appears to be thinking about the topic deeply, is actually doing a sleight-of-hand job here he assumes a certain style of AGI design, then he pulls out a number of assertions about various aspects of self-improvement without stopping to clearly justify where these come from. I guess I need to pick an example to make this clear. He says that AGIs will generally want to improve the way that they achieve their goals, and this is correct so long as we understand it to be a general tendency. But then he points out that self-modifying their goal systems can have disastrous effects (again, true in principle), and he speculates about how we should try to minimize the risks of self-modification: If we wanted to prevent a system from improving itself, couldnÂ’t we just lock up its hardware and not tell it how to access
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
J Storrs Hall, PhD wrote: On Saturday 24 May 2008 06:55:24 pm, Mark Waser wrote: ...Omuhundro's claim... YES! But his argument is that to fulfill *any* motivation, there are generic submotivations (protect myself, accumulate power, don't let my motivation get perverted) that will further the search to fulfill your motivation. It's perhaps a little more subtle than that. (BTW, note I made the same arguments re submotivations in Beyond AI p. 339) Steve points out that any motivational architecture that cannot be reduced to a utility function over world states is incoherent in the sense that the AI could be taken advantage of in purely uncoerced transactions by any other agent that understood its motivational structure. Thus one can assume that non-utility-function-equivalent AIs (not to mention humans) will rapidly lose resources in a future world and thus it won't particularly matter what they want. If you look at the suckerdom of average humans in todays sub-prime mortgage, easy credit, etc., markets, there's ample evidence that it won't take evil AI to make this economic cleansing environment happen. And the powers that be don't seem to be any too interested in shielding people from it... So Steve's point is that utility-function-equivalent AIs will predominate simply by lack of that basic vulnerability (and the fact that it is a vulnerability is a mathematically provable theorem) which is a part of ANY other motivational structure. The rest (self-interest, etc) follows, Q.E.D. In a post I just sent in reply to Mark, I point out that, far from giving us any coherent argument about the performance of motivational systems, Omohundro simply weaves a long trail of assumptions into something that looks like an argument. In particular, i analyzed one of the early claims he made in the paper, and I think I have demonstrated quite clearly that what he states (without justification) as the behavior of an AI system is just the behavior of an arbitrarily chosen, amazingly obsessive AI. This error is so egregious, and is repeated so many times in the paper that, in the end, all his 'arguments' are just statements about the behavior of one particular design of motivation mechanism, without any examination of why he makes the assumptions that he does. Every one of his statements about what an AGI would do can be attacked in the same way that I attacked that one on page 2 of the paper, with the same devastating results each time. So when you say that Steve points out that any motivational architecture that cannot be reduced to a utility function over world states is incoherent ..., I can only say that there is no place in the copy of his paper that I have here (ai_drives_final.pdf - there is no date on it), in which he produces a compelling argument (i.e. an argument not founded on handwaving and iimplicit assumptions) about the incoherence of systems that cannot be reduced to a utility function over world states. Richard Loosemore --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Kaj Sotala wrote: Richard, again, I must sincerely apologize for responding to this so horrendously late. It's a dreadful bad habit of mine: I get an e-mail (or blog comment, or forum message, or whatever) that requires some thought before I respond, so I don't answer it right away... and then something related to my studies or hobbies shows up and doesn't leave me with enough energy to compose responses to anybody at all, after which enough time has passed that the message has vanished from my active memory, and when I remember it so much time has passed already that a day or two more before I answer won't make any difference... and then *so* much time has passed that replying to the message so late feels more embarassing than just quietly forgetting about it. I'll try to better my ways in the future. On the same token, I must say I can only admire your ability to compose long, well-written replies to messages in what seem to be blinks of an eye to me. :-) Hey, no problem . you'll notice that I am pretty late getting back this time :-) . got too many things to keep up with here. In the spirit of our attempt to create the longest-indented discussion in the universe, I have left all the original text in and inserted my responses appropriately... On 3/11/08, Richard Loosemore [EMAIL PROTECTED] wrote: Kaj Sotala wrote: On 3/3/08, Richard Loosemore [EMAIL PROTECTED] wrote: Kaj Sotala wrote: Alright. But previously, you said that Omohundro's paper, which to me seemed to be a general analysis of the behavior of *any* minds with (more or less) explict goals, looked like it was based on a 'goal-stack' motivation system. (I believe this has also been the basis of your critique for e.g. some SIAI articles about friendliness.) If built-in goals *can* be constructed into motivational system AGIs, then why do you seem to assume that AGIs with built-in goals are goal-stack ones? I seem to have caused lots of confusion earlier on in the discussion, so let me backtrack and try to summarize the structure of my argument. 1) Conventional AI does not have a concept of a Motivational-Emotional System (MES), the way that I use that term, so when I criticised Omuhundro's paper for referring only to a Goal Stack control system, I was really saying no more than that he was assuming that the AI was driven by the system that all conventional AIs are supposed to have. These two ways of controlling an AI are two radically different designs. [...] So now: does that clarify the specific question you asked above? Yes and no. :-) My main question is with part 1 of your argument - you are saying that Omohundro's paper assumed the AI to have a certain sort of control system. This is the part which confuses me, since I didn't see the paper to make *any* mentions of how the AI should be built. It only assumes that the AI has some sort of goals, and nothing more. [...] Drive 1: AIs will want to self-improve This one seems fairly straightforward: indeed, for humans self-improvement seems to be an essential part in achieving pretty much *any* goal you are not immeaditly capable of achieving. If you don't know how to do something needed to achieve your goal, you practice, and when you practice, you're improving yourself. Likewise, improving yourself will quickly become a subgoal for *any* major goals. But now I ask: what exactly does this mean? In the context of a Goal Stack system, this would be represented by a top level goal that was stated in the knowledge representation language of the AGI, so it would say Improve Thyself. [...] The reason that I say Omuhundro is assuming a Goal Stack system is that I believe he would argue that that is what he meant, and that he assumed that a GS architecture would allow the AI to exhibit behavior that corresponds to what we, as humans, recognize as wanting to self-improve. I think it is a hidden assumption in what he wrote. At least I didn't read the paper in such a way - after all, the abstract says that it's supposed to apply equally to all AGI systems, regardless of the exact design: We identify a number of drives that will appear in sufficiently advanced AI systems of any design. We call them drives because they are tendencies which will be present unless explicitly counteracted. (You could, of course, suppose that the author was assuming that an AGI could *only* be built around a Goal Stack system, and therefore any design would mean any GS design... but that seems a bit far-fetched.) Oh, I don't think that would be far-fetched, because most AI people have not even begun to think about how to control an AI/AGI system, so they always just go for the default. And the default is a goal-stack system. I have not yet published my work on MES systems, so Omuhundro would probably not know of that. I did notice his claim that his 'drives' are completely general, and I found that amusing, because it does not cover the cases that I
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
he makes a direct reference to goal driven systems, but even more important he declares that these bad behaviors will *not* be the result of us programming the behaviors in at the start but in an MES system nothing at all will happen unless the designer makes an explicit decision to put some motivations into the system, so I can be pretty sure that he has not considered that type of motivational system when he makes these comments. Richard, I think that you are incorrect here. When Omohundro says that the bad behaviors will *not* be the result of us programming the behaviors in at the start, what he means is that the very fact of having goals or motivations and being self-improving will naturally lead (**regardless of architecture**) to certain (what I call generic) sub-goals (like the acquisition of power/money, self-preservation, etc.) and that the fulfillment of those subgoals, without other considerations (like ethics or common-sense), will result in what we would consider bad behavior. I believe that he is correct in that goals or motivations and self-improvement will lead to generic subgoals regardless of architecture. Do you believe that your MES will not derive generic subgoals under self-improvement? Omohundro's arguments aren't *meant* to apply to an MES system without motivations -- because such a system can't be considered to have goals. His arguments will start to apply as soon as the MES system does have motivations/goals. (Though, I hasten to add that I believe that his logical reasoning is flawed in that there are some drives that he missed that will prevent such bad behavior in any sufficiently advanced system). - Original Message - From: Richard Loosemore [EMAIL PROTECTED] To: agi@v2.listbox.com Sent: Friday, May 23, 2008 2:13 PM Subject: Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...] Kaj Sotala wrote: Richard, again, I must sincerely apologize for responding to this so horrendously late. It's a dreadful bad habit of mine: I get an e-mail (or blog comment, or forum message, or whatever) that requires some thought before I respond, so I don't answer it right away... and then something related to my studies or hobbies shows up and doesn't leave me with enough energy to compose responses to anybody at all, after which enough time has passed that the message has vanished from my active memory, and when I remember it so much time has passed already that a day or two more before I answer won't make any difference... and then *so* much time has passed that replying to the message so late feels more embarassing than just quietly forgetting about it. I'll try to better my ways in the future. On the same token, I must say I can only admire your ability to compose long, well-written replies to messages in what seem to be blinks of an eye to me. :-) Hey, no problem . you'll notice that I am pretty late getting back this time :-) . got too many things to keep up with here. In the spirit of our attempt to create the longest-indented discussion in the universe, I have left all the original text in and inserted my responses appropriately... On 3/11/08, Richard Loosemore [EMAIL PROTECTED] wrote: Kaj Sotala wrote: On 3/3/08, Richard Loosemore [EMAIL PROTECTED] wrote: Kaj Sotala wrote: Alright. But previously, you said that Omohundro's paper, which to me seemed to be a general analysis of the behavior of *any* minds with (more or less) explict goals, looked like it was based on a 'goal-stack' motivation system. (I believe this has also been the basis of your critique for e.g. some SIAI articles about friendliness.) If built-in goals *can* be constructed into motivational system AGIs, then why do you seem to assume that AGIs with built-in goals are goal-stack ones? I seem to have caused lots of confusion earlier on in the discussion, so let me backtrack and try to summarize the structure of my argument. 1) Conventional AI does not have a concept of a Motivational-Emotional System (MES), the way that I use that term, so when I criticised Omuhundro's paper for referring only to a Goal Stack control system, I was really saying no more than that he was assuming that the AI was driven by the system that all conventional AIs are supposed to have. These two ways of controlling an AI are two radically different designs. [...] So now: does that clarify the specific question you asked above? Yes and no. :-) My main question is with part 1 of your argument - you are saying that Omohundro's paper assumed the AI to have a certain sort of control system. This is the part which confuses me, since I didn't see the paper to make *any* mentions of how the AI should be built. It only assumes that the AI has some sort of goals, and nothing more. [...] Drive 1: AIs will want to self-improve This one seems fairly straightforward: indeed, for humans self-improvement seems to be an essential part
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Mark Waser wrote: he makes a direct reference to goal driven systems, but even more important he declares that these bad behaviors will *not* be the result of us programming the behaviors in at the start but in an MES system nothing at all will happen unless the designer makes an explicit decision to put some motivations into the system, so I can be pretty sure that he has not considered that type of motivational system when he makes these comments. Richard, I think that you are incorrect here. When Omohundro says that the bad behaviors will *not* be the result of us programming the behaviors in at the start, what he means is that the very fact of having goals or motivations and being self-improving will naturally lead (**regardless of architecture**) to certain (what I call generic) sub-goals (like the acquisition of power/money, self-preservation, etc.) and that the fulfillment of those subgoals, without other considerations (like ethics or common-sense), will result in what we would consider bad behavior. This I do not buy, for the following reason. What is this thing called being self improving? Complex concept, that. How are we going to get an AGI to do that? This is a motivation, pure and simple. So if Omuhundro's claim rests on that fact that being self improving is part of the AGI's makeup, and that this will cause the AGI to do certain things, develop certain subgoals etc. I say that he has quietly inserted a *motivation* (or rather assumed it: does he ever say how this is supposed to work?) into the system, and then imagined some consequences. Further, I do not buy the supposed consequences. Me, I have the self-improving motivation too. But it is pretty modest, and also it is just one among many, so it does not have the consequences that he attributes to the general existence of the self-improvement motivation. My point is that since he did not understand that he was making the assumption, and did not realize the role that it could play in a Motivational Emotional system (as opposed to a Goal Stack system), he made a complete dog's dinner of claiming how a future AGI would *necessarily* behave. Could an intelligent system be built without a rampaging desire for self-improvement (or, as Omuhundro would have it, rampaging power hunger)? Sure: a system could just modestly want to do interesting things and have new and pleasureful experiences. At the very least, I don't think that you could claim that such an unassuming, hedonistic and unambitious type of AGI is *obviously* impossible. I believe that he is correct in that goals or motivations and self-improvement will lead to generic subgoals regardless of architecture. Do you believe that your MES will not derive generic subgoals under self-improvement? See above: if self-improvement is just one motivation among many, then the answer depends on exactly how it is implemented. Only in a Goal Stack system is there a danger of a self-improvement supergoal going awol. Omohundro's arguments aren't *meant* to apply to an MES system without motivations -- because such a system can't be considered to have goals. His arguments will start to apply as soon as the MES system does have motivations/goals. (Though, I hasten to add that I believe that his logical reasoning is flawed in that there are some drives that he missed that will prevent such bad behavior in any sufficiently advanced system). As far as i can see, his arguments simply do not apply to MES systems: the arguments depend too heavily on the assumption that the architecture is a Goal Stack. It is simply that none of what he says *follows* if an MES is used. Just a lot of non-sequiteurs. When an MES system is set up with motivations (instead of being blank) what happens next depends on the mechanics of the system, and the particular motivations. Richard Loosemore --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=103754539-40ed26 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Vladamir, On 5/7/08, Vladimir Nesov [EMAIL PROTECTED] wrote: See http://www.overcomingbias.com/2008/01/newcombs-proble.html This is a PERFECT talking point for the central point that I have been trying to make. Belief in the Omega discussed early in that article is essentially a religious belief in a greater power. Most Christians see examples of the power of God at around a monthly rate. Whenever chance works for apparent good or against perceived evil, there is clear evidence of God doing his job. Story: A Baptist minister neighbor had his alternator come loose just as he was leaving for an important meeting, so I temporarily secured it with an industrial zip tie, and told him to remove the zip tie and properly bold the alternator back into place when he got back home. Three weeks later, his alternator came loose again. He explained that he had done NOTHING wrong this week, and so he just couldn't see why God took this occasion to smite his alternator. I suggested that we examine it for clues. Sure enough, there were the remnants of my zip tie which he had never replaced. He explained that God seemed to be holding things together OK, so why bother fixing it. Explaining the limitations of industrial zip ties seemed to be hopeless, so I translated my engineering paradigm to his religious paradigm: I explained that he had been testing God by seeing how long God would continue to hold his alternator in place, and apparently God had grown tired of playing this game. Oh, I see what you mean he said quite contritely, and he immediately proceeded to properly bolt his alternator back down. Clearly, God had yet again shown his presence to him. Christianity (and other theologies) are no less logical than the one-boxer in the page you cited. Indeed, the underlying thought process is essentially identical. It is precisely the notion that Nature does not care about our algorithm, which frees us up to pursue the winning Way - without attachment to any particular ritual of cognition, apart from our belief that it wins. Every rule is up for grabs, except the rule of winning. Now, consider that ~50% of our population believes that people who do not believe in God are fundamentally untrustworthy. This tends to work greatly to the disadvantage of atheists, thereby showing that God does indeed favor his believers. After many postings on this subject, I still assert that ANY rational AGI would be religious. Atheism is a radical concept and atheists generally do not do well in our society. What sort of rational belief (like atheism) would work AGAINST winning? In short, your Omega example has apparently made my point - that religious belief IS arguably just as logical (if not more so)than atheism. Do you agree? Thank you. Steve Richfield --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=101455710-f059c4 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Steve, I suspect I'll regret asking, but... Does this rational belief make a difference to intelligence? (For the moment confining the idea of intelligence to making good choices.) If the AGI rationalized the existence of a higher power, what ultimate bad choice do you see as a result? (I've assumed that you have a bias against religion and hence see a big zero or negative in it.) I agree that asking God to hold together what we ought to fix is a bad choice. But then again non-religious folks use bailing wire too. I prefer not to digress into a discussion of religion, but rather stay to the question of potential impact on AGI if such a belief was present in the assumptions of the AGI. If the subject can only lead to religious critiques, please ignore my response. Stan Steve Richfield wrote: Vladamir, On 5/7/08, *Vladimir Nesov* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: See http://www.overcomingbias.com/2008/01/newcombs-proble.html This is a PERFECT talking point for the central point that I have been trying to make. Belief in the Omega discussed early in that article is essentially a religious belief in a greater power. Most Christians see examples of the power of God at around a monthly rate. Whenever chance works for apparent good or against perceived evil, there is clear evidence of God doing his job. Story: A Baptist minister neighbor had his alternator come loose just as he was leaving for an important meeting, so I temporarily secured it with an industrial zip tie, and told him to remove the zip tie and properly bold the alternator back into place when he got back home. Three weeks later, his alternator came loose again. He explained that he had done NOTHING wrong this week, and so he just couldn't see why God took this occasion to smite his alternator. I suggested that we examine it for clues. Sure enough, there were the remnants of my zip tie which he had never replaced. He explained that God seemed to be holding things together OK, so why bother fixing it. Explaining the limitations of industrial zip ties seemed to be hopeless, so I translated my engineering paradigm to his religious paradigm: I explained that he had been testing God by seeing how long God would continue to hold his alternator in place, and apparently God had grown tired of playing this game. Oh, I see what you mean he said quite contritely, and he immediately proceeded to properly bolt his alternator back down. Clearly, God had yet again shown his presence to him. Christianity (and other theologies) are no less logical than the one-boxer in the page you cited. Indeed, the underlying thought process is essentially identical. It is precisely the notion that Nature does not care about our algorithm, which frees us up to pursue the winning Way - without attachment to any particular ritual of cognition, apart from our belief that it wins. Every rule is up for grabs, except the rule of winning. Now, consider that ~50% of our population believes that people who do not believe in God are fundamentally untrustworthy. This tends to work greatly to the disadvantage of atheists, thereby showing that God does indeed favor his believers. After many postings on this subject, I still assert that ANY rational AGI would be religious. Atheism is a radical concept and atheists generally do not do well in our society. What sort of rational belief (like atheism) would work AGAINST winning? In short, your Omega example has apparently made my point - that religious belief IS arguably just as logical (if not more so)than atheism. Do you agree? Thank you. Steve Richfield *agi* | Archives http://www.listbox.com/member/archive/303/=now http://www.listbox.com/member/archive/rss/303/ | Modify http://www.listbox.com/member/?; Your Subscription [Powered by Listbox] http://www.listbox.com --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=101455710-f059c4 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
On 5/7/08, Steve Richfield [EMAIL PROTECTED] wrote: Story: I recently attended an SGI Buddhist meeting with a friend who was a member there. After listening to their discussions, I asked if there was anyone there (from ~30 people) who had ever found themselves in a position of having to kill or injure another person, as I have. There were none, as such experiences tend to change people's outlook on pacifism. Then I mentioned how Herman Kahn's MAD solution to avoiding an almost certain WW3 involved an extremely non-Buddhist approach, gave a thumbnail account of the historical situation, and asked if anyone there had a Buddhist-acceptable solution. Not only was there no other solutions advanced, but they didn't even want to THINK about such things! These people would now be DEAD if not for Herman Kahn, yet they weren't even willing to examine the situation that he found himself in! The ultimate power on earth: An angry 3-year-old with a loaded gun. Hence, I come to quite the opposite solution - that AGIs will want to appear to be IRrational, like the 3-year-old, taking bold steps that force capitulation. Certainly a rational AGI may find it useful to appear irrational, but that doesn't change the conclusion that it'll want to think rationally at the bottom, does it? -- http://www.saunalahti.fi/~tspro1/ | http://xuenay.livejournal.com/ Organizations worth your time: http://www.singinst.org/ | http://www.crnano.org/ | http://lifeboat.com/ --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=101455710-f059c4 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Matt, On 5/6/08, Matt Mahoney [EMAIL PROTECTED] wrote: --- Steve Richfield [EMAIL PROTECTED] wrote: I have played tournament chess. However, when faced with a REALLY GREAT chess player (e.g. national champion), as I have had the pleasure of on a couple of occasions, they at first appear to play as novices, making unusual and apparently stupid moves that I can't quite capitalize on, only to pull things together later on and soundly beat me. While retrospective analysis would show them to be brilliant, that would not be my evaluation early in these games. As your example illustrates, a higher intelligence will appear to be irrational, but you cannot conclude from this that irrationality implies intelligence. Neither does it imply a lack of intelligence. Note that had the master left the table and another good but less than masterful player taken his position, the master's moves would probably have left his replacement at a disadvantage. The test of intelligence is whether it is successful in achieving the desired goal. Irrationality may be a help or a hindrance, depending on how it is applied. I once found myself in the process of being stiffed for $30K by a business associate who clearly had the money, but with no obvious means for me to force collection. Cutting a LONG story short, I collected by composing and sending my associate a copy of a letter to government regulators explaining exactly what the problem was - that would probably have sunk BOTH of our careers - a sort of doomsday machine but still under my control as I held the letter. This worked only because I successfully projected that I really was crazy enough to actually send this letter and sink both of our careers, rather than see $30K successfully stolen from me. Had I projected a calm and calculating mindset, this wouldn't have worked at all. It was at once irrational and brilliantly successful - but only because I projected irrationality/insanity. Steve Richfield --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=101455710-f059c4 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Kaj, On 5/6/08, Kaj Sotala [EMAIL PROTECTED] wrote: Certainly a rational AGI may find it useful to appear irrational, but that doesn't change the conclusion that it'll want to think rationally at the bottom, does it? The concept of rationality contains a large social component. For example, the Eastern concept of face forces actions there that might seem to us to be quite irrational. Polygamy works quite well under Islam, but fails here, because of social perceptions and expectations. Sure, our future AGI must calculate these things, but I suspect that machines will never understand people as well as people do, and hence will never become a serious social force. Take for example the very intelligent people on this forum. We aren't any more economically successful in the world than people with half our our average IQs - or else we would be too busy to make all of these postings. If you are so smart, then why aren't you rich? Of course you know that you have directed your efforts in other directions, but is that path really worth more *to you* than the millions of dollars that you may have left on the table? The whole question of goals also contains a large social component. What is a LOGICAL goal?! Steve Richfield --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=101455710-f059c4 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
On 5/7/08, Kaj Sotala [EMAIL PROTECTED] wrote: Certainly a rational AGI may find it useful to appear irrational, but that doesn't change the conclusion that it'll want to think rationally at the bottom, does it? Oh - and see also http://www.saunalahti.fi/~tspro1/reasons.html , especially parts 5 - 6. -- http://www.saunalahti.fi/~tspro1/ | http://xuenay.livejournal.com/ Organizations worth your time: http://www.singinst.org/ | http://www.crnano.org/ | http://lifeboat.com/ --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=101455710-f059c4 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
On Wed, May 7, 2008 at 11:14 AM, Steve Richfield [EMAIL PROTECTED] wrote: On 5/6/08, Matt Mahoney [EMAIL PROTECTED] wrote: As your example illustrates, a higher intelligence will appear to be irrational, but you cannot conclude from this that irrationality implies intelligence. Neither does it imply a lack of intelligence. Note that had the master left the table and another good but less than masterful player taken his position, the master's moves would probably have left his replacement at a disadvantage. The test of intelligence is whether it is successful in achieving the desired goal. Irrationality may be a help or a hindrance, depending on how it is applied. I think you are using a wrong concept for 'rationality'. It is not a particular procedure, fixed and eternal. If your 'rationality' is bad for achieving your goals, you are not being rational. See http://www.overcomingbias.com/2008/01/newcombs-proble.html It is precisely the notion that Nature does not care about our algorithm, which frees us up to pursue the winning Way - without attachment to any particular ritual of cognition, apart from our belief that it wins. Every rule is up for grabs, except the rule of winning. -- Vladimir Nesov [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=101455710-f059c4 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Steve Richfield wrote: ... have played tournament chess. However, when faced with a REALLY GREAT chess player (e.g. national champion), as I have had the pleasure of on a couple of occasions, they at first appear to play as novices, making unusual and apparently stupid moves that I can't quite capitalize on, only to pull things together later on and soundly beat me. While retrospective analysis would show them to be brilliant, that would not be my evaluation early in these games. Steve Richfield But that's a quite reasonable action on their part. Many players have memorized some number of standard openings. But by taking the game away from the standard openings (or into the less commonly known ones) they enable the player with the stronger chess intuition to gain an edge...and they believe that it will be themselves. E.g.: The Orangutan opening is a trifle weak, but few know it well. But every master would know it, and know both it's strengths and weaknesses. If you don't know the opening, though, it just looks weak. Looks, however, are deceptive. If you don't know it, you're quite likely to find it difficult to deal with against someone who does know it, even if they're a generally weaker player than you are. --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=101455710-f059c4 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Kaj, Richard, et al, On 5/5/08, Kaj Sotala [EMAIL PROTECTED] wrote: Drive 2: AIs will want to be rational This is basically just a special case of drive #1: rational agents accomplish their goals better than irrational ones, and attempts at self-improvement can be outright harmful if you're irrational in the way that you try to improve yourself. If you're trying to modify yourself to better achieve your goals, then you need to make clear to yourself what your goals are. The most effective method for this is to model your goals as a utility function and then modify yourself to better carry out the goals thus specified. Well, again, what exactly do you mean by rational? There are many meanings of this term, ranging from generally sensible to strictly following a mathematical logic. Rational agents accomplish their goals better than irrational ones? Can this be proved? And with what assumptions? Which goals are better accomplished is the goal of being rational better accomplished by being rational? Is the goal of generating a work of art that has true genuineness something that needs rationality? And if a system is trying to modify itself to better achieve its goals, what if it decides that just enjoying the subjective experience of life is good enough as a goal, and then realizes that it will not get more of that by becoming more rational? This was somewhat wrung out in the 1950s by Herman Kahn of the RAND Corp, who is credited with inventing MAD (Mutually Assured Destruction) built on vengeance, etc. Level1: People are irrational, so a rational path may play on that irrationality, and hence be irrational against an unemotional opponent. Level 2: By appearing to be irrational you also appear to be dangerous/violent, and hence there is POWER in apparent irrationality, most especially if on a national and thermonuclear scale. Hence, a maximally capable AGI may appear to be quite crazy to us all-too-human observers. Story: I recently attended an SGI Buddhist meeting with a friend who was a member there. After listening to their discussions, I asked if there was anyone there (from ~30 people) who had ever found themselves in a position of having to kill or injure another person, as I have. There were none, as such experiences tend to change people's outlook on pacifism. Then I mentioned how Herman Kahn's MAD solution to avoiding an almost certain WW3 involved an extremely non-Buddhist approach, gave a thumbnail account of the historical situation, and asked if anyone there had a Buddhist-acceptable solution. Not only was there no other solutions advanced, but they didn't even want to THINK about such things! These people would now be DEAD if not for Herman Kahn, yet they weren't even willing to examine the situation that he found himself in! The ultimate power on earth: An angry 3-year-old with a loaded gun. Hence, I come to quite the opposite solution - that AGIs will want to appear to be IRrational, like the 3-year-old, taking bold steps that force capitulation. I have played tournament chess. However, when faced with a REALLY GREAT chess player (e.g. national champion), as I have had the pleasure of on a couple of occasions, they at first appear to play as novices, making unusual and apparently stupid moves that I can't quite capitalize on, only to pull things together later on and soundly beat me. While retrospective analysis would show them to be brilliant, that would not be my evaluation early in these games. Steve Richfield --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=101455710-f059c4 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
--- Steve Richfield [EMAIL PROTECTED] wrote: I have played tournament chess. However, when faced with a REALLY GREAT chess player (e.g. national champion), as I have had the pleasure of on a couple of occasions, they at first appear to play as novices, making unusual and apparently stupid moves that I can't quite capitalize on, only to pull things together later on and soundly beat me. While retrospective analysis would show them to be brilliant, that would not be my evaluation early in these games. As your example illustrates, a higher intelligence will appear to be irrational, but you cannot conclude from this that irrationality implies intelligence. -- Matt Mahoney, [EMAIL PROTECTED] --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=101455710-f059c4 Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Richard, again, I must sincerely apologize for responding to this so horrendously late. It's a dreadful bad habit of mine: I get an e-mail (or blog comment, or forum message, or whatever) that requires some thought before I respond, so I don't answer it right away... and then something related to my studies or hobbies shows up and doesn't leave me with enough energy to compose responses to anybody at all, after which enough time has passed that the message has vanished from my active memory, and when I remember it so much time has passed already that a day or two more before I answer won't make any difference... and then *so* much time has passed that replying to the message so late feels more embarassing than just quietly forgetting about it. I'll try to better my ways in the future. On the same token, I must say I can only admire your ability to compose long, well-written replies to messages in what seem to be blinks of an eye to me. :-) On 3/11/08, Richard Loosemore [EMAIL PROTECTED] wrote: Kaj Sotala wrote: On 3/3/08, Richard Loosemore [EMAIL PROTECTED] wrote: Kaj Sotala wrote: Alright. But previously, you said that Omohundro's paper, which to me seemed to be a general analysis of the behavior of *any* minds with (more or less) explict goals, looked like it was based on a 'goal-stack' motivation system. (I believe this has also been the basis of your critique for e.g. some SIAI articles about friendliness.) If built-in goals *can* be constructed into motivational system AGIs, then why do you seem to assume that AGIs with built-in goals are goal-stack ones? I seem to have caused lots of confusion earlier on in the discussion, so let me backtrack and try to summarize the structure of my argument. 1) Conventional AI does not have a concept of a Motivational-Emotional System (MES), the way that I use that term, so when I criticised Omuhundro's paper for referring only to a Goal Stack control system, I was really saying no more than that he was assuming that the AI was driven by the system that all conventional AIs are supposed to have. These two ways of controlling an AI are two radically different designs. [...] So now: does that clarify the specific question you asked above? Yes and no. :-) My main question is with part 1 of your argument - you are saying that Omohundro's paper assumed the AI to have a certain sort of control system. This is the part which confuses me, since I didn't see the paper to make *any* mentions of how the AI should be built. It only assumes that the AI has some sort of goals, and nothing more. [...] Drive 1: AIs will want to self-improve This one seems fairly straightforward: indeed, for humans self-improvement seems to be an essential part in achieving pretty much *any* goal you are not immeaditly capable of achieving. If you don't know how to do something needed to achieve your goal, you practice, and when you practice, you're improving yourself. Likewise, improving yourself will quickly become a subgoal for *any* major goals. But now I ask: what exactly does this mean? In the context of a Goal Stack system, this would be represented by a top level goal that was stated in the knowledge representation language of the AGI, so it would say Improve Thyself. [...] The reason that I say Omuhundro is assuming a Goal Stack system is that I believe he would argue that that is what he meant, and that he assumed that a GS architecture would allow the AI to exhibit behavior that corresponds to what we, as humans, recognize as wanting to self-improve. I think it is a hidden assumption in what he wrote. At least I didn't read the paper in such a way - after all, the abstract says that it's supposed to apply equally to all AGI systems, regardless of the exact design: We identify a number of drives that will appear in sufficiently advanced AI systems of any design. We call them drives because they are tendencies which will be present unless explicitly counteracted. (You could, of course, suppose that the author was assuming that an AGI could *only* be built around a Goal Stack system, and therefore any design would mean any GS design... but that seems a bit far-fetched.) Drive 2: AIs will want to be rational This is basically just a special case of drive #1: rational agents accomplish their goals better than irrational ones, and attempts at self-improvement can be outright harmful if you're irrational in the way that you try to improve yourself. If you're trying to modify yourself to better achieve your goals, then you need to make clear to yourself what your goals are. The most effective method for this is to model your goals as a utility function and then modify yourself to better carry out the goals thus specified. Well, again, what exactly do you mean by rational? There are many meanings of this term,
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Charles D Hixson wrote: Richard Loosemore wrote: Kaj Sotala wrote: On 3/3/08, Richard Loosemore [EMAIL PROTECTED] wrote: ... goals. But now I ask: what exactly does this mean? In the context of a Goal Stack system, this would be represented by a top level goal that was stated in the knowledge representation language of the AGI, so it would say Improve Thyself. Next, it would subgoal this (break it down into subgoals). Since the top level goal is so unbelievably vague, there are a billion different ways to break this down into subgoals: it might get out a polishing cloth and start working down its beautiful shiny exterior, or it might start a transistor-by-transistor check of all its circuits, or all the way up to taking a course in Postmodern critiques of the Postmodern movement. And included in that range of improvement activities would be the possibility of something like Improve my ability to function efficiently which gets broken down into subgoals like Remove all sources of distraction that reduce efficiency and then Remove all humans, because they are a distraction. My point here is that a Goal Stack system would *interpret* this goal in any one of an infinite number of ways, because the goal was represented as an explicit statement. The fact that it was represented explicitly meant that an extremely vague concept (Improve Thyself) had to be encoded in such a way as to leave it open to ambiguity. As a result, what the AGI actually does as a result of this goal, which is embedded in a Goal Stack architecture, is completely indeterminate. Stepping back from the detail, we can notice that *any* vaguely worded goal is going to have the same problem in a GS architecture. And if we dwell on that for a moment, we start to wonder exactly what would happen to an AGI that was driven by goals that had to be stated in vague terms ... will the AGI *ever* exhibit coherent, intelligent behavior when driven by such a GS drive system, or will it have flashes of intelligence puncuated by the wild pursuit of bizarre obsessions? Will it even have flashes of intelligence? So long as the goals that are fed into a GS architecture are very, very local and specific (like Put the red pyramid on top of the green block) I can believe that the GS drive system does actually work (kind of). But no one has ever built an AGI that way. Never. Everyone assumes that a GS will scale up to a vague goal like Improve Thyself, and yet no one has tried this in practice. Not on a system that is supposed to be capable of a broad-based, autonomous, *general* intelligence. So when you paraphrase Omuhundro as saying that AIs will want to self-improve, the meaning of that statement is impossible to judge. ... Perhaps I don't understand Goal-Stack System. You seem to be presuming that the actual implementation would involve statements in English (or some equivalent language). To me it seems more as if Goals would be represented as internal states reflecting such things as sensor state and projected sensor state, etc. Thus Improve yourself would need to be represented by something which would be more precisely translated into English as something like change your program so that a smaller or faster implementation will predictions regarding future sensor states that are no worse than the current version of the program. (I'm not at all clear how something involving external objects could be encoded here. Humans seem to imprint on a human face, and work out from there. This implies a predictable sensory configuration for the initial state, but there must also be backup mechanisms, or Helen Keller would have been totally asocial.) At all events, the different versions of improve yourself that you mentioned would seem to require different internal representations. Also, the existence of one goal doesn't preclude the existence of other goals, and which goal was of top priority would be expected to shift over time. Additionally, any proposed method of reaching a goal would have costs as well as benefits. No AGI would reasonably have only a few high level goals, so rather than *a* goal stack, even with my interpretation there would need to be several of them. To me it seems as if the problem that you are foreseeing is more to be expected from a really powerful narrow AI than from an AGI. The idea of a Goal Stack drive mechanism is my term for the only idea currently on the table from the conventional AGI folks: a stack of explicitly represented goals. I have written the top level goal statement in English only as a paraphrase for the real representation (which would be in the AGI's internal representation language). But although the real version would look more logical and less English, it could not be of the sort you suggest, with just a simple function over sensor states: after all, what I am doing is trying to ask what the conventional
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
On 3/3/08, Richard Loosemore [EMAIL PROTECTED] wrote: Kaj Sotala wrote: Alright. But previously, you said that Omohundro's paper, which to me seemed to be a general analysis of the behavior of *any* minds with (more or less) explict goals, looked like it was based on a 'goal-stack' motivation system. (I believe this has also been the basis of your critique for e.g. some SIAI articles about friendliness.) If built-in goals *can* be constructed into motivational system AGIs, then why do you seem to assume that AGIs with built-in goals are goal-stack ones? I seem to have caused lots of confusion earlier on in the discussion, so let me backtrack and try to summarize the structure of my argument. 1) Conventional AI does not have a concept of a Motivational-Emotional System (MES), the way that I use that term, so when I criticised Omuhundro's paper for referring only to a Goal Stack control system, I was really saying no more than that he was assuming that the AI was driven by the system that all conventional AIs are supposed to have. These two ways of controlling an AI are two radically different designs. [...] So now: does that clarify the specific question you asked above? Yes and no. :-) My main question is with part 1 of your argument - you are saying that Omohundro's paper assumed the AI to have a certain sort of control system. This is the part which confuses me, since I didn't see the paper to make *any* mentions of how the AI should be built. It only assumes that the AI has some sort of goals, and nothing more. I'll list all of the drives Omohundro mentions, and my interpretation of them and why they only require existing goals. Please correct me where our interpretations differ. (It is true that it will be possible to reduce the impact of many of these drives by constructing an architecture which restricts them, and as such they are not /unavoidable/ ones - however, it seems reasonable to assume that they will by default emerge in any AI with goals, unless specifically counteracted. Also, the more that they are restricted, the less effective the AI will be.) Drive 1: AIs will want to self-improve This one seems fairly straightforward: indeed, for humans self-improvement seems to be an essential part in achieving pretty much *any* goal you are not immeaditly capable of achieving. If you don't know how to do something needed to achieve your goal, you practice, and when you practice, you're improving yourself. Likewise, improving yourself will quickly become a subgoal for *any* major goals. Drive 2: AIs will want to be rational This is basically just a special case of drive #1: rational agents accomplish their goals better than irrational ones, and attempts at self-improvement can be outright harmful if you're irrational in the way that you try to improve yourself. If you're trying to modify yourself to better achieve your goals, then you need to make clear to yourself what your goals are. The most effective method for this is to model your goals as a utility function and then modify yourself to better carry out the goals thus specified. Drive 3: AIs will want to preserve their utility functions Since the utility function constructed was a model of the AI's goals, this drive is equivalent to saying AIs will want to preserve their goals (or at least the goals that are judged as the most important ones). The reasoning for this should be obvious - if a goal is removed from the AI's motivational system, the AI won't work to achieve the goal anymore, which is bad from the point of view of an AI that currently does want the goal to be achieved. Drive 4: AIs try to prevent counterfeit utility This is an extension of drive #2: if there are things in the environment that hijack existing motivation systems to make the AI do things not relevant for its goals, then it will attempt to modify its motivation systems to avoid those vulnerabilities. Drive 5: AIs will be self-protective This is a special case of #3. Drive 6: AIs will want to acquire resources and use them efficiently More resources will help in achieving most goals: also, even if you had already achieved all your goals, more resources would help you in making sure that your success wouldn't be thwarted as easily. -- http://www.saunalahti.fi/~tspro1/ | http://xuenay.livejournal.com/ Organizations worth your time: http://www.singinst.org/ | http://www.crnano.org/ | http://lifeboat.com/ --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Ahah! :-) Upon reading Kaj's excellent reply, I spotted something that I missed before that grated on Richard (and he even referred to it though I didn't realize it at the time) . . . . The Omohundro drives #3 and #4 need to be rephrased from Drive 3: AIs will want to preserve their utility functions Drive 4: AIs try to prevent counterfeit utility to Drive 3: AIs will want to preserve their goals Drive 4: AIs will want to prevent fake feedback on the status of their goals The current phrasing *DOES* seem to strongly suggest a goal-stack type architecture since, although I argued that a MES system has an implicit utility function that it just doesn't refer to it, it makes no sense that it is trying to preserve and prevent counterfeits of something that it ignores. sorry for missing/overlooking this before, Richard :- (And this is why I'm running all this past the mailing list before believing that my paper is anywhere close to final :-) - Original Message - From: Kaj Sotala [EMAIL PROTECTED] To: agi@v2.listbox.com Sent: Tuesday, March 11, 2008 10:07 AM Subject: Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...] On 3/3/08, Richard Loosemore [EMAIL PROTECTED] wrote: Kaj Sotala wrote: Alright. But previously, you said that Omohundro's paper, which to me seemed to be a general analysis of the behavior of *any* minds with (more or less) explict goals, looked like it was based on a 'goal-stack' motivation system. (I believe this has also been the basis of your critique for e.g. some SIAI articles about friendliness.) If built-in goals *can* be constructed into motivational system AGIs, then why do you seem to assume that AGIs with built-in goals are goal-stack ones? I seem to have caused lots of confusion earlier on in the discussion, so let me backtrack and try to summarize the structure of my argument. 1) Conventional AI does not have a concept of a Motivational-Emotional System (MES), the way that I use that term, so when I criticised Omuhundro's paper for referring only to a Goal Stack control system, I was really saying no more than that he was assuming that the AI was driven by the system that all conventional AIs are supposed to have. These two ways of controlling an AI are two radically different designs. [...] So now: does that clarify the specific question you asked above? Yes and no. :-) My main question is with part 1 of your argument - you are saying that Omohundro's paper assumed the AI to have a certain sort of control system. This is the part which confuses me, since I didn't see the paper to make *any* mentions of how the AI should be built. It only assumes that the AI has some sort of goals, and nothing more. I'll list all of the drives Omohundro mentions, and my interpretation of them and why they only require existing goals. Please correct me where our interpretations differ. (It is true that it will be possible to reduce the impact of many of these drives by constructing an architecture which restricts them, and as such they are not /unavoidable/ ones - however, it seems reasonable to assume that they will by default emerge in any AI with goals, unless specifically counteracted. Also, the more that they are restricted, the less effective the AI will be.) Drive 1: AIs will want to self-improve This one seems fairly straightforward: indeed, for humans self-improvement seems to be an essential part in achieving pretty much *any* goal you are not immeaditly capable of achieving. If you don't know how to do something needed to achieve your goal, you practice, and when you practice, you're improving yourself. Likewise, improving yourself will quickly become a subgoal for *any* major goals. Drive 2: AIs will want to be rational This is basically just a special case of drive #1: rational agents accomplish their goals better than irrational ones, and attempts at self-improvement can be outright harmful if you're irrational in the way that you try to improve yourself. If you're trying to modify yourself to better achieve your goals, then you need to make clear to yourself what your goals are. The most effective method for this is to model your goals as a utility function and then modify yourself to better carry out the goals thus specified. Drive 3: AIs will want to preserve their utility functions Since the utility function constructed was a model of the AI's goals, this drive is equivalent to saying AIs will want to preserve their goals (or at least the goals that are judged as the most important ones). The reasoning for this should be obvious - if a goal is removed from the AI's motivational system, the AI won't work to achieve the goal anymore, which is bad from the point of view of an AI that currently does want the goal to be achieved. Drive 4: AIs try to prevent counterfeit utility This is an extension of drive #2: if there are things in the environment that hijack existing motivation systems
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Drive 1: AIs will want to self-improve This one seems fairly straightforward: indeed, for humans self-improvement seems to be an essential part in achieving pretty much *any* goal you are not immeaditly capable of achieving. If you don't know how to do something needed to achieve your goal, you practice, and when you practice, you're improving yourself. Likewise, improving yourself will quickly become a subgoal for *any* major goals. But now I ask: what exactly does this mean? It means that they will want to improve their ability to achieve their goals (i.e. in an MES system, optimize their actions/reactions to more closely correspond to what is indicated/appropriate for their urges and constraints). In the context of a Goal Stack system, this would be represented by a top level goal that was stated in the knowledge representation language of the AGI, so it would say Improve Thyself. One of the shortcomings of your current specification of the MES system is that it does not, at the simplest levels, provide a mechanism for globally optimizing (increasing the efficiency of) the system. This makes it safer because such a mechanism *would* conceivably be a single point of failure for Friendliness but evolution will favor the addition of any such a system -- as would any humans that would like a system to improve itself. I don't currently see how an MES system could be a seed AGI unless such a system is added. My point here is that a Goal Stack system would *interpret* this goal in any one of an infinite number of ways, because the goal was represented as an explicit statement. The fact that it was represented explicitly meant that an extremely vague concept (Improve Thyself) had to be encoded in such a way as to leave it open to ambiguity. As a result, what the AGI actually does as a result of this goal, which is embedded in a Goal Stack architecture, is completely indeterminate. Oh. I disagree *entirely*. It is only indeterminate because you gave it an indeterminate goal with *no* evaluation criteria. Now, I *assume* that you ACTUALLY mean Improve Thyself So That You Are More Capable Of Achieving An Arbitrary Set Of Goals To Be Specified Later and I would argue that the most effective way for the system to do so is to increase it's intelligence (the single-player version of goal-achieving ability) and friendliness (the multi-player version of intelligence). Stepping back from the detail, we can notice that *any* vaguely worded goal is going to have the same problem in a GS architecture. But I've given a more explicitly worded goal that *should* (I believe) drive a system to intelligence. The long version of Improve Thyself is the necessary motivating force for a seed AI. Do you have a way to add it to an MES system? If you can't, then I would have to argue that an MES system will never achieve intelligence (though I'm very hopeful that either we can add it to the MES *or* there is some form of hybrid system that has the advantages of both and disadvantages of neither). So long as the goals that are fed into a GS architecture are very, very local and specific (like Put the red pyramid on top of the green block) I can believe that the GS drive system does actually work (kind of). But no one has ever built an AGI that way. Never. Everyone assumes that a GS will scale up to a vague goal like Improve Thyself, and yet no one has tried this in practice. Not on a system that is supposed to be capable of a broad-based, autonomous, *general* intelligence. Well, actually I'm claiming that *any* optimizing system with the long version of Improve Thyself that is sufficiently capable is a seed AI. The problem is that sufficiently capable seems to be a relatively high bar -- particularly when we, as humans, don't even know which way is up. My Friendliness theory is (at least) an attempt to identify up. So when you paraphrase Omuhundro as saying that AIs will want to self-improve, the meaning of that statement is impossible to judge. As evidenced by my last several e-mails, the best paraphrase of Omohundro is Goal-achievement optimizing AIs will want to self-improve so that they are more capable of achieving goals which is basically a definition or a tautology. The reason that I say Omuhundro is assuming a Goal Stack system is that I believe he would argue that that is what he meant, and that he assumed that a GS architecture would allow the AI to exhibit behavior that corresponds to what we, as humans, recognize as wanting to self-improve. I think it is a hidden assumption in what he wrote. Optimizing *is* a hidden assumption in what he wrote which you caused me to catch later and add to my base assumption. I don't believe that optimizing necessarily assumes a Goal Stack system but it *DOES* assume a self-reflecting system which the MES system does not appear to be (yet) at the lowest levels. In order
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Kaj Sotala wrote: On 2/16/08, Richard Loosemore [EMAIL PROTECTED] wrote: Kaj Sotala wrote: Well, the basic gist was this: you say that AGIs can't be constructed with built-in goals, because a newborn AGI doesn't yet have built up the concepts needed to represent the goal. Yet humans seem tend to have built-in (using the term a bit loosely, as all goals do not manifest in everyone) goals, despite the fact that newborn humans don't yet have built up the concepts needed to represent those goals. Oh, complete agreement here. I am only saying that the idea of a built-in goal cannot be made to work in an AGI *if* one decides to build that AGI using a goal-stack motivation system, because the latter requires that any goals be expressed in terms of the system's knowledge. If we step away from that simplistic type of GS system, and instead use some other type of motivation system, then I believe it is possible for the system to be motivated in a coherent way, even before it has the explicit concepts to talk about its motivations (it can pursue the goal seek Momma's attention long before it can explicitly represent the concept of [attention], for example). Alright. But previously, you said that Omohundro's paper, which to me seemed to be a general analysis of the behavior of *any* minds with (more or less) explict goals, looked like it was based on a 'goal-stack' motivation system. (I believe this has also been the basis of your critique for e.g. some SIAI articles about friendliness.) If built-in goals *can* be constructed into motivational system AGIs, then why do you seem to assume that AGIs with built-in goals are goal-stack ones? I seem to have caused lots of confusion earlier on in the discussion, so let me backtrack and try to summarize the structure of my argument. 1) Conventional AI does not have a concept of a Motivational-Emotional System (MES), the way that I use that term, so when I criticised Omuhundro's paper for referring only to a Goal Stack control system, I was really saying no more than that he was assuming that the AI was driven by the system that all conventional AIs are supposed to have. These two ways of controlling an AI are two radically different designs. 2) Not only are MES and GS different classes of drive mechanism, they also make very different assumptions about the general architecture of the AI. When I try to explain how an MES works, I often get tangled up in the problem of explaining the general architecture that lies behind it (which does, I admit, cause much confusion). I sometimes use the terms molecular or sub-symbolic to describe that architecture. 2(a) I should say something about the architecture difference. In a sub-symbolic architecture you would find that the significant thought events are the result of clouds of sub-symbolic elements interacting with one another across a broad front. This is to be contrasted with the way that symbols interact in a regular symbolic AI, where symbols are single entities that get plugged into well-defined mechanisms like deduction operators. In a sub-symbolic system, operations are usually the result of several objects *constraining* one another in a relatively weak manner, not the result of a very small number of objects slotting into a precisely defined, rigid mechanism. There is a flexibility inherent in the sub-symbolic architecture that is completely lacking in the conventional symbolic system. 3) It is important to understand that in an AI that uses the MES drive system, there is *also* a goal stack, quite similar to what is found in a GS-driven AI, but this goal stack is entirely subservient to the MES, and it plays a role only in the day to day (and moment to moment) thinking of the system. 4) I plead guilty to saying things like ... Goal-Stack motivation system... when what I should do is use the word motivation only in the context of an MES system. A better wording would have been ... Goal-Stack *drive* system Or perhaps ... Goal-Stack *control* system 5) The main thrust of my attack on GS-driven AIs is that goal stacks were invented in the context of planning problems, and were never intended to be used as the global control system for an AI that is capable of long-range development. So, you will find me saying things like A GS drive system is appropriate for handling goals like 'Put the red pyramid on top of the green block', but it makes no sense in the context of goals like 'Be friendly to humans'. Most AI people assume that a GS control system *must* be the way to go, but I would argue that they are in denial about the uselessness of a GS. Also, most conventional AI people assume that a GS is valid simply because they see no alternative ... and this is because the architecture used by most conventional AI does not easily admit of any other type of drive system. In a sense, they have to support the GS idea because
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
On 2/16/08, Richard Loosemore [EMAIL PROTECTED] wrote: Kaj Sotala wrote: Well, the basic gist was this: you say that AGIs can't be constructed with built-in goals, because a newborn AGI doesn't yet have built up the concepts needed to represent the goal. Yet humans seem tend to have built-in (using the term a bit loosely, as all goals do not manifest in everyone) goals, despite the fact that newborn humans don't yet have built up the concepts needed to represent those goals. Oh, complete agreement here. I am only saying that the idea of a built-in goal cannot be made to work in an AGI *if* one decides to build that AGI using a goal-stack motivation system, because the latter requires that any goals be expressed in terms of the system's knowledge. If we step away from that simplistic type of GS system, and instead use some other type of motivation system, then I believe it is possible for the system to be motivated in a coherent way, even before it has the explicit concepts to talk about its motivations (it can pursue the goal seek Momma's attention long before it can explicitly represent the concept of [attention], for example). Alright. But previously, you said that Omohundro's paper, which to me seemed to be a general analysis of the behavior of *any* minds with (more or less) explict goals, looked like it was based on a 'goal-stack' motivation system. (I believe this has also been the basis of your critique for e.g. some SIAI articles about friendliness.) If built-in goals *can* be constructed into motivational system AGIs, then why do you seem to assume that AGIs with built-in goals are goal-stack ones? The way to get around that problem is to notice two things. One is that the sex drives can indeed be there from the very beginning, but in very mild form, just waiting to be kicked into high gear later on. I think this accounts for a large chunk of the explanation (there is evidence for this: some children are explictly thinking engaged in sex-related activities at the age of three, at least). The second part of the explanation is that, indeed, the human mind *does* have trouble making a an easy connection to those later concepts: sexual ideas do tend to get attached to the most peculiar behaviors. Perhaps this is a sigh that the hook-up process is not straightforward. This sounds like the beginnings of the explanation, yes. -- http://www.saunalahti.fi/~tspro1/ | http://xuenay.livejournal.com/ Organizations worth your time: http://www.singinst.org/ | http://www.crnano.org/ | http://lifeboat.com/ --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
interesting you're attempting that via goals, because goals will mutate; one alternative is to control the infrastructure eg have systems that die when they've run a certain course., and watcher systems that check mutations. - Original Message - From: Kaj Sotala [EMAIL PROTECTED] To: agi@v2.listbox.com Subject: Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...] Date: Sun, 2 Mar 2008 19:58:28 +0200 On 2/16/08, Richard Loosemore [EMAIL PROTECTED] wrote: Kaj Sotala wrote: Well, the basic gist was this: you say that AGIs can't be constructed with built-in goals, because a newborn AGI doesn't yet have built up the concepts needed to represent the goal. Yet humans seem tend to have built-in (using the term a bit loosely, as all goals do not manifest in everyone) goals, despite the fact that newborn humans don't yet have built up the concepts needed to represent those goals. Oh, complete agreement here. I am only saying that the idea of a built-in goal cannot be made to work in an AGI *if* one decides to build that AGI using a goal-stack motivation system, because the latter requires that any goals be expressed in terms of the system's knowledge. If we step away from that simplistic type of GS system, and instead use some other type of motivation system, then I believe it is possible for the system to be motivated in a coherent way, even before it has the explicit concepts to talk about its motivations (it can pursue the goal seek Momma's attention long before it can explicitly represent the concept of [attention], for example). Alright. But previously, you said that Omohundro's paper, which to me seemed to be a general analysis of the behavior of *any* minds with (more or less) explict goals, looked like it was based on a 'goal-stack' motivation system. (I believe this has also been the basis of your critique for e.g. some SIAI articles about friendliness.) If built-in goals *can* be constructed into motivational system AGIs, then why do you seem to assume that AGIs with built-in goals are goal-stack ones? The way to get around that problem is to notice two things. One is that the sex drives can indeed be there from the very beginning, but in very mild form, just waiting to be kicked into high gear later on. I think this accounts for a large chunk of the explanation (there is evidence for this: some children are explictly thinking engaged in sex-related activities at the age of three, at least). The second part of the explanation is that, indeed, the human mind *does* have trouble making a an easy connection to those later concepts: sexual ideas do tend to get attached to the most peculiar behaviors. Perhaps this is a sigh that the hook-up process is not straightforward. This sounds like the beginnings of the explanation, yes. -- http://www.saunalahti.fi/~tspro1/ | http://xuenay.livejournal.com/ Organizations worth your time: http://www.singinst.org/ | http://www.crnano.org/ | http://lifeboat.com/ --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?; Powered by Listbox: http://www.listbox.com -- Want an e-mail address like mine? Get a free e-mail account today at www.mail.com! --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Gah, sorry for the awfully late response. Studies aren't leaving me the energy to respond to e-mails more often than once in a blue moon... On Feb 4, 2008 8:49 PM, Richard Loosemore [EMAIL PROTECTED] wrote: They would not operate at the proposition level, so whatever difficulties they have, they would at least be different. Consider [curiosity]. What this actually means is a tendency for the system to seek pleasure in new ideas. Seeking pleasure is only a colloquial term for what (in the system) would be a dimension of constraint satisfaction (parallel, dynamic, weak-constraint satisfaction). Imagine a system in which there are various micro-operators hanging around, which seek to perform certain operations on the structures that are currently active (for example, there will be several micro-operators whose function is to take a representation such as [the cat is sitting on the mat] and try to investigate various WHY questions about the representation (Why is this cat sitting on this mat? Why do cats in general like to sit on mats? Why does this cat Fluffy always like to sit on mats? Does Fluffy like to sit on other things? Where does the phrase 'the cat sat on the mat' come from? And so on). [cut the rest] Interesting. This sounds like it might be workable, though of course, the exact assosciations and such that the AGI develops sound hard to control. But then, that'd be the case for any real AGI system... Humans have lots of desires - call them goals or motivations - that manifest in differing degrees in different individuals, like wanting to be respected or wanting to have offspring. Still, excluding the most basic ones, they're all ones that a newborn child won't understand or feel before (s)he gets older. You could argue that they can't be inborn goals since the newborn mind doesn't have the concepts to represent them and because they manifest variably with different people (not everyone wants to have children, and there are probably even people who don't care about the respect of others), but still, wouldn't this imply that AGIs *can* be created with in-built goals? Or if such behavior can only be implemented with a motivational-system AI, how does that avoid the problem of some of the wanted final motivations being impossible to define in the initial state? I must think about this more carefully, because I am not quite sure of the question. However, note that we (humans) probably do not get many drives that are introduced long after childhood, and that the exceptions (sex, motherhood desires, teenage rebellion) could well be sudden increases in the power of drives that were there from the beginning. Ths may not have been your question, so I will put this one on hold. Well, the basic gist was this: you say that AGIs can't be constructed with built-in goals, because a newborn AGI doesn't yet have built up the concepts needed to represent the goal. Yet humans seem tend to have built-in (using the term a bit loosely, as all goals do not manifest in everyone) goals, despite the fact that newborn humans don't yet have built up the concepts needed to represent those goals. It is true that many of those drives seem to begin in early childhood, but it seems to me that there are still many goals that aren't activated until after infancy, such as the drive to have children. -- http://www.saunalahti.fi/~tspro1/ | http://xuenay.livejournal.com/ Organizations worth your time: http://www.singinst.org/ | http://www.crnano.org/ | http://lifeboat.com/ --- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Kaj Sotala wrote: Richard, [Where's your blog? Oh, and this is a very useful discussion, as it's given me material for a possible essay of my own as well. :-)] It is in the process of being set up: I am currently wrestling with the process of getting to know the newest version (just released a few days ago) of the Joomla content management system, so this has put yet another delay in my plans. Will let you know as soon as it is in a respectable state. I will give (again) quick responses to some of your questions. Thanks for the answer. Here's my commentary - I quote and respond to parts of your message somewhat out of order, since there were some issues about ethics scattered throughout your mail that I felt were best answered with a single response. The most important reason that I think this type will win out over a goal-stack system is that I really think the latter cannot be made to work in a form that allows substantial learning. A goal-stack control system relies on a two-step process: build your stack using goals that are represented in some kind of propositonal form, and then (when you are ready to pursue a goal) *interpret* the meaning of the proposition on the top of the stack so you can start breaking it up into subgoals. The problem with this two-step process is that the interpretation of each goal is only easy when you are down at the lower levels of the stack - Pick up the red block is easy to interpret, but Make humans happy is a profoundly abstract statement that has a million different interpretations. This is one reason why nobody has build an AGI. To make a completely autonomous system that can do such things as learn by engaging in exploratory behavior, you have to be able insert goals like Do some playing, and there is no clear way to break that statement down into unambiguous subgoals. The result is that if you really did try to build an AGI with a goal like that, the actual behavior of the system would be wildly unpredictable, and probably not good for the system itself. Further: if the system is to acquire its own knowledge independently from a child-like state (something that, for separate reasons, I think is going to be another prerequisite for true AGI), then the child system cannot possibly have goals built into it that contain statements like Engage in an empathic relationship with your parents because it does not have the knowledge base built up yet, and cannot understand such a propositions! I agree that it could very well be impossible to define explict goals for a child AGI, as it doesn't have enough built up knowledge to understand the propositions involved. I'm not entirely sure of how the motivation approach avoids this problem, though - you speak of setting up an AGI with motivations resembling the ones we'd call curiosity or empathy. How are these, then, defined? Wouldn't they run into the same difficulties? They would not operate at the proposition level, so whatever difficulties they have, they would at least be different. Consider [curiosity]. What this actually means is a tendency for the system to seek pleasure in new ideas. Seeking pleasure is only a colloquial term for what (in the system) would be a dimension of constraint satisfaction (parallel, dynamic, weak-constraint satisfaction). Imagine a system in which there are various micro-operators hanging around, which seek to perform certain operations on the structures that are currently active (for example, there will be several micro-operators whose function is to take a representation such as [the cat is sitting on the mat] and try to investigate various WHY questions about the representation (Why is this cat sitting on this mat? Why do cats in general like to sit on mats? Why does this cat Fluffy always like to sit on mats? Does Fluffy like to sit on other things? Where does the phrase 'the cat sat on the mat' come from? And so on). Now, if a person were absolutely consumed by curiosity, the class of operators that tended to unpack in this particular way would be given license to activate a great deal. Why would the person do this? Because during the course of the person's development these operators tended to cause a particular type of event to occur, and that type of event would be what we might call a discovery-pleasure event a situation where the brain put some ideas together in a way that suddenly caused some representations to collapse into a particularly simple form (think of Kekule suddenly realising that the ring of snakes in his dream could be seen as a ring of connected carbon atoms, which then suddenly causes all of his knowledge of the structure of Benzene to fall into a simple form). Now, these collapse events will specifically trigger a certain type of signal which basically amounts to a reward for causing the discovery event, which in turn is the same as getting pleasure from the activity of discovery of ideas, and as a
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
On 1/30/08, Richard Loosemore [EMAIL PROTECTED] wrote: Kaj, [This is just a preliminary answer: I am composing a full essay now, which will appear in my blog. This is such a complex debate that it needs to be unpacked in a lot more detail than is possible here. Richard]. Richard, [Where's your blog? Oh, and this is a very useful discussion, as it's given me material for a possible essay of my own as well. :-)] Thanks for the answer. Here's my commentary - I quote and respond to parts of your message somewhat out of order, since there were some issues about ethics scattered throughout your mail that I felt were best answered with a single response. The most important reason that I think this type will win out over a goal-stack system is that I really think the latter cannot be made to work in a form that allows substantial learning. A goal-stack control system relies on a two-step process: build your stack using goals that are represented in some kind of propositonal form, and then (when you are ready to pursue a goal) *interpret* the meaning of the proposition on the top of the stack so you can start breaking it up into subgoals. The problem with this two-step process is that the interpretation of each goal is only easy when you are down at the lower levels of the stack - Pick up the red block is easy to interpret, but Make humans happy is a profoundly abstract statement that has a million different interpretations. This is one reason why nobody has build an AGI. To make a completely autonomous system that can do such things as learn by engaging in exploratory behavior, you have to be able insert goals like Do some playing, and there is no clear way to break that statement down into unambiguous subgoals. The result is that if you really did try to build an AGI with a goal like that, the actual behavior of the system would be wildly unpredictable, and probably not good for the system itself. Further: if the system is to acquire its own knowledge independently from a child-like state (something that, for separate reasons, I think is going to be another prerequisite for true AGI), then the child system cannot possibly have goals built into it that contain statements like Engage in an empathic relationship with your parents because it does not have the knowledge base built up yet, and cannot understand such a propositions! I agree that it could very well be impossible to define explict goals for a child AGI, as it doesn't have enough built up knowledge to understand the propositions involved. I'm not entirely sure of how the motivation approach avoids this problem, though - you speak of setting up an AGI with motivations resembling the ones we'd call curiosity or empathy. How are these, then, defined? Wouldn't they run into the same difficulties? Humans have lots of desires - call them goals or motivations - that manifest in differing degrees in different individuals, like wanting to be respected or wanting to have offspring. Still, excluding the most basic ones, they're all ones that a newborn child won't understand or feel before (s)he gets older. You could argue that they can't be inborn goals since the newborn mind doesn't have the concepts to represent them and because they manifest variably with different people (not everyone wants to have children, and there are probably even people who don't care about the respect of others), but still, wouldn't this imply that AGIs *can* be created with in-built goals? Or if such behavior can only be implemented with a motivational-system AI, how does that avoid the problem of some of the wanted final motivations being impossible to define in the initial state? But beyond this technical reason, I also believe that when people start to make a serious efort to build AGI systems - i.e. when it is talked about in government budget speeches across the world - there will be questions about safety, and the safety features of the two types of AGI will be examined. I believe that at that point there will be enormous pressure to go with the system that is safer. This makes the assumption that the public will become aware of AGI being near well ahead of the time, and takes the possibility seriously. If that assumption holds, then I agree with you. Still, the general public seems to think that AGI will never be created, or at least not in hundreds of years - and many of them remember the overoptimistic promises of AI researchers in the past. If a sufficient amount of scientists thought that AGI was doable, the public might be convinced - but most scientists want to avoid making radical-sounding statements, so they won't appear as crackpots to the people reviewing their research grant applications. Combine this with the fact that the keys for developing AGI might be scattered across so many disciplines that very few people have studied them all, or that sudden breakthroughs may accelerate the research, I don't think it's a
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
On Jan 29, 2008 6:52 PM, Richard Loosemore [EMAIL PROTECTED] wrote: Okay, sorry to hit you with incomprehensible technical detail, but maybe there is a chance that my garbled version of the real picture will strike a chord. The message to take home from all of this is that: 1) There are *huge* differences between the way that a system would behave if it had a single GS, or even a group of conflicting GS modules (which is the way you interpreted my proposal, above) and the kind of MES system I just described: the difference would come from the type of influence exerted, because the vector field is operating on a completely different level than the symbl processing. 2) The effect of the MES is to bias the system, but this bias amounts to the following system imperative: [Make your goals consistent with this *massive* set of constraints] where the massive set of constraints is a set of ideas built up throughout the entire development of the system. Rephrasing that in terms of an example: if the system gets an idea that it should take a certain course of action because it seems to satisfy an immediate goal, the implications of that action will be quickly checked against a vast range o constraints, and if there is any hint of an inconsistency with teh value system, this will pull the thoughts of the AGI toward that issue, whereupon it will start to elaborate the issue in more detail and try to impose an even wider net of constraits, finally making a decision based on the broadest possible set of considerations. This takes care of all the dumb examples where people suggest that an AGI could start with the goal Increase global happiness and then finally decide that this would be accomplished by tiling the universe with smiley faces. Another way to say this: there is no such thing as a single utility function in this type of system, nor is there a small set of utility functions there is a massive-dimensional set of utility functions (as many as there are concepts or connections in the system), and this diffuse utility function is what gives the system its stability. I got the general gist of that, I think. You've previously expressed that you don't think a seriously unfriendly AGI will be likely, apparently because you assume the motivational-system AGI will be the kind that'll be constructed and not, for instance, a goal stack-driven one. Now, what makes you so certain that people will build a this kind of AGI? Even if we assume that this sort of architecture would be the most viable one, a lot seems to depend on how tight the constraints on its behavior are, and what kind they are - you say that they are a a set of ideas built up throughout the entire development of the system. The ethics and values of humans are the result of a long, long period of evolution, and our ethical system is pretty much of a mess. What makes it likely that it really will build up a set of ideas constraints that we humans would *want* it to build? Could it not just as well pick up ones that are seriously unfriendly, especially if its designers or the ones raising it are in the least bit careless? Even among humans, there exist radical philosophers whose ideas of a perfect society are repulsive to the vast majority of the populace, and a countless number of disagreements about ethics. If we humans have such disagreements - we who all share the same evolutionary origin biasing us to develop our moral systems in a certain direction - what makes it plausible to assume that the first AGIs put together (probably while our understanding of our own workings is still incomplete) will develop a morality we'll like? -- http://www.saunalahti.fi/~tspro1/ | http://xuenay.livejournal.com/ Organizations worth your time: http://www.singinst.org/ | http://www.crnano.org/ | http://lifeboat.com/ - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244id_secret=91461196-a87c48
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Kaj Sotala wrote: On Jan 29, 2008 6:52 PM, Richard Loosemore [EMAIL PROTECTED] wrote: Okay, sorry to hit you with incomprehensible technical detail, but maybe there is a chance that my garbled version of the real picture will strike a chord. The message to take home from all of this is that: 1) There are *huge* differences between the way that a system would behave if it had a single GS, or even a group of conflicting GS modules (which is the way you interpreted my proposal, above) and the kind of MES system I just described: the difference would come from the type of influence exerted, because the vector field is operating on a completely different level than the symbl processing. 2) The effect of the MES is to bias the system, but this bias amounts to the following system imperative: [Make your goals consistent with this *massive* set of constraints] where the massive set of constraints is a set of ideas built up throughout the entire development of the system. Rephrasing that in terms of an example: if the system gets an idea that it should take a certain course of action because it seems to satisfy an immediate goal, the implications of that action will be quickly checked against a vast range o constraints, and if there is any hint of an inconsistency with teh value system, this will pull the thoughts of the AGI toward that issue, whereupon it will start to elaborate the issue in more detail and try to impose an even wider net of constraits, finally making a decision based on the broadest possible set of considerations. This takes care of all the dumb examples where people suggest that an AGI could start with the goal Increase global happiness and then finally decide that this would be accomplished by tiling the universe with smiley faces. Another way to say this: there is no such thing as a single utility function in this type of system, nor is there a small set of utility functions there is a massive-dimensional set of utility functions (as many as there are concepts or connections in the system), and this diffuse utility function is what gives the system its stability. I got the general gist of that, I think. You've previously expressed that you don't think a seriously unfriendly AGI will be likely, apparently because you assume the motivational-system AGI will be the kind that'll be constructed and not, for instance, a goal stack-driven one. Now, what makes you so certain that people will build a this kind of AGI? Even if we assume that this sort of architecture would be the most viable one, a lot seems to depend on how tight the constraints on its behavior are, and what kind they are - you say that they are a a set of ideas built up throughout the entire development of the system. The ethics and values of humans are the result of a long, long period of evolution, and our ethical system is pretty much of a mess. What makes it likely that it really will build up a set of ideas constraints that we humans would *want* it to build? Could it not just as well pick up ones that are seriously unfriendly, especially if its designers or the ones raising it are in the least bit careless? Even among humans, there exist radical philosophers whose ideas of a perfect society are repulsive to the vast majority of the populace, and a countless number of disagreements about ethics. If we humans have such disagreements - we who all share the same evolutionary origin biasing us to develop our moral systems in a certain direction - what makes it plausible to assume that the first AGIs put together (probably while our understanding of our own workings is still incomplete) will develop a morality we'll like? Perhaps we make too much of the idea of moral and ethical. As noted, this leads to endless debate. The alternative is to use law even though it may be arbitrary and haphazard in formulation. The importance of law is that it establishes risk. As humans we understand risk. Will an AI understand risk? Or, should we rephrase this to read will there be a risk for an AI? examples of what an AI might risk... 1. banishment - not allowed to run. No loading into hardware 2. isolation - prevention of access to published material or experimentation 3. imprisonment - similar to isolation, with more access than isolation 4. close supervision - imposing control through close supervision, constant oversight, actions subject to approval... 5. economic sanction - not allowed to negotiate any deals or take control of resources. I expect Matt Mahoney to point out that resistance is futile, the AI's will outsmart us. Does that mean that criminals will ultimately be smarter than non-criminals? Maybe the AI's of the future will want an even playing field and be motivated to enforce laws. I see Richards design as easily being able to implement risk factors that could lead to intelligent and legal behavior. I'm impressed by the design. Thanks for the explanation. Stan Nilsen
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Kaj Sotala wrote: On Jan 29, 2008 6:52 PM, Richard Loosemore [EMAIL PROTECTED] wrote: Okay, sorry to hit you with incomprehensible technical detail, but maybe there is a chance that my garbled version of the real picture will strike a chord. The message to take home from all of this is that: 1) There are *huge* differences between the way that a system would behave if it had a single GS, or even a group of conflicting GS modules (which is the way you interpreted my proposal, above) and the kind of MES system I just described: the difference would come from the type of influence exerted, because the vector field is operating on a completely different level than the symbl processing. 2) The effect of the MES is to bias the system, but this bias amounts to the following system imperative: [Make your goals consistent with this *massive* set of constraints] where the massive set of constraints is a set of ideas built up throughout the entire development of the system. Rephrasing that in terms of an example: if the system gets an idea that it should take a certain course of action because it seems to satisfy an immediate goal, the implications of that action will be quickly checked against a vast range o constraints, and if there is any hint of an inconsistency with teh value system, this will pull the thoughts of the AGI toward that issue, whereupon it will start to elaborate the issue in more detail and try to impose an even wider net of constraits, finally making a decision based on the broadest possible set of considerations. This takes care of all the dumb examples where people suggest that an AGI could start with the goal Increase global happiness and then finally decide that this would be accomplished by tiling the universe with smiley faces. Another way to say this: there is no such thing as a single utility function in this type of system, nor is there a small set of utility functions there is a massive-dimensional set of utility functions (as many as there are concepts or connections in the system), and this diffuse utility function is what gives the system its stability. I got the general gist of that, I think. You've previously expressed that you don't think a seriously unfriendly AGI will be likely, apparently because you assume the motivational-system AGI will be the kind that'll be constructed and not, for instance, a goal stack-driven one. Now, what makes you so certain that people will build a this kind of AGI? Kaj, [This is just a preliminary answer: I am composing a full essay now, which will appear in my blog. This is such a complex debate that it needs to be unpacked in a lot more detail than is possible here. Richard]. The answer is a mixture of factors. The most important reason that I think this type will win out over a goal-stack system is that I really think the latter cannot be made to work in a form that allows substantial learning. A goal-stack control system relies on a two-step process: build your stack using goals that are represented in some kind of propositonal form, and then (when you are ready to pursue a goal) *interpret* the meaning of the proposition on the top of the stack so you can start breaking it up into subgoals. The problem with this two-step process is that the interpretation of each goal is only easy when you are down at the lower levels of the stack - Pick up the red block is easy to interpret, but Make humans happy is a profoundly abstract statement that has a million different interpretations. This is one reason why nobody has build an AGI. To make a completely autonomous system that can do such things as learn by engaging in exploratory behavior, you have to be able insert goals like Do some playing, and there is no clear way to break that statement down into unambiguous subgoals. The result is that if you really did try to build an AGI with a goal like that, the actual behavior of the system would be wildly unpredictable, and probably not good for the system itself. Further: if the system is to acquire its own knowledge independently from a child-like state (something that, for separate reasons, I think is going to be another prerequisite for true AGI), then the child system cannot possibly have goals built into it that contain statements like Engage in an empathic relationship with your parents because it does not have the knowledge base built up yet, and cannot understand such a propositions! These technical reasons seem to imply that the first AGI that is successful will, in fact, have a motivational-emotional system. Anyone else trying to build a goal-stack system will simply never get there. But beyond this technical reason, I also believe that when people start to make a serious efort to build AGI systems - i.e. when it is talked about in government budget speeches across the world - there will be questions about safety, and the safety features of the two types of AGI will be examined. I believe that at
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
On 1/29/08, Richard Loosemore [EMAIL PROTECTED] wrote: Summary of the difference: 1) I am not even convinced that an AI driven by a GS will ever actually become generally intelligent, because of the self-contrdictions built into the idea of a goal stack. I am fairly sure that whenever anyone tries to scale one of those things up to a real AGI (something that has never been done, not by a long way) the AGI will become so unstable that it will be an idiot. 2) A motivation-system AGI would have a completely different set of properties, and among those properties would be extreme stability. It would be possible to ensure that the thing stayed locked on to a goal set that was human-empathic, and which would stay that way. Omohundros's analysis is all predicated on the Goal Stack approach, so my response is that nothing he says has any relevance to the type of AGI that I talk about (which, as I say, is probably going to be the only type ever created). Hmm. I'm not sure of exact definition that you're using of the term motivational AGI, so let me wager a guess based on what I remember reading from you before - do you mean something along the lines of a system built out of several subsystems, each with partially conflicting desires, that are constantly competing for control and exerting various kinds of pull to the behavior of the system as a whole? And you contrast this with a goal stack AGI, which would only have one or a couple of such systems? While this is certainly a major difference on the architectural level, I'm not entirely convinced how large of a difference it makes in behavioral terms, at least in this context. In order to accomplish anything, the motivational AGI would still have to formulate goals and long-term plans. Once it managed to hammer out acceptable goals that the majority of its subsystems agreed on, it would set out on developing ways to fulfill those goals as effectively as possible, making it subject to the pressures outlined in Omohundro's paper. The utility function that it would model for itself would be considerably more complex than for an AGI with less subsystems, as it would have to be a compromise between the desires of each subsystem in power, and if the balance of power would be upset too radically, the modeled utility function may even be changed entirely (like the way different moods in humans give control to different networks, altering the current desires and effective utility functions). However, AGI designers likely wouldn't make the balance of power between the different subsystems /too/ unstable, as an agent that constantly changed its mind about what it wanted would just go around in circles. So it sounds plausible that the utility function it generated would remain relatively stable, and the motivational AGI's behavior optimized just as Omohundro analysis suggests. -- http://www.saunalahti.fi/~tspro1/ | http://xuenay.livejournal.com/ Organizations worth your time: http://www.singinst.org/ | http://www.crnano.org/ | http://lifeboat.com/ - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244id_secret=91075649-b77bad
Re: [agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Kaj Sotala wrote: On 1/29/08, Richard Loosemore [EMAIL PROTECTED] wrote: Summary of the difference: 1) I am not even convinced that an AI driven by a GS will ever actually become generally intelligent, because of the self-contrdictions built into the idea of a goal stack. I am fairly sure that whenever anyone tries to scale one of those things up to a real AGI (something that has never been done, not by a long way) the AGI will become so unstable that it will be an idiot. 2) A motivation-system AGI would have a completely different set of properties, and among those properties would be extreme stability. It would be possible to ensure that the thing stayed locked on to a goal set that was human-empathic, and which would stay that way. Omohundros's analysis is all predicated on the Goal Stack approach, so my response is that nothing he says has any relevance to the type of AGI that I talk about (which, as I say, is probably going to be the only type ever created). Hmm. I'm not sure of exact definition that you're using of the term motivational AGI, so let me wager a guess based on what I remember reading from you before - do you mean something along the lines of a system built out of several subsystems, each with partially conflicting desires, that are constantly competing for control and exerting various kinds of pull to the behavior of the system as a whole? And you contrast this with a goal stack AGI, which would only have one or a couple of such systems? While this is certainly a major difference on the architectural level, I'm not entirely convinced how large of a difference it makes in behavioral terms, at least in this context. In order to accomplish anything, the motivational AGI would still have to formulate goals and long-term plans. Once it managed to hammer out acceptable goals that the majority of its subsystems agreed on, it would set out on developing ways to fulfill those goals as effectively as possible, making it subject to the pressures outlined in Omohundro's paper. The utility function that it would model for itself would be considerably more complex than for an AGI with less subsystems, as it would have to be a compromise between the desires of each subsystem in power, and if the balance of power would be upset too radically, the modeled utility function may even be changed entirely (like the way different moods in humans give control to different networks, altering the current desires and effective utility functions). However, AGI designers likely wouldn't make the balance of power between the different subsystems /too/ unstable, as an agent that constantly changed its mind about what it wanted would just go around in circles. So it sounds plausible that the utility function it generated would remain relatively stable, and the motivational AGI's behavior optimized just as Omohundro analysis suggests. I obviously need to get more detail of this idea down in a published paper, but in the mean time let me try to give a quick feel for what I mean. AGIs need *two* types of control system. One of these certainly does look like a conventional goal stack: this is the planner that handles everyday priorities and missions. There would probably be some strong differences from a conventional GS, but these we can talk about another time. For brevity, I'll just call this the GS. The second component would be the Motivational-Emotional System (MES). To understand the difference between this and the GS, try to imagine something like a Boltzmann Machine or a backpropagation neural net that has been designed in such a way as to implement a kind-of regular symbolic AI system. This would be a horrible, ugly hybrid (as you can imagine), but you could probably see how it could be done in principle, and it serves as a good way for me to get my point across. Now imagine that the neural net level of this system was built with some modulating parameters that biassed the functioning of the neurons, but where these parameters are not global, but rather are local to the neurons (so we could talk about a vector field of these parameters across the net). The purpose of these parameters is to bias the behavior of the neurons, so that if one parameter goes high in one area of the net, the firing of those neurons is elevated, and the functioning of the system is somehow enhanced in that area. What *exactly* is the effect of the vector field on the behavior of the system? Well, that is not easy to say, because the field has a diffuse effect at the symbol level - there is no one-to-one correspondence between particular symbols and he field values. Instead what you get is a soft change in the functioning of the system. Without getting into details, I am sure you can see how, in general, such a thing could be possible. Now one more idea: the field itself is locally connected, so it is almost as if there is a complete parallel universe of neurons lying
[agi] Goal Driven Systems and AI Dangers [WAS Re: Singularity Outcomes...]
Kaj Sotala wrote: On 1/24/08, Richard Loosemore [EMAIL PROTECTED] wrote: Theoretically yes, but behind my comment was a deeper analysis (which I have posted before, I think) according to which it will actually be very difficult for a negative-outcome singularity to occur. I was really trying to make the point that a statement like The singularity WILL end the human race is completely ridiculous. There is no WILL about it. Richard, I'd be curious to hear your opinion of Omohundro's The Basic AI Drives paper at http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf (apparently, a longer and more technical version of the same can be found at http://selfawaresystems.files.wordpress.com/2008/01/nature_of_self_improving_ai.pdf , but I haven't read it yet). I found the arguments made relatively convincing, and to me, they implied that we do indeed have to be /very/ careful not to build an AI which might end up destroying humanity. (I'd thought that was the case before, but reading the paper only reinforced my view...) Kaj, I have only had time to look at it briefly this evening, but it looks like Omohundro is talking about Goal Stack systems. I made a distinction, once before, between Standard-AI Goal Stack systems and another type that had a diffuse motivation system. Summary of the difference: 1) I am not even convinced that an AI driven by a GS will ever actually become generally intelligent, because of the self-contrdictions built into the idea of a goal stack. I am fairly sure that whenever anyone tries to scale one of those things up to a real AGI (something that has never been done, not by a long way) the AGI will become so unstable that it will be an idiot. 2) A motivation-system AGI would have a completely different set of properties, and among those properties would be extreme stability. It would be possible to ensure that the thing stayed locked on to a goal set that was human-empathic, and which would stay that way. Omohundros's analysis is all predicated on the Goal Stack approach, so my response is that nothing he says has any relevance to the type of AGI that I talk about (which, as I say, is probably going to be the only type ever created). I will try to go into this in more depth as soon as I get a chance. Richard Loosemore - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244id_secret=90892197-f7fae5
