[agi] The Metamorphosis of Prime Intellect
I've just read the first chapter of The Metamorphosis of Prime Intellect. http://www.kuro5hin.org/prime-intellect It makes you realise that Ben's notion that ethical structures should be based on a hierarchy going from general to specific is very valid - if Prime Intellect had been programmed to respect all *life* and not just humans then the 490 worlds with sentient life not to mention the 14,623 worlds with life of some type might have been spared. It also makes it clear that when we talk about building AGIs for 'human friendliness' we are using language that does not follow Ben's recommended ethical goal structure. I'm wondering (seriously) whether the AGI movement needs to change it short hand language (human friendly) in this case - in other arenas people talk about the need for ethical behaviour. Would that term suffice? Cheers, Philip --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] The Metamorphosis of Prime Intellect
Eliezer S. Yudkowsky wrote: There may be additional rationalization mechanisms I haven't identified yet which are needed to explain anosognosia and similar disorders. Mechanism (4) is the only one deep enough to explain why, for example, the left hemisphere automatically and unconsciously rationalizes the actions of the left hemisphere; and mechanism (4) doesn't necessarily explain that, it only looks like it might someday do so. That is, the left hemisphere automatically and unconsciously rationalizes the actions of the right hemisphere in split-brain patients. Sorry. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] The Metamorphosis of Prime Intellect
Ben Goertzel wrote: This is exactly why I keep trying to emphasize that we all should forsake those endlessly fascinating, instinctively attractive political arguments over our favorite moralities, and instead focus on the much harder problem of defining an AI architecture which can understand that its morality is wrong in various ways; wrong definitions, wrong reinforcement procedures, wrong source code, wrong Friendliness architecture, wrong definition of wrongness, and many others. These are nontrivial problems! Each turns out to require nonobvious structural qualities in the architecture of the goal system. Hmmm. It seems to me the ability to recognize one's own potential wrongness comes along automatically with general intelligence... Ben, I've been there, 1996-2000, and that turned out to be the WRONG ANSWER. There's an enormous amount of moral complexity that does *not* come along with asymptotically increasing intelligence. Thankfully, despite the tremendous emotional energy I put into believing that superintelligences are inevitably moral, and despite the amount of published reasoning I had staked on it, I managed to spot this mistake before I pulled a Lawrence on the human species. Please, please, please don't continue where I left off. The problem here is the imprecision of words. *One* form of wrongness, such as factual error, or wrong source code which is wrong because it is inefficient or introduces factual errors, is readily conceivable by a general intelligence without extra moral complexity. You do, indeed, get recognition of *that particular* kind of wrongness for free. It does not follow that all the things we recognize as wrong, in moral domains especially, can be recognized by a general intelligence without extra moral complexity. If it is the case that a general intelligence necessarily has the ability to conceive of a wrongness in a top-level goal definition and has a mechanism for correcting it, this is not obvious to me - not for any definition of wrongness at all. Prime Intellect, with its total inability to ask any moral question except how desirable is X, under the Three Laws as presently defined, seems to me quite realistic. Note also that the ability to identify *a* kind of wrongness, does not necessarily mean the ability to see - as a human would - the specific wrongness of your own programmer standing by and screaming That's not what I meant! Stop! Stop! If this realization is a necessary ability of all minds-in-general it is certainly not clear why. Recognizing wrong source code requires a codic modality, of course, and recognizing wrong Friendliness architecture requires an intellectual knowledge of philosophy and software design. What is there about recognizing one's wrongness in the ways you mention, that doesn't come for free with general cognition and appropriate perception? So... you think a real-life Prime Intellect would have, for free, recognized that it should not lock Lawrence out? But why? I guess there is an attitude needed to recognize one's own wrongness: a lack of egoistic self-defensive certainty in one's own correctness A skeptical attitude even about one's own most deeply-held beliefs. In Novamente, this skeptical attitude has two aspects: 1) very high level schemata that must be taught not programmed 2) some basic parameter settings that will statistically tend to incline the system toward skepticism of its own conclusions [but you can't turn the dial too far in the skeptical direction either...] That's for skepticism about facts. I agree you get that for free with general intelligence. If *all* questions of morality, means and ends and ultimate goals, were reducible to facts and deducible by logic or observation, then the issue would end right there. That was my position 1996-2000. Is this your current position? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
RE: [agi] The Metamorphosis of Prime Intellect
In Novamente, this skeptical attitude has two aspects: 1) very high level schemata that must be taught not programmed 2) some basic parameter settings that will statistically tend to incline the system toward skepticism of its own conclusions [but you can't turn the dial too far in the skeptical direction either...] That's for skepticism about facts. I agree you get that for free with general intelligence. If *all* questions of morality, means and ends and ultimate goals, were reducible to facts and deducible by logic or observation, then the issue would end right there. That was my position 1996-2000. Is this your current position? Not exactly, no... that is not my current position. For example: Of course, there is no logical way to deduce that killing chimpanzees is morally worse than killing fleas, from no assumptions. If one assumes that killing humans is morally bad, then from this assumption, reasoning (probabilistic analogical reasoning, for instance) leads one to conclude that killing chimpanzees is morally worse than killing fleas... My view is fairly subtle, and has progressed a bit since I wrote Thoughts on AI Morality. I don't have time to write a long e-mail on it today, but I will do so tomorrow. I think that will be better than writing something hasty and hard-to-understand right now. Ben --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
RE: [agi] The Metamorphosis of Prime Intellect
Eliezer S. Yudkowsky wrote: I don't think we are the beneficiaries of massive evolutionary debugging. I think we are the victims of massive evolutionary warpage to win arguments in adaptive political contexts. I've identified at least four separate mechanisms of rationalization in human psychology so far: Well, yes. Human minds are tuned for fitness in the ancestral environment, not for correspondence with objective reality. But just getting to the point where implementing those rationalizations is possible would be a huge leap forward from current AI systems. In any case, I think your approach to the problem is a step in the right direction. We need a theory of AI ethics before we can test it, and we need lots of experimental testing before we start building things that have any chance of taking off. Sometimes I think it is a good thing that AI is still stuck in a mire of wishful thinking, because we aren't ready to build AGI safely. Billy Brown --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
