Ben Goertzel wrote:
Ben, I've been there, 1996-2000, and that turned out to be the WRONG ANSWER. There's an enormous amount of moral complexity that does *not* come along with asymptotically increasing intelligence. Thankfully, despite the tremendous emotional energy I put into believing that superintelligences are inevitably moral, and despite the amount of published reasoning I had staked on it, I managed to spot this mistake before I "pulled a Lawrence" on the human species. Please, please, please don't continue where I left off.This is exactly why I keep trying to emphasize that we all should forsake those endlessly fascinating, instinctively attractive political arguments over our favorite moralities, and instead focus on the much harder problem of defining an AI architecture which can understand that its morality is "wrong" in various ways; wrong definitions, wrong reinforcement procedures, wrong source code, wrong Friendliness architecture, wrong definition of "wrongness", and many others. These are nontrivial problems! Each turns out to require nonobvious structural qualities in the architecture of the goal system.Hmmm. It seems to me the ability to recognize one's own potential wrongness comes along automatically with general intelligence...
The problem here is the imprecision of words. *One* form of wrongness, such as factual error, or "wrong source code" which is "wrong" because it is inefficient or introduces factual errors, is readily conceivable by a general intelligence without extra moral complexity. You do, indeed, get recognition of *that particular* kind of "wrongness" for free. It does not follow that all the things we recognize as wrong, in moral domains especially, can be recognized by a general intelligence without extra moral complexity.
If it is the case that a general intelligence necessarily has the ability to conceive of a "wrongness" in a top-level goal definition and has a mechanism for correcting it, this is not obvious to me - not for any definition of "wrongness" at all. Prime Intellect, with its total inability to ask any moral question except "how desirable is X, under the Three Laws as presently defined", seems to me quite realistic.
Note also that the ability to identify *a* kind of wrongness, does not necessarily mean the ability to see - as a human would - the specific wrongness of your own programmer standing by and screaming "That's not what I meant! Stop! Stop!" If this realization is a necessary ability of all minds-in-general it is certainly not clear why.
So... you think a real-life Prime Intellect would have, for free, recognized that it should not lock Lawrence out? But why?Recognizing "wrong source code" requires a codic modality, of course, and recognizing "wrong Friendliness architecture" requires an intellectual knowledge of philosophy and software design.What is there about recognizing one's wrongness in the ways you mention, that doesn't come "for free" with general cognition and appropriate perception?
That's for skepticism about facts. I agree you get that for free with general intelligence. If *all* questions of morality, means and ends and ultimate goals, were reducible to facts and deducible by logic or observation, then the issue would end right there. That was my position 1996-2000. Is this your current position?I guess there is an attitude needed to recognize one's own wrongness: a lack of egoistic self-defensive certainty in one's own correctness.... A skeptical attitude even about one's own most deeply-held beliefs. In Novamente, this skeptical attitude has two aspects: 1) very high level schemata that must be taught not programmed 2) some basic parameter settings that will statistically tend to incline the system toward skepticism of its own conclusions [but you can't turn the dial too far in the skeptical direction either...]
--
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
-------
To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
