> This is exactly why I keep trying to emphasize that we all should forsake > those endlessly fascinating, instinctively attractive political arguments > over our favorite moralities, and instead focus on the much > harder problem > of defining an AI architecture which can understand that its morality is > "wrong" in various ways; wrong definitions, wrong reinforcement > procedures, wrong source code, wrong Friendliness architecture, wrong > definition of "wrongness", and many others. These are nontrivial > problems! Each turns out to require nonobvious structural qualities in > the architecture of the goal system.
Hmmm. It seems to me the ability to recognize one's own potential wrongness comes along automatically with general intelligence... Recognizing "wrong source code" requires a codic modality, of course, and recognizing "wrong Friendliness architecture" requires an intellectual knowledge of philosophy and software design. What is there about recognizing one's wrongness in the ways you mention, that doesn't come "for free" with general cognition and appropriate perception? I guess there is an attitude needed to recognize one's own wrongness: a lack of egoistic self-defensive certainty in one's own correctness.... A skeptical attitude even about one's own most deeply-held beliefs. In Novamente, this skeptical attitude has two aspects: 1) very high level schemata that must be taught not programmed 2) some basic parameter settings that will statistically tend to incline the system toward skepticism of its own conclusions [but you can't turn the dial too far in the skeptical direction either...] -- Ben G ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
