Billy Brown wrote:
I don't think we are the beneficiaries of massive evolutionary debugging. I think we are the victims of massive evolutionary warpage to win arguments in adaptive political contexts. I've identified at least four separate mechanisms of rationalization in human psychology so far:I think it is worth reflecting here on the fact that many (perhaps most) adult human beings routinely fail to recognize errors in their own thinking, even when the mistake is pointed out. It is also quite common for humans to become locked into self-reinforcing belief systems that have little or no relation to anything real, and this state often lasts a lifetime.If humans, who have the benefit of massive evolutionary debugging, are so prone to meta-level errors, it seems unwise to assume that intelligence alone will automatically solve the problem. At a minimum, we should look for a coherent theory as to why humans make these kinds of mistakes, but the AI is unlikely to do so.
1) Deliberate rationalization by people who do not realize, or do not choose, that rationalization is wrong.
2) Instinctive, unconscious rationalization in political contexts.
3) Emergent rationalization as a product of the human reinforcement architecture (we flinch away from unpleasant thoughts); this emergent phenomenon of our goal architecture may have been evolutionarily fixed as a mechanism leading to adaptive rationalizations.
4) Inertial rationalization as a product of the human pattern-completion mechanism for extending world-models; we have mechanisms which looks selectively for data consistent with what we already know, without an equal and opposing search for inconsistent data, or better yet a fully general search for relevant data, and without continuing fine-grained readjustment of probabilities using a Bayesian support model. Essentially, once we're in the flood of an argument, whether political or not, we tend to continue onward, inertially, without continually re-evaluating the conclusion.
There may be additional rationalization mechanisms I haven't identified yet which are needed to explain anosognosia and similar disorders. Mechanism (4) is the only one deep enough to explain why, for example, the left hemisphere automatically and unconsciously rationalizes the actions of the left hemisphere; and mechanism (4) doesn't necessarily explain that, it only looks like it might someday do so.
However, mechanism (4) is also the only mechanism that would, even in theory, be likely to apply to a nonevolved AI; and it should be easy to avoid this by instituting searches for *relevant* evidence rather than *supporting* evidence and by continually adjusting Bayesian support on the basis of *all* evidence found, as opposed to the human mind's search for *consistent* evidence and simple is/is-not instead of continuous updating of fine probabilities.
Any form of purely intellectual rationalization, such as (4), would probably be spotted and corrected by a seed AI renormalizing its own source code - there is no extra moral component needed to see the desirability of this.
--
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
-------
To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
