A perfect optimizer who behaves according to decision theory (or some bounded-rationality version of it) is very vulnerable to small changes in its utility function definition or the module responsible for interpreting the meaning of terms in the utility function definition. Such a change, say a bit flip caused by cosmic radiation, or the introduction of a new philosophical idea, could cause the agent to behave completely counter to the designer's intentions.
In the rule-based agent, on the other hand, the utility function definition and its interpretation are effectively dispersed throughout the set of rules. If the rules are designed with appropriate redundancy, it should be much less likely for a catastrophic change in behavior to occur.
This seems to me to confuse the decision with how the decision is represented and implemented. There are presumably many ways to disperse a decision process and make it robust to random errors, and some of those ways may be compatible with pretty optimal behavior.
Robin Hanson [EMAIL PROTECTED] http://hanson.gmu.edu
Assistant Professor of Economics, George Mason University
MSN 1D3, Carow Hall, Fairfax VA 22030-4444
703-993-2326 FAX: 703-993-2323
