On Wed, Apr 24, 2002 at 04:51:18PM +0200, Marcus Hutter wrote: > In "A Theory of Universal Artificial Intelligence based on > Algorithmic Complexity" http://www.idsia.ch/~marcus/ai/pkcunai.htm > I developed a rational decision maker which makes optimal > decisions in any environment. The only assumption I make is that > the environment is sampled from a computable (but unknown!) > probability distribution (or in a deterministic world is > computable), which should fit nicely into the basic assumptions of > this list. Although logic plays a role in optimal resource bounded > decisions, it plays no role in the unrestricted model. > > I would be pleased to see this work discussed here.
I'm glad to see you bring it up, because I do want to discuss it. :) For people who haven't read Marcus's paper, the model consist of two computers, one representing an intelligent being, and the other one the environment, communicating with each other. The subject sends its decisions to the environment, and the environment sends information and rewards to the subject. The subject's goal is to maximize the sum of rewards over some time period. The paper then presents an algorithm that solves the subject's problem, and shows that it's close to optimal in some sense. In this model, the real goals of the subject (who presumably wants to acomplish objectives other than maximizing some abstract number) are encoded in the environment algorithm. But how can the environment algorithm be smart enough to evaluate the decisions of the subject? Unless the evaluation part of the environment algorithm is as intelligent as the subject, you'll have problems with the subject exploiting vulnerabilities in the evaluation algorithm to obtain rewards without actually acomplishing any real objectives. You can see an example of this problem in drug abusers. If we simply assume that the environment is smart enough, then we've just moved the problem around. So, how can we change the model so that the evaluation algorithm is part of the subject rather than the environment? First we have to come up with some way to formalize the real objectives of the subject. I think the formalism must be able to handle objectives that are about the internal state of the environment, rather than just the information the subject receives from the environment, otherwise we can't explain why people care about things that they'll never see, for example things that happen after they die. Then we would invent a universal decision algorithm for acomplishing any set of objectives and show that it's close to optimal. This seems very difficult because we'll have to talk about the internal state of general algorithms, which we have very little theory for.
