> > The harmfulness or benevolence of an AIXI system is therefore > closely tied > > to the definition of the goal that is given to the system in advance. > > Actually, Ben, AIXI and AIXI-tl are both formal systems; there is no > internal component in that formal system corresponding to a "goal > definition", only an algorithm that humans use to determine when and how > hard they will press the reward button. > > -- > Eliezer S. Yudkowsky
Well, the definitions of AIXI and AIXItl assume the existence of a "reward function" or "goal function" (denoted V in the paper). The assumption of the math is that this reward function is specified up-front, before AIXI/AIXItl starts running. If the reward function is allowed to change adaptively, based on the behavior of the AIXI/AIXItl algorithm, then the theorems don't work anymore, and you have a different sort of "synergetic" system such as Bill Hibbard was describing. If human feedback IS the reward function, then you have a case where the reward function may well change adaptively based on the AI system's behavior. Whether the system will ever achieve any intelligence at all then depends on how clever the humans are in doing the rewarding... as i said, Hutter's theorems about intelligence don't apply... -- Ben ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
