Wei Dai wrote: > This seems to be a non-sequitor. The weakness of AIXI is not that it's > goals don't change, but that it has no goals other than to maximize an > externally given reward. So it's going to do whatever it predicts will > most efficiently produce that reward, which is to coerce or subvert > the evaluator. If you start with such a goal, I don't see how allowing > the system to change its goals is going to help. > > But I think Eliezer's real point, which I'm not sure has come across, is > that if you didn't spot such an obvious flaw right away, maybe you > shouldn't trust your intuitions about what is safe and what is not.
The real flaw in the AIXI discussion was Eliezer's statement: > Lee Corbin can work out his entire policy in step (2), before step > (3) occurs, knowing that his synchronized other self - whichever one > he is - is doing the same. He was assuming that a human could know that another mind would behave identically. Of course they cannot, but can only estimate other mind's intentions based on observations. Eliezer backed off from this, and the discussion was reduced to whether humans or AIXI-tls are better at estimating intentions from behaviors. It was Eliezer who failed to spot the obvious flaw. I also want to comment on your substantive point about "the subject exploiting vulnerabilities in the evaluation algorithm to obtain rewards without actually acomplishing any real objectives. You can see an example of this problem in drug abusers" from your post at: http://www.mail-archive.com/everything-list@eskimo.com/msg03620.html This is why a solution to the credit assignment problem is so important for reinforcement learning, to account for long term rewards as well as short term rewards. Drug abusers seek reward in the short term, but that is far outweighed by their long term losses. Bill ---------------------------------------------------------- Bill Hibbard, SSEC, 1225 W. Dayton St., Madison, WI 53706 [EMAIL PROTECTED] 608-263-4427 fax: 608-263-6738 http://www.ssec.wisc.edu/~billh/vis.html ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
