On 06/11/2014 05:07, Bill Hibbard via AGI wrote:
http://arxiv.org/abs/1411.1373
Thanks Bill. The document illustrates what Bill Hibbard and MIRI look like as a team. It looks like a lamentable alliance to me - but I suppose we should gratefully accept whatever Bill offers us. My first impressions: I was pleased to see an analysis of a flaw with model-based utility functions - namely that the model may not correspond to reality - and thus utility in the model might be a kind of counterfeit utility. I was also pleased to see some proposed solutions. However, I found myself highly sceptical of most of the propositions in section 7. It describes an optimizing agent with an "implicit" utility function which can only take "implicit" actions - and claims that the agent's actions can't increase the utility function and that it "doesn't make any predictions". It offers a mathematical proof that this agent will behave itself. I can't take this seriously as a security analysis of a highly-intelligent machine. For one thing, if the agent can see previous copies of itself, the first sentence of the proof unravels. Putting the word "implicit" all over the place, makes no difference: this is an intelligent agent that can act in the world. The idea of unlinking world model from goals seems like an attempt to keep humans in the loop - by having them dictate how much resources are devoted to model building, and what aspects of the world to study and model. However, of course such "keeping humans in the loop" would make the machines slower and less competitive. This could cause serious security problems - if the machines are then out-competed by other terms of humans and machines. If human supervision is considered desirable. We should consider where best to put it. I think Bill's security-driven proposal here needs to be weighed against alternatives. -- __________ |im |yler http://timtyler.org/ [email protected] Remove lock to reply. ------------------------------------------- AGI Archives: https://www.listbox.com/member/archive/303/=now RSS Feed: https://www.listbox.com/member/archive/rss/303/21088071-f452e424 Modify Your Subscription: https://www.listbox.com/member/?member_id=21088071&id_secret=21088071-58d57657 Powered by Listbox: http://www.listbox.com
