Sorry about the long thread jack 2008/7/3 Vladimir Nesov <[EMAIL PROTECTED]>: > On Thu, Jul 3, 2008 at 4:05 PM, William Pearson <[EMAIL PROTECTED]> wrote: >> Because it is dealing with powerful stuff, when it gets it wrong it >> goes wrong powerfully. You could lock the experimental code away in a >> sand box inside A, but then it would be a separate program just one >> inside A, but it might not be able to interact with programs in a way >> that it can do its job. >> >> There are two grades of faultiness. frequency and severity. You cannot >> predict the severity of faults of arbitrary programs (and accepting >> arbitrary programs from the outside world is something I want the >> system to be able to do, after vetting etc). >> > > You can't prove any interesting thing about an arbitrary program. It > can behave like a Friendly AI before February 25, 2317, and like a > Giant Cheesecake AI after that. > Whoever said you could? The whole system is designed around the ability to take in or create arbitrary code, give it only minimal access to other programs that it can earn and lock it out from that ability when it does something bad.
By arbitrary code I don't mean random, I mean stuff that has not formally been proven to have the properties you want. Formal proof is too high a burden to place on things that you want to win. You might not have the right axioms to prove the changes you want are right. Instead you can see the internals of the system as a form of continuous experiments. B is always testing a property of A or A', if at any time it stops having the property that B looks for then B flags it as buggy. I know this doesn't have the properties you would look for in a friendly AI set to dominate the world. But I think it is similar to the way humans work, and will be as chaotic and hard to grok as our neural structure. So as likely as humans are to explode intelligently. Will Pearson ------------------------------------------- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244&id_secret=106510220-47b225 Powered by Listbox: http://www.listbox.com
