William and Vladimir, IMHO this discussion is based entirely on the absence of any sort of interface spec. Such a spec is absolutely necessary for a large AGI project to ever succeed, and such a spec could (hopefully) be wrung out to at least avoid the worst of the potential traps.
For example: Suppose that new tasks stated the maximum CPU resources needed to complete. Then, exceeding that would be cause for abnormal termination. Of course, this doesn't cover logical failure. More advanced example: Suppose that tasks provided a "chain of consciousness" log as they execute, and a monitor watches that chain of consciousness to see that new entries are repeatedly made, that they are "grammatically" (machine grammar) correct, and verifies anything that is easily verifiable. Even more advanced example: Suppose that a new pseudo-machine were proposed, whose fundamental "code" consisted of reasonable operations in the logic-domain being exploited by the AGI. The interpreter for this pseudo-machine could then employ countless internal checks as it operated, and quickly determine when things went wrong. Does anyone out there have something, anything in the way of an interface spec to really start this discussion? Steve Richfield =============== On 7/3/08, William Pearson <[EMAIL PROTECTED]> wrote: > > Sorry about the long thread jack > > 2008/7/3 Vladimir Nesov <[EMAIL PROTECTED]>: > > On Thu, Jul 3, 2008 at 4:05 PM, William Pearson <[EMAIL PROTECTED]> > wrote: > >> Because it is dealing with powerful stuff, when it gets it wrong it > >> goes wrong powerfully. You could lock the experimental code away in a > >> sand box inside A, but then it would be a separate program just one > >> inside A, but it might not be able to interact with programs in a way > >> that it can do its job. > >> > >> There are two grades of faultiness. frequency and severity. You cannot > >> predict the severity of faults of arbitrary programs (and accepting > >> arbitrary programs from the outside world is something I want the > >> system to be able to do, after vetting etc). > >> > > > > You can't prove any interesting thing about an arbitrary program. It > > can behave like a Friendly AI before February 25, 2317, and like a > > Giant Cheesecake AI after that. > > > Whoever said you could? The whole system is designed around the > ability to take in or create arbitrary code, give it only minimal > access to other programs that it can earn and lock it out from that > ability when it does something bad. > > By arbitrary code I don't mean random, I mean stuff that has not > formally been proven to have the properties you want. Formal proof is > too high a burden to place on things that you want to win. You might > not have the right axioms to prove the changes you want are right. > > Instead you can see the internals of the system as a form of > continuous experiments. B is always testing a property of A or A', if > at any time it stops having the property that B looks for then B flags > it as buggy. > > I know this doesn't have the properties you would look for in a > friendly AI set to dominate the world. But I think it is similar to > the way humans work, and will be as chaotic and hard to grok as our > neural structure. So as likely as humans are to explode intelligently. > > Will Pearson > > > ------------------------------------------- > agi > Archives: http://www.listbox.com/member/archive/303/=now > RSS Feed: http://www.listbox.com/member/archive/rss/303/ > Modify Your Subscription: > http://www.listbox.com/member/?&; > Powered by Listbox: http://www.listbox.com > ------------------------------------------- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244&id_secret=106510220-47b225 Powered by Listbox: http://www.listbox.com
