The issue of control over an AGI was discussed in the singularity mailing list. The question was whether it is possible to guarantee that an AGI will be friendly. It was hotly debated with no consensus.
My position is that once you make machines that are smarter than humans, and they do the same, you cannot guarantee anything. This limitation is fundamental, in the same way that you cannot predict if a Turing machine will halt. I cited two papers by Hutter and Legg to support this. Hutter's paper on AIXI proves that the optimal behavior of a rational agent (as a Turing machine) with the goal of maximizing the accumulated reward signal from an unknown interactive environment is to guess that the environment is simulated by the shortest Turing machine consistent with past observation. Legg's paper on the limits of learnability proves that the shortest Turing machine capable of learning to predict the output of another machine of Kolmogorov complexity n is between n and n + log n.
Taken together, the papers explain a lot about the nature of uncertainty in a deterministic universe (as Einstein asserted, in spite of quantum mechanics). Hutter's proof requires the assumption that the universe be computable by a Turing machine. I think his paper (which essentially proves Occam's Razor) would not be so compelling if the universe were not in fact computable, or a simulation. The source of uncertainty is therefore due to the universe having greater Kolmogorov complexity than your brain.
Your programming example illustrates this nicely. You can't understand a 30,000 line program all at once, so you divide it into modules with well defined interfaces. You can develop, test, debug, model, predict, etc. one small module while treating the rest of the program as unpredictable, even though you know it is really deterministic. If you didn't model the program this way, you wouldn't need to check function arguments or throw exceptions. So you are really supporting my argument that you cannot predict (and therefore cannot control) an AGI.
-- Matt Mahoney, [EMAIL PROTECTED]
My position is that once you make machines that are smarter than humans, and they do the same, you cannot guarantee anything. This limitation is fundamental, in the same way that you cannot predict if a Turing machine will halt. I cited two papers by Hutter and Legg to support this. Hutter's paper on AIXI proves that the optimal behavior of a rational agent (as a Turing machine) with the goal of maximizing the accumulated reward signal from an unknown interactive environment is to guess that the environment is simulated by the shortest Turing machine consistent with past observation. Legg's paper on the limits of learnability proves that the shortest Turing machine capable of learning to predict the output of another machine of Kolmogorov complexity n is between n and n + log n.
Taken together, the papers explain a lot about the nature of uncertainty in a deterministic universe (as Einstein asserted, in spite of quantum mechanics). Hutter's proof requires the assumption that the universe be computable by a Turing machine. I think his paper (which essentially proves Occam's Razor) would not be so compelling if the universe were not in fact computable, or a simulation. The source of uncertainty is therefore due to the universe having greater Kolmogorov complexity than your brain.
Your programming example illustrates this nicely. You can't understand a 30,000 line program all at once, so you divide it into modules with well defined interfaces. You can develop, test, debug, model, predict, etc. one small module while treating the rest of the program as unpredictable, even though you know it is really deterministic. If you didn't model the program this way, you wouldn't need to check function arguments or throw exceptions. So you are really supporting my argument that you cannot predict (and therefore cannot control) an AGI.
----- Original Message ----
From: David Clark <[EMAIL PROTECTED]>
To: [email protected]
Sent: Friday, October 6, 2006 6:03:58 PM
Subject: Re: [agi] G0 theory completed
From: David Clark <[EMAIL PROTECTED]>
To: [email protected]
Sent: Friday, October 6, 2006 6:03:58 PM
Subject: Re: [agi] G0 theory completed
Matt Mahoney said: If you can't model an AGI in your head, then you can't
program it, understand it, test it, control it, or predict what it will do
either.
Can't model can't program- I have programmed a
number of large (30,000+ lines of C) systems and I couldn't disagree more with
the above statement. At any one time, I can only think about 10 or fewer
details of any program at a time. It is only because of the
organization of my code that I am able to make a program that actually
works. As the program grows, the levels of abstraction grows so that (from
my personal experience) I see no limit to the size of complexity of code that I
can create or work on. I rely, for much of the detail memory, on the
source code and I don't try to model all the details at once at all. I
have other tools that help with higher levels of abstraction.
Understand- I "understand" all parts (in
isolation) of all programs that I write but for some of the largest ones, I
can't predict (without tools) what the system will always do without actually
running the program. If I wanted to know exactly then I would probably
just run the program and find out for sure.
Testing - There
are many methods of testing and I see no limit to the size of code that can be
tested. Microsoft Windows is the largest set of programs in the world and
it can be argued that they are not fully tested but obviously enough for most
people to use the programs.
Control- Control has special meaning when talking
about an AGI. If you truly had an AGI, would you have any more control
over it if you could totally model or understand the AGI versus
not?
Predict- We can't predict how most people will
react and think even if we have known them our whole lives. If an AGI had
intelligence on par with humans, how could we expect to always predict what they
would think or do when we can't do that with ourselves.
What limit is there to knowledge in general?
What tiny fraction of that knowledge can any single person embody? I don't
know but it must be a tiny fraction of the knowledge we currently have and our
rate of acquiring new knowledge is increasing exponentially. What limit is
there to a database given an increasingly growing memory store? I think
the answer if not infinite, then it must be many orders of magnitude larger than
what a single human is capable of. Using a human mind as an example of the
only way an AGI could possibly be created is flawed. Humans might have a
huge number of limits to learning that AGI's do not. The hardware for an
AGI is not limited to any specific amount and it is also not limited to current
hardware or algorithms. A human is limited to the brain in his skull and
only if future technologies provide new tools to augment our brains can we
strive to get where AGI's will go.
I agree that an AGI fundamentally will be created
by a combination of data (databases) and procedures (programs) but how large and
by who the programs will be created has yet to be determined. Why do you
assume that all AGI programs will be created by humans? Why couldn't an
initial set of programs be created by AGI researchers that will when complete
start creating programs on it's own? Why would you have to start out with
the minimal data and set of algorithms that humans do?
I am having a hard time seeing where such wild and
unsupported statements benefit others?
David Clark
This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED]----- Original Message -----From: Matt MahoneySent: Friday, October 06, 2006 1:21 PMSubject: Re: [agi] G0 theory completedIf you can't model an AGI in your head, then you can't program it, understand it, test it, control it, or predict what it will do either.Until then, I'm afraid researchers will still be falling into the trap of encoding knowledge explicitly for the sake of computational efficiency. There is a simple information theoretic reason why this will fail: a Turing machine (your brain) cannot simulate a machine with greater Kolmogorov complexity (more knowledge) than itself [1]. If you can't model an AGI in your head, then you can't program it, understand it, test it, control it, or predict what it will do either.
What you can do is develop a system with little initial knowledge and the ability to learn. Of course the same restrictions will apply. Whan it has acquired more knowedge than yourself, you will not be able to predict, test, model, or control it, or even know if you have succeeded. But you can still build it. For an example, see the human brain. It's complexity at its formation cannot exceed 6 x 10^9 bits, the information content of your DNA. This is much less than the 10^13 or so bits encoded in your synapses. You can create children but not control them.
I think to succeed at AGI, we need to understand the theoretical limits of learning [1,2], then develop a system not based on methods that have already been shown not to work. Then build a system that can learn, give it enough raw data to do so, and set it loose.
This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED]
