Ben,
Ben: That paragraph gave one possible dynamic in a society of AGI's, but there are many many other possible social dynamics
Of course. What you say is quite true. But so what?
Let's go back to that one possible dynamic. Can't you bring yourself to agree that if a one-and-only super-AGI went feral that humans would then be at a greater disadvantage relative to it than if there was more than one AGI around and the humans could call on the help of one or more of the other AGIs??
Forget about all the other possible hypotheticals. Is my assessment of the specific scenario above about right or not - doesn't it have */some /*element of common sense about it?
Common sense is just not reliable for cases like this.
Suppose that you have N cognitive processes, labeled 1..N, generated by a moderate divergence of design methods and variables. The cognitive processes have a shared chance P of all undergoing simultaneous moral failure due to a common cause. They also have various individual factors A, B, C, for undergoing moral failure independently - for an individual cognitive process the factors would be A1, B1, C1, etc.
Okay, now for analysis:
First, the entire argument for having a society stems from the idea that the intelligent designers of the cognitive processes cannot simultaneously minimize all A, B, C, or that there is expectable divergence among unknown risk factors for the set of design variables being manipulated. That is, either you have a known tradeoff between A and B, or you expect that there are some risk factors C that you don't understand well enough to minimize, but which will have variant properties depending on the set of design differences among the N cognitive processes.
Already we are dealing with a far lesser design motivation for multiplicity than your human intuitions ("common sense") would lead you to expect. Humans are not intelligently designed and there is no way to redesign them. Humans are designed to pay attention to independent factors among humans, since it is these variations that are reproductively relevant and hence perceptually salient. (Note that this translated into both you and Ben Goertzel making the assumption that the chance P was independent among all AIs.) There is currently no possible way to optimize individual variables among humans - no thinking about the kind of design tradeoff between A, B, C because in fact you have no control whatsoever over A, B, C; you're stuck with the factory settings on whatever humans you're dealing with.
Intelligent design, by simultaneously solving for A, B, C, will tend to reduce the amount of *independent* variance among these factors and hence place almost all of the controllable risk factor into P. I find myself in very serious doubt that a pragmatic morality designer will be faced with known tradeoffs between independent variables that are of significant size compared to the common risk P. For one thing, the Friendly AI theory I've been working on calls for the straightforward, even "common-sensical" precaution of computing it *both ways, dammit* and turning any disagreement into "entropy" that kicks the problem back to the programmers. You might consider a Friendly AI as already computing the divergence between A, B, C, and accounting it into a more sophisticated entropy-handling process than "a group of individuals fighting it out". (For one thing, how do you know that the individual who disagrees with the group is the one failure rather than the one success?)
Really, a group of cognitive processes fighting it out is a very unsophisticated and human way of handling disagreements. Again, I emphasize that this is the *only humanly available* way of building cognitive processes larger than a single human, but actually a *very poor* way with respect to minds-in-general. Friendly AI theory requires seeing on a level lower than either "singleton" or "multiplicity"; there are only physically implemented moral processes.
A far more serious issue is that for recursively self-improving AIs where the speed of ascent is itself tied into Friendliness - where the rate of the AI's self-improvement is limited by the programmers' and AI's estimate of the AI's current moral ability - then catastrophic failure of Friendliness in an AI that's progressed far enough not to be humanly restrainable (i.e., far enough that you'd need an AI to help you defeat it) may be linked to that AI improving itself far beyond the AIs you thought would help defeat it...
...and even if the other AIs notice this in time to recursively self-improve themselves in time to match the rogue, you may STILL end up losing because the OTHER AIs weren't mature enough to handle that speed of ascent!
In other words, for recursively self-improving systems, multiplying tradeable variances in the independent causes of failure into different cognitive processes may actually be a VERY BAD IDEA because it greatly increases the chance of at least one catastrophic error occurring, while not increasing by much the chance of recovering from the catastrophic error.
If there is */any /*benefit in having more than one AGI around in the case where an AGI does go feral then your comment "I'm just not so sure that there's _any _benefit to the "society of AGI" as opposed to "one big AGI" approach" no longer holds as an absolute.
Ya know, this is why (no offense, Ben) I really dislike statements like "I'm just not so sure that there's any benefit to X". It doesn't encourage analysis the way that "I think X's disadvantages outweigh its advantages" does. If you're not so sure, you might as well try anyway, or not try, or whatever. If you phrase it as a comparison, that encourages enumeration of the advantages and disadvantages.
It then gets back to ....having a society of AGIs might be an advantage in certain cercumstances, but having more than one AGI might have the following down sides. At this point a balanced risk/benefit assessment can be made (not definitive of course since we haven't seen super-intelligent AGIs operation yet). But at least we've got some relevant issues on the table to think about.
-- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence
-------
To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
