I suppose I'd like to see the list management weigh in on whether this type of talk belongs on this particular list or whether it is more appropriate for the "singularity" list. Assuming it's okay for now, especially if such talk has a technical focus: One thing that could improve safety is to reject the notion that AGI projects should be focused on, or even capable of, recursive self improvement in the sense of reprogramming its core implementation. Let's take Novamente as an example. Imagine that Ben G is able to take a break at some point from standing behind the counter of his Second Life pet store for a few years, and he gets his 1000-pc cluster and the implementation goes just as imagined and "baby novamente" is born some years down the road. At this point, Ben & co. begin teaching it the difference between its virtual ass and a virtual hole in the ground. Novamente's model of mind is not the same thing as the C++ code that implements it; Baby Novamente has no particular affinity for computer programming or built in knowledge about software engineering. It cannot improve itself until the following things happen: 1) It acquires the knowledge and skills to become a competent programmer, a task that takes a human many years of directed training and practical experience. 2) It is given access to its own implementation and permission to alter it. 3) It understands its own implementation well enough to make a helpful change. Even if the years of time and effort were deliberately taken to make those things possible, further things would be necessary for it to be particularly worrisome: 1) Its programming abilities need to expand to the superhuman somehow -- a "human equivalent" programmer is not going to make radical improvements to a huge software system with man-decades of work behind it in a short period of time. A 100x or 1000x "programming intelligence" enhancement would be needed for that to happen. 2) The core implementation has to be incredibly flawed to squeeze orders of magnitude of extra efficiency into it. We're not really worried about a 30% improvement, we're worried about radical conceptual breakthroughs leading to huge peformance boosts. It stretches the imagination past its breaking point to imagine all of the above happening accidentally without Ben noticing. Therefore, to me, Novamente gets the "Safe AGI" seal of approval until such time as the above steps seem feasible and are undertaken. By that point, there will be years of time to consider its wisdom and hopefully apply some sort of friendliness theory to an actually dangerous stage. I think the development of such a theory is valuable (which is why I give money to SIAI) but I neither expect or want Ben to drop his research until it is ready. There is no need. I could imagine an approach to AGI that has at its core a reflexive understanding of its own implementation; a development pathway where algorithmic complexity theory, predictive models of its own code, code generation from an abstract specification language that forms a fluid self-model, unrestricted invention of new core components, and similar things. Such an approach might, in flights of imagination, be vulnerable to the "oops, it's smarter than me now and I can't pull the plug" scenario. But there's an easy answer to this: Don't build AGI that way. It is clearly not necessary for general intelligence (I don't understand my neural substrate and cannot rewire it arbitrarily at will). Surely certain AGI efforts are more dangerous than others, and the "opaqueness" that Yudkowski writes about is, at this point, not the primary danger. However, in that context, I think that Novamente is, to an extent, opaque in the sense that its actions may not be reduceable to anything clear (call such things "emergent" if you like, or just "complex"). If I understand Loosemore's argument, he might say that AGI without this type of opaqueness is inherently impossible, which could mean that Friendly AI is impossible. Suppose that's true... what do we do then? Minimize risks, I suppose. Perhaps certain protocol issues could be developed and agreed to. As an example: 1. A method to determine whether a given project at a certain developmental stage is "dangerous enough" to require restrictions. It is conceivable, for example, that any genetic programming homework, corewars game, or random programming error could accidentally generate the 200-instruction "key to intelligence" that wreaks havok on the planet... but it's so unlikely that forcing all programming to occur in cement bunkers seems like overkill. 2. Precautions for dangerous programs, such as limits to network access, limits to control of physical devices, and various types of deadman and emergency power cutoffs. I think we're a while away from needing any of this, but agree that it is not too soon to start thinking about it and, as has been pointed out, fostering the mindset that will make sure carefulness is internalized by the field by the time it becomes necessary.
----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=48229463-a08f1c
