Re: [singularity] Defining the Singularity
Starglider wrote: I have no wish to rehash the fairly futile and extremely disruptive discussion of Loosemore's assertions that occurred on the SL4 mailing list. I am willing to address the implicit questions/assumptions about my own position. You may not have noticed that at the end of my previous message I said: If I am right, this is clearly an extremely important issue. For that reason the pros and cons of the argument deserve to be treated with as much ego-less discussion as possible. Let's hope that that happens on the occasions that it is discussed, now and in the future. So what did you do? You immediately went back to the same old, personal-abuse style of trying to win an argument, as exemplified by your opening paragraph, above, and by several similar statements below (for example: You appear to have zero understanding of the functional mechanisms involved in a 'rational/normative' AI system). My new policy is to discuss issues only with people who can resist the temptation to behave like this. For that reason, Michael, you're now killfiled. If anyone else wants to discuss the issues, feel free. Richard Loosemore. Richard Loosemore wrote: The contribution of complex systems science is not to send across a whole body of plug-and-play theoretical work: they only need to send across one idea (an empirical fact), and that is enough. This empirical idea is the notion of the disconnectedness of global from local behavior - what I have called the 'Global-Local Disconnect' and what, roughly speaking, Wolfram calls 'Computational Irreducibility'. This is only an issue if you're using open-ended selective dynamics on or in a substrate with softly-constrained, implicitly-constrained or unconstrained side effects. Nailing that statement down precisely would take a few more paragraphs of definition, but I'll skip that for now. The point is that plenty of complex engineered systems, including almost all existing software systems, don't have this property. The assertion that it is possible (for humans) to design an AGI with fully explicit and rigorous side effect control is contraversial and unproven; I'm optimistic about it, but I'm not sure and I certainly wouldn't call it a fact. What you failed to do was show that it is impossible, and indeed below you seem to acknowledge that it may in fact be possible. The assertion that it is more desirable to build an AGI with strong structural constraints is more complicated. Eliezer Yudkowsky has spent hundreds of thousands of words arguing fairly convincingly for this, and I'm not going to revist that subject here. It is entirely possible to build an AI in such a way that the general course of its behavior is as reliable as the behavior of an Ideal Gas: can't predict the position and momentum of all its particles, but you sure can predict such overall characteristics as temperature, pressure and volume. A highly transhuman intelligence could probably do this, though I suspect it would be very inefficient, partially I expect you'd need strong passive constraints on the power of local mechanisms (the kind the brain has in abundance), which will always sacrifice performance on many tasks compared to unconstrained or intelligently-verified mechanisms. The chances of humans being able to do this are pretty remote, much worse than the already not-promising chances for doing constraint/logic-based FAI. Part of that is due to the fact that while there are people making theoretical progress on constraint-based analysis of AGI, all the suggestions for developing the essential theory for this kind of FAI seem to involve running experiments on highly dangerous proto-AGI or AGI systems (necessarily built before any such theory can be developed and verified). Another problem is the fact that people advocating this kind of approach usually don't appreciate the difficult of designing a good set of FAI goals in the first place, nor the difficulty of verifying that an AGI has a precisely human-like motivational structure if they're going with the dubious plan of hoping an enhanced-human-equivalent can steer humanity through the Singularity successfully. Finally the most serious problem is that an AGI of this type isn't capable of doing safe full-scale self modification until it has full competence in applying all of this as yet undeveloped emergent-FAI theory; unlike constraint-based FAI you don't get any help from the basic substrate and the self-modification competence doesn't grow with the main AI. Until both the abstract knowledge of the reliable-emergent-goal-system-design and the Friendly goal system to use it properly are fully in place (i.e. in all of your prototypes) you're relying on adversarial methods to prevent arbitary self-modification, hard takeoff and general bad news. In short it's ridiculously risky and unlikely to work, orders of magnitude more so than actively verified FAI on a rational AGI substrate, which is already
Re: [singularity] Defining the Singularity
My apologies for the duplication of my previous post; I thought my mail client failed to send the original, but actually it just dropped the echo from the server. Matt Mahoney wrote: Michael Wilson wrote: Hybrid approaches (e.g. what Ben's probably envisioning) are almost certainly better than emergence-based theories... if fully formal FAI turns out to be impossibly difficult we might have to downgrade to some form of probabilistic verification. I think a hybrid approach is still risky. Alas, Seed AI is inherently risky. All we can do is compare levels of risk. By hybrid, do you mean AGI augmented with conventional computing capability? All AGIs implemented on general purpose computers will have access to 'conventional computing capability' unless (sucessfully) kept in a sandbox - and even then anything with a Turing-complete substrate has the potential to develop such capability internally. 'Access to' isn't the same thing as 'augmented with' of course, but I'm not sure exactly what you mean by this (and I'd rather wait for you to explain than guess). Certainly there is the potential for combing formal and informal control mechanisms (as opposed to just local inference and learning machanisms, where informality is much easier to render safe) in an FAI system. Given a good understanding of what's going on I would expect this to be a big improvement on purely informal/implicit methods, though it is possible to imagine someone throwing together the two approaches in such a way that the result is even worse than an informal approach on its own (largely because the kind of reflective analysis and global transforms a constraint-based system can support override what little protection the passive causality constraints in a typical localised-connectionist system give you). My statement above was referencing the structure of the theory used to design/verify the FAI though, not the structure of the FAI itself. I'd characterise a hybrid FAI theory as one that uses some directly provable constraints to narrow down the range of possible behaviours, and then some probabilistic calculation (possibly incorporating experimental evidence) to show that the probability of the AGI staying Friendly is high. The biggest issues with probabilistic calculations are the difficultly of generalising them across self-modification, the fact that any nontrivial uncertainty that compounds across self-modification steps will quickly render the theory useless when applied to an AGI undergoing takeoff, and the fact that humans are just so prone to making serious mistakes when trying to reason probabilistically (even when formal probability theory is used, though that's still /much/ better than intuition/guessing for a problem this complex). As I've mentioned previously, I am optimistic about using narrow AI to help develop AGI designs and FAI theories, and have had some modest success in this area already. I'm not sure if this counts as 'augmenting with conventional computing capability'. Suppose we develop an AGI using a neural model, with all the strengths and weaknesses of humans, such as limited short term memory, inefficient and error prone symbolic reasoning and arithmetic skills, slow long term learning rate, inability to explicitly delete data, etc. Then we observe: A human with pencil and paper can solve many more problems than one without. A human with a calculator is even more powerful. A human with a computer and programming skills is even more powerful. A human with control over a network of millions of computers is even more powerful. Substitute AGI for human and you have all the ingredients to launch a singularity. Absolutely. Plus the AGI has the equivalent of these things directly interfaced into a human's brain, not manipulated through slow and unreliable physical interfaces, and even a brain-like AGI may well be running at a much higher effective clock rate than a human in the first place. This is essentially why AGI is so dangerous even if you don't accept hard and/or early takeoff in an AGI system on its own. If your goal is friendly AI, then not only must you get it right, but so must the AGI when it programs the network to build a more powerful AGI, and so must that AGI, and so on. You cannot make a mistake anywhere along the chain. Thus probabilistic methods have a serious problem remaining effective under recursive self-modification; any flaws in the original theory that don't get quickly and correctly fixed by the AGI (which requires an accurate meta-description of what your FAI theory is supposed to do...) are likely to deviate the effective goal system out of the human-desirable space. If you /have/ to use probabilistic methods, they are all kinds of mitigating strategies you can take; Eliezer actually covered quite a few of them back in Creating A Friendly AI. But provable Friendliness (implemented with many layers of redundancy just to be sure) is better if it's
Re: [singularity] Motivational Systems that are stable
The last I heard, computers are spied upon because of the language the computer is generating. Why would the government care about the guy that picks up garbage? Richard Loosemore wrote, Wed, Oct 25, 2006: The word trapdoor is a reference to trapdoor algorithms that allow computers to be spied upon. If you feet guilty about something then you will feel that your ethical values are being compromised. Technology is without a doubt the age of the future If you have posted, said or done, chances are if will come and haunt you. The only way to change the algorithms is to change the thoughts. Just my thoughts, let me know what you think. Anna:) On 10/25/06, Richard Loosemore [EMAIL PROTECTED] wrote: Anna Taylor wrote: On, Wed, Oct 25, 2006 at 10:11 R. Loosemore wrote: What I have in mind here is the objection (that I know some people will raise) that it might harbor some deep-seated animosity such as an association between human beings in general and something 'bad' that happened to it when it was growing up ... we would easily be able to catch something like that if we had a trapdoor on the motivational system. I'm not clear what you meant, could you rephrase? I understood, what I have in mind is a trapdoor of the motivational system:) Do you think motivation is a key factor that generates singularity-level events? Am I understanding properly? Just curious Anna:) Anna, The word trapdoor is a reference to trapdoor algorithms that allow computers to be spied upon: I meant it in a similar sense, that the AI would be built in such a way that we could (in the development stages) spy on what was happening in the motivational system to find out whether the AI was developing any nasty intentions. The purpose of the essay was to establish that this alternative approach to creating a friendly AI would be both viable and (potentially) extremely stable. It is a very different approach to the one currently thought to be the only method, which is to prove properties of the AI's goal system mathematically a task that many consider impossible. By suggesting this alternative I am saying that mathematical proof may be impossible, but guarantees of very strong kind may well be possible. As you probably know, many people (including me) are extremely concerned that AI be developed safely. Hope that helps, Richard Loosemore - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED] - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [singularity] Defining the Singularity
- Original Message From: Starglider [EMAIL PROTECTED] To: singularity@v2.listbox.com Sent: Wednesday, October 25, 2006 2:32:27 PM Subject: Re: [singularity] Defining the Singularity All AGIs implemented on general purpose computers will have access to 'conventional computing capability' unless (sucessfully) kept in a sandbox - and even then anything with a Turing-complete substrate has the potential to develop such capability internally. 'Access to' isn't the same thing as 'augmented with' of course, but I'm not sure exactly what you mean by this (and I'd rather wait for you to explain than guess). I was referring to one possible implementation of AGI consisting of part neural or brainlike implementation and part conventional computer (or network) to combine the strengths of both. The architecture of this system would be that the neural part has the capability to write programs and run them on the conventional part in the same way that humans interact with computers. This seems to me to be the most logical way to build an AGI, and probably the most dangerous Vinge described four ways in which the Singularity can happen (quoting from http://mindstalk.net/vinge/vinge-sing.html) There may be developed computers that are awake and superhumanly intelligent. Large computer networks (and their associated users) may wake up as a superhumanly intelligent entity. Computer/human interfaces may become so intimate that users may reasonably be considered superhumanly intelligent. Biological science may provide means to improve natural human intellect. Vinge listed these possibilities in order from least to most interaction with humans. I believe that less interaction means less monitoring and control, and therefore greater possibility that something will go wrong. As long as human brains remain an essential component of a superhuman intelligence, it seems less likely that this combined intelligence will destroy itself. If AGI is external or independent of human existence, then there is a great risk. But if you follow the work of people trying to develop AGI, it seems that is where we are headed, if they are successful. We already have hybrid computational systems that depend on human cooperation with machines. For some hybrid systems such as customer service, the airline reservation/travel agency system, or the management of large corporations, there is economic pressure to automate the human parts of the computation. Consider this possibility. We build an AGI that is part neural, part conventional computer, modeled after a system of humans with programming skills and a network of computers. Even if you could prove friendliness (which you can't), then you still have the software engineering problem. Program specifications are written in natural language, which is ambiguous, imprecise and incomplete. People make assumptions. People make mistakes. Neural models of people will make mistakes. Each time the AGI programs a more intelligent AGI, it will make programming errors. Proving program correctness is equivalent to the halting problem, so the problem will not go away no matter how smart the AGI is. Using heuristic methods won't help, because after the first cycle of the AGI programming itself, the level of sophistication of the software will be beyond our capability to understand it (or else we would have written it that way in the first place). You will have no choice but to trust the AGI detect its own errors. -- Matt Mahoney, [EMAIL PROTECTED] - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED]
