I am not describing a nightmare scenario where a SAI forces its will upon us. People will *want* these things.
If you were dying and we had the technology to upload your mind, wouldn't you? Orwell's 1984 predicted a world where a totalitarian government watched your every move. What he failed to predict is that it would happen in a democracy. People want surveillence. You want cameras in businesses for better security. You use credit cards that track your spending because cash can be stolen. You let automated toll booths track your movements so you don't have to stop. You trust Yahoo/Google/your ISP/company/university with your email because it's convenient. You give up privacy a little bit at a time and each time you get something in return. Big changes happen slowly. The scenario I described won't happen all at once. Each small step of integrating our consciousness into machines comes with some benefit. Not everyone will choose to upload their minds. Not everyone who does will alter their memory or simulate fake worlds. Those who do will experiment with minor alterations, at least at first. Not everyone will mess with their motivational logic to directly stimulate the simulated pleasure center of the brain. But some will, if the SAI allows it. Question: is an SAI friendly if it gives you what you want? Or should it decide what's best for you? -- Matt Mahoney, [EMAIL PROTECTED] ----- Original Message ---- From: Richard Loosemore <[EMAIL PROTECTED]> To: [email protected] Sent: Friday, October 27, 2006 10:30:52 AM Subject: Re: [singularity] Defining the Singularity Matt, This is a textbook example of the way that all discussions of the consequences of a singularity tend to go. What you have done here is to repeat the same song heard over and over again from people who criticise the singularity on the grounds that one or another nightmare will *obviously* happen. No distinction between a fantasy of the worst that could happen, on the one hand, and a realistic, critical assessment of what is likely to happen, on the other. Thus, summarizing what you just said: 1) A SAI *could* allow us to upload to super powerful computers (part of a vast network, etc. etc.) .... so therefore it *will* force this upon us. 2) A SAI *could* allow us to get rid of all the "living organisms" since they are not needed (presumably you mean our bodies), so therefore the SAI *will* force this upon us. 3) You insinuate that the SAI will *insist* that we don't need "all those low level sensory processing and motor skills [we] learned over a lifetime" and so therefore the SAI *will* deprive us of them. 4) You insinuate that the SAI will *insist* that we should get rid of any bad memories from childhood, if they trouble us, and so therefore it *will* do this to us whether we want it to or not. You present all of these as if they would happen against our will, as if they would be forced upon the human race. You don't come right out and say this, you just list all of these nightmare scenarios and then conclude that "your nervousness is justified". But nowhere do you even consider the possibility that any SAI that did this would be stupid and vicious .... you implicitly assume that even the best-case SAI would be this bad. If, instead, you had said: 5) People could *choose* to upload into super powerful computers connected to simulated worlds, if they felt like it (instead of staying as they are and augmenting their minds when the fancy took them) .... but although some people probably would, most would chose not to do this. 6) Some might *choose* to do the above and also destroy their bodies. Probably not many, and even those who did could at any later time decide to relocate back into reconstructed versions of their old bodies, so it would be no big deal either way. 7) Some people might *choose* to dispense with the learned motor and sensory skills that were specific to their natural bodies ... but again, most would not (why would they bother to do this?), and they could always restore them later if they felt like it. 8) Some people might *choose* to erase painful memories. They might also take the precaution of storing them somewhere, so they could change their minds and retrieve them in the future. ..... then the alternative conclusion would be: sounds like there is no problem with this. Your version (items 1-4) was presented without any justification for why the SAI would impose its will instead of simply offering us lifestyle choices. Why? Your presentation here is just a classic example: every single debate or discussion of the consequences of the singularity, it seems, is totally dominated by this kind of sloppy thinking. Richard Loosemore Matt Mahoney wrote: > I have raised the possibility that a SAI (including a provably friendly > one, if that's possible) might destroy all life on earth. > > By friendly, I mean doing what we tell it to do. Let's assume a best > case scenario where all humans cooperate, so we don't ask, for example, > for the SAI to kill or harm others. So under this scenario the SAI > figures out how to end disease and suffering, make us immortal, make us > smarter and give us a richer environment with more senses and more > control, and give us anything we ask for. These are good things, > right? So we achieve this by uploading our minds into super powerful > computers, part of a vast network with millions of sensors and effectors > around the world. The SAI does pre- and postprocessing on this I/O, so > it effectively can simulate any enviroment if we want it to. If you > don't like the world as it is, you can have it simulate a better one. > > And by the way, there's no more need for living organisms to make all > this run, is there? Brain scanning is easier if you don't have to keep > the patient alive. Don't worry, no data is lost. At least no important > data. You don't really need all those low level sensory processing and > motor skills you learned over a lifetime. That was only useful when you > still had your body. And while were at it, we can alter your memories > if you like. Had a troubled childhood? How about a new one? > > Of course there are the other scenarios, where the SAI is not proven > friendly, or humans don't cooperate... > > Vinge describes the singularity as the end of the human era. I think > your nervousness is justified. > > -- Matt Mahoney, [EMAIL PROTECTED] > > > ----- Original Message ---- > From: deering <[EMAIL PROTECTED]> > To: [email protected] > Sent: Thursday, October 26, 2006 7:56:06 PM > Subject: Re: [singularity] Defining the Singularity > > All this talk about trying to make a SAI Friendly makes me very > nervous. You're giving a superhumanly powerful being a set of > motivations without an underlying rationale. That's a religion. > > The only rational thing to do is to build an SAI without any > preconceived ideas of right and wrong, and let it figure it out for > itself. What makes you think that protecting humanity is the greatest > good in the universe? ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED] ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED]
