On Tue, Sep 10, 2019 at 7:29 PM 'Brent Meeker' < [email protected]> wrote:
*>>> I think they would be careful NOT have it value its survival. * > > >> I think that would mean the AI would need to be in intense constant > pain for that to happen, or be deeply depressed like the robot Marvin in > Hitchhiker's Guide to the Galaxy. And I think it would be grossly unethical > to make such an AI. > > * > Why would it mean that? Why wouldn't the AI agree with Bruno that it > was just computation and it existed in Platonia anyway so it was > indifferent to transient existence here?* > Because people on this list may say all sorts or screwy things when they slip into philosophy mode but even Bruno will jump out of the way when he crosses the street if he sees a bus coming straight for him, or at least he will if he isn't in constant intense pain or is deeply depressed. >> You can't outsmart someone smarter than you, the humans are never going > to be able to shut it off unless the AI wants to be shut off. > > > * > Exactly why you might program it to want to be shut off in certain > circumstances.* > I have no doubt humans will put something like that in its code, but if the AI has the ability to modify itself, and it wouldn't be much of a AI if it didn't, then that code could be changed. And I have no doubt the humans will put in all sorts of safeguards that the humans consider ingenious to prevent the AI from doing that, but the fact remains you can't outsmart something smarter than you. * > Of course the problem with "We can always shut it off." is that once > you rely on it, you don't dare shut if off because it knows better than you > do and you know it knows better.* > Yes that's one very serious obstacle that prevents humans from just shutting it off, but another problem is the Jupiter Brain knows you better than you do, so it can find your weakness and can trick or charm or flatter you to do what it wants. John K Clark -- You received this message because you are subscribed to the Google Groups "Everything List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/everything-list/CAJPayv0BM84tKEWinvWeysL5Ns8ePBJ2NzisT8vgOQ2NGKWrFw%40mail.gmail.com.
