On 12/1/06, Matt Mahoney <[EMAIL PROTECTED]> wrote:
--- Hank Conn <[EMAIL PROTECTED]> wrote: > The further the actual target goal state of that particular AI is away from > the actual target goal state of humanity, the worse. > > The goal of ... humanity... is that the AGI implemented that will have the > strongest RSI curve also will be such that its actual target goal state is > exactly congruent to the actual target goal state of humanity. This was discussed on the Singularity list. Even if we get the motivational system and goals right, things can still go badly. Are the following things good? - End of disease. - End of death. - End of pain and suffering. - A paradise where all of your needs are met and wishes fulfilled. You might think so, and program an AGI with these goals. Suppose the AGI figures out that by scanning your brain and copying the information into a computer and making many redundant backups, that you become immortal. Furthermore, once your consciousness becomes a computation in silicon, your universe can be simulated to be anything you want it to be. The "goals of humanity", like all other species, was determined by evolution. It is to propagate the species.
That's not the goal of humanity. That's the goal of the evolution of humanity, which has been defunct for a while. This goal is met by a genetically programmed
individual motivation toward reproduction and a fear of death, at least until you are past the age of reproduction and you no longer serve a purpose. Animals without these goals don't pass on their DNA. A property of motivational systems is that cannot be altered. You cannot turn off your desire to eat or your fear of pain. You cannot decide you will start liking what you don't like, or vice versa. You cannot because if you could, you would not pass on your DNA.
You are confusing this abstract idea of an optimization target with the actual motivation system. You can change your motivation system all you want, but you woulnd't (intentionally) change the fundamental specification of the optimization target which is maintained by the motivation system as a whole. Once your brain is in software, what is to stop you from telling the AGI
(that you built) to reprogram your motivational system that you built so you are happy with what you have?
Uh... go for it. To some extent you can do this. When rats can
electrically stimulate their nucleus accumbens by pressing a lever, they do so nonstop in preference to food and water until they die. I suppose the alternative is to not scan brains, but then you still have death, disease and suffering. I'm sorry it is not a happy picture either way.
Or you have no death, disease, or suffering, but not wireheading. -- Matt Mahoney, [EMAIL PROTECTED]
----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
