Alex, I can SO relate to your thoughts here. Back in 2001 I was on my way out - atrial fibrillation making me unconscious most afternoons, a gigantic pot belly that was getting bigger, etc. After 3 months of full time R&D I came up with what looked like the solution - kick my daytime body temperature back up to 37C where it belonged. Early testing showed that this would fundamentally change ME - the way I thought, etc. It would make me considerably more intelligent, and doubtless expose that my lesser-intelligent efforts through my preceding 56 years had been largely a waste of effort, etc. Further, there was no foreseeable prospect for going back, once this step had been taken. I would probably like the new "me", but the old "me" would almost certainly be gone for good. As you might guess from the fact that I am here to write this, I made the decision to "go for it", but this decision was NOT as simple as it might sound.
In contrast, I knew a guy in England named Rob who developed a tumor behind one eye that was starting to invade his brain. ALL of the mainstream medical approaches offered little chance for success (<30% survive for one year) and involved Rob getting a frontal lobotomy, whereas there were some unproven alternative methods that might be able to selectively kill just the tumor cells while leaving the brain cells. For me the choice would have been simple - I would have gone for the alternative medical methods with the knowledge that I would probably die, rather than going for the mainstream medical methods that absolutely guaranteed that I would become an idiot. Rob chose the mainstream medical route, which killed him about 6 weeks later. Rob's wife was devastated, not so much because Rob had gotten sick and died, but rather because Rob chose a path that meant that no matter how things turned out, that Rob had in effect chosen to abandon her. Rob's widow is one of the top alternative cancer treatment guru's in England, which makes Rob's decision even more inexplicable. Every night we close our eyes and "die", and the next morning we are born anew and somewhat different. However, sometimes the changes are MUCH bigger than a night's sleep. The 13 years from my own decision have shown me that intelligence, and most especially changes in intelligence, have a cumulative effect that is MUCH larger than the immediate effect. I am constantly developing new logical methods to save and use to lever solutions to difficult problems, e.g. like reverse reductio ad absurdum on which I have posted in the past. This is a logical method usable to resolve otherwise intractable arguments. Similarly, for an AGI to achieve its potential, it too must be able to build its own mental toolbox, part of which would be to improve its own thought processes. Anything we might build into an AGI would eventually become just another problem to be circumvented. Steve ===================== On Mon, May 5, 2014 at 3:44 PM, Alex Miller <[email protected]> wrote: > On Mon, May 5, 2014 at 1:11 PM, Aaron Hosford <[email protected]> wrote: > >> I think the trick lies in multiple redundancies, both for triggering and >> effecting termination. >> >> We should also design in as many mechanisms as possible to avoid the >> problem in the first place. For example, a very strong negative reward >> signal for the AGI even considering modifications to certain critical zones >> of its own software or hardware, particularly those that determine the >> reward levels themselves, in a reinforcement learning-based AGI. (This >> could be interpreted as an overpowering urge to "stay true to oneself" on >> the part of the AGI, meaning that it would try to preserve its own personal >> identity.) >> >> > (I realize that AIXI is not practical, but this is a philosophical > discussion) > > Suppose you were gifted with an infinitely fast computer, and wanted to > put it to work for the good of the humankind running AIXI. How would you > actually implement its reward function, in code, so that it doesn't go off > the reservation? > > Is this still very much an open problem? > > *AGI* | Archives <https://www.listbox.com/member/archive/303/=now> > <https://www.listbox.com/member/archive/rss/303/10443978-6f4c28ac> | > Modify > <https://www.listbox.com/member/?&;> > Your Subscription <http://www.listbox.com> > -- Full employment can be had with the stoke of a pen. Simply institute a six hour workday. That will easily create enough new jobs to bring back full employment. ------------------------------------------- AGI Archives: https://www.listbox.com/member/archive/303/=now RSS Feed: https://www.listbox.com/member/archive/rss/303/21088071-f452e424 Modify Your Subscription: https://www.listbox.com/member/?member_id=21088071&id_secret=21088071-58d57657 Powered by Listbox: http://www.listbox.com
