On Sat, Mar 8, 2008 at 6:30 PM, Mark Waser <[EMAIL PROTECTED]> wrote: > > > > This sounds like magic thinking, sweeping the problem under the rug of > > 'attractor' word. Anyway, even if this trick somehow works, it doesn't > > actually address the problem of friendly AI. The problem with > > unfriendly AI is not that it turns "selfish", but that it doesn't get > > what we want from it or can't foresee consequences of its actions in > > sufficient detail. > > You need to continue reading but it's also clear that you and I don't have > the same view of Friendliness (since your view appears to me to be closer to > that of Eliezer). It does not matter if the FAI doesn't get what we want > from it. That is entirely irrelevant. All that it needs to get is what we > *DON'T* want it to do. > > Foreseeing consequences of it's actions is an intelligence argument, *NOT* a > Friendliness argument. > > You have raised two irrelevant points. > > Also, I do not mean to "sweep the problem under the rug" with the magical > attractor word. It's just the simplest descriptor for what I trying to > explain. If you don't *clearly* see my whole argument, please ask me to > explain. There is no magical mumbo-jumbo here. Call me on anything that > you think I am glossing over or getting wrong. >
OK, I'll elucidate relevance of my comments about AI's intelligence and cause of remarking about magical thinking. I asked about the reason why dominant AGI won't be able to choose to annihilate all lesser forms to assure permanency of its domination. You replied thusly: > > What is different in my theory is that it handles the case where "the > dominant theory turns unfriendly". The core of my thesis is that the > particular Friendliness that I/we are trying to reach is an "attractor" -- > which means that if the dominant structure starts to turn unfriendly, it is > actually a self-correcting situation. > Can you explain it without using the word "attractor"? I can't see why sufficiently intelligent system without "brittle" constraints should be unable to do that. By "brittle" constraint I mean some arbitrary thing that system is prevented from doing, which we would expect a rational agent to do in some circumstances, like a taboo on ever using a word "attractor". I come to believe that if we have a sufficiently intelligent AGI that can understand what we mean by saying "friendly AI", we can force this AGI to actually produce a verified friendly AI, with minimum the risk of it being defective or a Trojan horse of our captive ad-hoc AGI, after which we place this friendly AI in dominant position. So the problem of friendly AI comes down to producing a sufficiently intelligent ad-hoc AGI (which will probably will have to be not that ad-hoc to be sufficiently intelligent). > > All that it needs to get is what we *DON'T* want it to do. > I don't see why we should create an AGI that we can't extract useful things from (although it doesn't necessarily follow from your remark). On the other hand, if AGI is not sufficiently intelligent, it may be dangerous even if it seems to understand some simpler constraint, like "don't touch the Earth". If it can't foresee consequences of its actions, it can do something that will lead to demise of old humanity some hundred years later. It can accidentally produce a seed AI that will grow into something completely unfriendly and take over. It can fail to contain an outbreak an unfriendly seed AI created by humans. And so on, and so forth. We really want place of power to be filled by something smart and beneficial. As an aside, I think that safety of future society can only be guaranteed by mandatory uploading and keeping all intelligent activities within an "operation system"-like environment which prevents direct physical influence and controls rights of computation processes that inhabit it, with maybe some exceptions to this rule, but only given verified surveillance on all levels to prevent a physical space-based seed AI from being created. -- Vladimir Nesov [EMAIL PROTECTED] ------------------------------------------- agi Archives: http://www.listbox.com/member/archive/303/=now RSS Feed: http://www.listbox.com/member/archive/rss/303/ Modify Your Subscription: http://www.listbox.com/member/?member_id=8660244&id_secret=95818715-a78a9b Powered by Listbox: http://www.listbox.com
