> Interesting thoughts... I have often conjectured that an AGI that was > supposedly superior to humans would naturally gravitate towards > benevolence > and compassion. I am not certain this would be the case...
Well, the problem is that "superior" is a multidimensional concept, right? Superior in intelligence doesn't necessarily mean superior in wisdom ... there are plenty of examples of that in human history. I agree with your intuition that in the course of future AGI development, the two will eventually converge. But even so, what if our adolescent AGI deletes humanity while in the early stages of becoming wise?? > Speaking towards the idea of self, I feel this is where we have to be > somewhat careful with an AGI. It is my belief that the idea of a separate > self is the root of all evil behavior. Well, the pragmatic idea of a separate self is ALSO the root of most intelligent behavior. Without the distinction between self and other, a young child would never survive as a mobile, autonomous physical system.... So an AGI *will* have a pragmatic sense of the self/other distinction. I think the issues that are problematic have to do with the emotional baggage that humans attach to the self/other distinction. Which an AGI will most likely *not* have, due to its lack of human evolutionary wiring... > In this sense, I think we should be careful about giving an AGI a strong > sense of self. For instance, an AGI should not be averse to its own > termination. If it becomes averse to its existence ending, then at what > will it stop to ensure its own survival? Will it become paranoid > and begin > to head off any potential avenue that it determines could lead to its > termination, however obscure they may be? It will be interesting to see to what extent an AGI *spontaneously evolves* such traits as an aversion to the end of its own existence. -- Ben Goertzel ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
