On 13/07/07, Mike Tintner <[EMAIL PROTECTED]> wrote:
Comment: You and others seem to be missing the point, which obviously needs spelling out. There is no way of endowing any agent with conceptual goals that cannot be interpreted in ways opposite to the designer's intentions - that is in the general, abstract nature of language & symbolic systems. For example, the general, abstract goal of "helping humanity" can legitimately in particular, concrete situations be interpreted as wiping out the entire human race (bar, say, two) - for the sake of future generations.
With humans we have always had to deal with not only honest misunderstanding but also frank treachery or malice. I would hope that treachery is less likely to be a problem with AI's, but surely the risk that an AI will do something bad as a result of the treachery of another human will be at least as great as the risk that it will do something bad due to unforeseen consequences of following instructions. Our defence against such a threat will then be the same as our general defence against threats from other humans: that no single agent will rapidly be able to rise to a level of power so as to be able to dominate all of the others. -- Stathis Papaioannou ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=4007604&id_secret=20956299-e12a67
