On 10/1/07, Matt Mahoney <[EMAIL PROTECTED]> wrote: > > I remain skeptical. Your argument applies to an AGI not modifying its own > motivational system. It does not apply to an AGI making modified copies of > itself. In fact you say: > > > Also, during the development of the first true AI, we would monitor the > > connections going from motivational system to thinking system. It would > > be easy to set up alarm bells if certain kinds of thoughts started to > > take hold -- just do it by associating with certain keys sets of > > concepts and keywords. While we are designing a stable motivational > > system, we can watch exactly what goes on, and keep tweeking until it > > gets to a point where it is clearly not going to get out of the large > > potential well. > > You refer to the humans building the first AGI. Humans, being imperfect, > might not get the algorithm for friendliness exactly right in the first > iteration. So it will be up to the AGI to tweak the second copy a little more > (according to the first AGI's interpretation of friendliness). And so on. So > the goal drifts a little with each iteration. And we have no control over > which way it drifts.
It assumes that first AGI doesn't explicitly thinks about this problem. If it does, and there's a safety net in place to steer these thoughts in the right direction, it won't release another device capable of not being friendly enough. Friendliness for AGI exists on two levels: it itself must be friendly, and it must understand what friendliness is. If it itself only follows friendliness approximately, but understands precisely what friendliness is, it will be able to create new AGIs which are as close to theoretical take on friendliness (probably humans-grounded thing) as it is, or more so. -- Vladimir Nesov mailto:[EMAIL PROTECTED] ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=48373224-683ea6
