The problem isn't that the AGI will violate its original goals; it's that the AGI will eventually do something that will destroy something really important in such a way as to satisfy all of its constraints. By setting constraints on the AGI, you're trying to think of everything bad the AGI might possibly do in advance, and that isn't going to work.
- Tom --- Stathis Papaioannou <[EMAIL PROTECTED]> wrote: > On 02/07/07, Tom McCabe <[EMAIL PROTECTED]> > wrote: > > But killing someone and then beating them on the > > chessboard due to the lack of opposition does > count as > > winning under the formal rules of chess, since > there's > > nothing in the rules of chess about killing the > > opponent. The rule "don't kill, strangle, drug, > maim, > > injure, or otherwise physically hurt" the opponent > > exists only in your head; it doesn't exist in any > > chess rulebook and isn't automatically transferred > to > > the AGI. > > The problem can easily be specified with > constraints, and the > constraints can't be changed without changing the > problem. You seem to > be assuming that the AI will look at the task and > decide that winning > is the real goal, and the constraints are something > that has to be > overcome. But there is no reason to think this way > unless there is > another explicit goal which does specify that > winning is the important > thing, trumping other considerations. > > > > -- > Stathis Papaioannou > > ----- > This list is sponsored by AGIRI: > http://www.agiri.org/email > To unsubscribe or change your options, please go to: > http://v2.listbox.com/member/?&; > ____________________________________________________________________________________ Choose the right car based on your needs. Check out Yahoo! Autos new Car Finder tool. http://autos.yahoo.com/carfinder/ ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=4007604&user_secret=7d7fb4d8
