On Sun, Sep 11, 2016 at 5:15 PM, John Clark <[email protected]> wrote: > On Sun, Sep 11, 2016 at 7:07 AM, Telmo Menezes <[email protected]> > wrote: > >> > >> That is one type of worry. The other (e.g.: the "paper clip" scenario) > > > The paper clip scenario could only happen in a intelligence that had a top > goal that was fixed and inflexible. Humans have no such goal, not even the > goal of self preservation, and there is a reason Evolution never came up > with a mind built that way, Turing proved in 1935 that a mind like that > couldn't work.
Are you referring to the halting problem? > You could have a machine with very little intelligence > obsessed with making paper clips, rather like a virus is obsessed with > making copies of itself, but in the long run Mr. Jupiter Brain will be able > to outwit the dumb machine just as we are making progress in outwitting > viruses. It is doubtful that a superintelligence could develop under such a dumb utility function. A superintelligence will likely require heuristics similar to us (e.g. curiosity). But we could direct it toward the task of paper-clip-maximization by encoding that in the utility function. My claim is that it would probably figure out how to change the utility function to constant infinity much before turning the planet into paper clips. Telmo. > John K Clark > > > > -- > You received this message because you are subscribed to the Google Groups > "Everything List" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/everything-list. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Everything List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/everything-list. For more options, visit https://groups.google.com/d/optout.
