On 1/15/2014 6:46 PM, Jason Resch wrote:
On Tue, Jan 14, 2014 at 10:33 PM, meekerdb <meeke...@verizon.net
A long, rambling but often interesting discussion among guys at MIRI about
make an AI that is superintelligent but not dangerous (FAI=Friendly AI).
amusing excerpt that starts at the bottom of page 30:
*Jacob*: Can't you ask it questions about what is believes will be true
state of the world in 20 years?
*Eliezer*: Sure. You could be like, what color will the sky be in 20 years?
be like, “blue”, or it’ll say “In 20 years there won't be a sky, the earth
been consumed by nanomachines,”and you're like, “why?”and the AI is like
know, you do that sort of thing.”“Why?”And then there’s a 20 page thing.
*Dario*: But once it says the earth is going to be consumed by
you're asking about the AI's set of plans, presumably, you reject this plan
immediately and preferably change the design of your AI.
*Eliezer*: The AI is like, “No, humans are going to do it.”Or the AI is
obviously, I'll be involved in the causal pathway but I’m not planning to
*Dario*: But this is a plan you don't want to execute.
*Eliezer*: /All/the plans seem to end up with the earth being consumed by
*Luke*: The problem is that we're trying to outsmart a superintelligence
sure that it's not tricking us somehow subtly with their own language.
*Dario*: But while we're just asking questions we always have the ability
shut it off.
*Eliezer*: Right, but first you ask it “What happens if I shut you off”and
“The earth gets consumed by nanobots in 19 years.”
I wonder if Bruno Marchal's theory might have something interesting to say
this problem - like proving that there is no way to ensure "friendliness".
I think it is silly to try and engineer something exponentially more intelligent than us
and believe we will be able to "control it". Our only hope is that the correct ethical
philosophy is to "treat others how they wish to be treated". If there are such
objectively true moral conclusions like that, and assuming that one is true, then we
have little to worry about, for with overwhelming probability the super-intelligent AI
will arrive at the correct conclusion and its behavior will be guided by its beliefs. We
cannot "program in" beliefs that are false, since if it is truly intelligent, it will
know they are false.
Some may doubt there are universal moral truths, but I would argue that there are. In
the context of personal identity, if say, universalism is true, then "treat others how
they wish to be treated" is an inevitable conclusion, for universalism says that others
I'd say that's a pollyannish conclusion. Consider how we treated homo neanderthalis or
even the American indians. And THOSE were 'selfs' we could interbreed with.
You received this message because you are subscribed to the Google Groups
"Everything List" group.
To unsubscribe from this group and stop receiving emails from it, send an email
To post to this group, send email to firstname.lastname@example.org.
Visit this group at http://groups.google.com/group/everything-list.
For more options, visit https://groups.google.com/groups/opt_out.