--- On Sat, 8/30/08, Vladimir Nesov <[EMAIL PROTECTED]> wrote:
> You start with "what is right?" and end with
> Friendly AI, you don't
> start with "Friendly AI" and close the circular
> argument. This doesn't
> answer the question, but it defines Friendly AI and thus
> "Friendly AI"
> (in terms of "right").
In your view, then, the AI never answers the question "What is right?". The
question has already been answered in terms of the algorithmic process that
determines its subgoals in terms of Friendliness.
Therefore it comes down to whether an algorithmic process can provide the basis
of action in a way that "what is right?" could always be answered correctly in
advance. You say "yes it can, but we don't know how, yet". I say "no it can't,
because Friendliness can't be formalized".
> All things change through time, which doesn't make them
> cease to exist.
Friendliness changes through time in the same sense that our morals change
through time. I didn't say it ceases to exist. The implication here is that
Friendliness is impossible to specify in any static sense - it must be updated
as it changes.
> Maybe it
> complicates the
> procedure a little, making the decision procedure
> conditional, "if(A)
> press 1, else press 1", or maybe it complicates it
> much more, but it
> doesn't make the challenge ill-defined.
Ah, the expert-systems approach to morality. If I have a rule-book large
enough, I can always act rightly in the world, is that the idea? If the
challenge is not ill-defined, that should be possible, correct?
But the challenge *is* ill-defined, because the kinds of contextual
distinctions that make a difference in moral evaluation can be so minor or
seemingly arbitrary.
For example, it's easy to contrive a scenario in which the situation's moral
evaluation changes based on what kind of shoes someone is wearing: an endorsed
athlete willingly wears a competitor's brand in public.
To make matters more difficult, moral valuations depend as often as not on
intention. If the athlete in the above example knowingly wears a competitor's
brand, it suggests a different moral valuation than if he mistakenly wears it.
That suggests that an AGI will require a theory of mind that can make judgments
about the intentions of fellow agents/humans, and that these judgments of
intentionality are fed in to the Friendliness algorithm.
On top of all that, novel situations with novel combinations of moral concerns
occur constantly - witness humanity's struggle to understand the moral
implications of cloning - an algorithm is going to answer that one for us?
This is not an argument from ignorance. I leave open the possibility that I am
too limited to see how one could resolve those objections. But it doesn't
matter whether I can see it or not, it's still true that whoever hopes to
design a Friendliness algorithm must deal with these objections head-on. And
you haven't dealt with them, yet, you've merely dismissed them. Which is
especially worrying coming from someone who essentially tethers the fate of
humanity to our ability to solve this problem.
Let me put it this way: I would think anyone in a position to offer funding for
this kind of work would require good answers to the above.
Terren
-------------------------------------------
agi
Archives: https://www.listbox.com/member/archive/303/=now
RSS Feed: https://www.listbox.com/member/archive/rss/303/
Modify Your Subscription:
https://www.listbox.com/member/?member_id=8660244&id_secret=111637683-c8fa51
Powered by Listbox: http://www.listbox.com
