[EMAIL PROTECTED] wrote:
Hello Richard,
If it's not too lengthy and unwieldy to answer, or give a general sense
as to why yourself and various researchers think so...
Why is it that in the same e-mail you can make the statement so
confidently that "ego" or sense of selfhood is not something that the
naive observer should expect to just emerge naturally as a consequence
of succedding in building an AGI (and the qualities of which, such as
altruism, will have to be specifically designed in), while you just as
confidently state that consciousness itself will merely arise 'for free'
as an undesigned emergent gift of building an AGI?
I'm really curious about researcher's thinking on this and similar
points. It seems to lay at the core of what is so socially
controversial about singualrity-seeking in the first place.
Thanks,
~Robert S.
First, bear in mind that opinions are all over the map, so what I say
here is one point of view, not everyone's.
First, about consciousness.
The full story is a long one, but I will try to cut to the part that is
relevant to your question.
Consciousness itself, I believe, is something that arises because of
certain aspects of how the mind represents the world, and how it uses
those mechanisms to represent what is going on inside itself. There is
not really one thing that is "consciousness", of course (people use that
word to designate many different things), but the most elusive aspects
are the result of strange things happening in these representation
mechanisms.
The thing that actually gives rise to the thing we might call pure
"subjective consciousness" (including qualia, etc) is a weirdness that
happens when the system "bottoms out" during an attempt to unpack the
meaning of things: normally, the mind can take any concept and ask
itself "What *is* this thing?", and come up with a meaningful answer
that involves more primitive concepts. Ask this of the concept [chair]
and you might get a bunch of other concepts involving legs, a seat, a
back, the act of human sitting, and so on. But when this same analysis
mechanism is applied to certain concepts that are at the root of the
mind's representation system, something peculiar happens: the system
sets up a new temporary concept (a placeholder) ready to take the
answer, but then it fails to actually attach anything to it. So when it
asks itself "What is the essence of redness?" the answer is that it is
"....", and nothing happens. Or rather something *more* than nothing
happens, because the placeholder concept is set up, and then nothing is
attached to it. The mind thinks "There is *something* it is like to be
the essence of redness, but it is mysterious and indescribable".
Now, you might want to quickly jump to the conclusion that what I am
saying here is that "consciousness" is an artifact of the way minds
represent the world.
This is a very important point: I am not aligning myself with those who
dismiss consciousness as just an artifact (or an epiphenomenon). In a
sense, what I have said above does look like a dismissal of
consciousness, but there is a second step in the argument.
In this second step I point out that if you look deeply into what this
mechanism does, and the question of how the mind assesses what is "real"
or what things actually exist and can be analyzed or talked about
meaningfully, you are forced to the conclusion that our best possible
ideas about which things in the world "really exist" and which things
are merely artifacts of our minds, it turns out that most of the time
you can make a good separation, but there is one specific area where it
will always be impossible to make a separation. In this one unique area
- namely, the thing we call "consciousness" - we will always be forced
to say that, scientifically, we have to accept that there is the thing
we call consciousness is as real as anything else in the world, but
unlike all other real things, it cannot be analyzed further. This is
not an expression of "we don't know how to analyze this yet, but maybe
in the future we will...." it is a clear statement that consciousness is
just as real as anything else in the world, but it must necessarily be
impossible to analyze.
Now, going back to your question, this means that if we put the same
kinds of mechanisms into a thinking machine as we have in our minds,
then it will have "consciousness" just as we do, and it will experience
the same feeling of mystery about it. We will never be able to
objectively verify that consciousness is there (just as we cannot do
this for each other, as humans) but we will be able to say precisely why
we would expect the system to report its experience, and (most
importantly) we will be able to give solid reasons for why we cannot
analyze the nature of consciousness any further.
But would those mechanisms be present in a machine? This is fairly easy
to answer: if the machine were able to understand the world as well as
us, then it is pretty much inevitable that the same class of mechanisms
will be there. It is not really the exact mechanisms themselves that
cause the problem, it is a fundamental issue to do with representations,
and any sufficiently powerful representation system will have to show
this effect. No way around it.
So that is the answer to why I can say that consciousness will emerge
"for free". We will not deliberately put it in, it will just come along
if we make the system able to fully understand the world (and we are
assuming, in this discussion, that the system is able to do that).
(I described this entire theory of consciousness in a poster that I
presented at the Tucson conference two years ago, but still have not had
time to write it up completely. For what it is worth, I got David
Chalmers to stand in front of the poster and debate the argument with me
for a short while, and his verdict was that it was an original line of
argument.)
The second part of your question was why the "ego" or "self" will, on
the other hand, not be something that just emerges for free.
I was speaking a little loosely here, because there are many meanings
for "ego" and "self", and I was just zeroing in on one aspect that was
relevant to the original question asked by someone else. What I am
menaing here is the stuff that determines how the system behaves, the
things that drive it to do things, its agenda, desires, motivations,
character, and so on. (The important question is whether it could be
trusted to be benign).
Here, it is important to understand that the mind really consists of two
separate parts: the "thinking part" and the motivation/emotional
system. We know this from our own experience, if we think about it
enough: we talk about being "overcome by emotion" or "consumed by
anger", etc. If you go around collecting expressions like this, you
will notice that people frequently talk about these strong emotions and
motivations as if they were caused by a separate module inside
themselves. This appears to be a good intuition: they are indeed (as
far as we can tell) the result of something distinct.
So, for example, if you built a system capable of doing lots of thinking
about the world, it would just randomly muse about things in a
disjointed (and perhaps autic) way, never guiding itself to do anythig
in particular.
To make a system do something organized, you would have to give it goals
and motivations. These would have to be designed: you could not build
a "thinking part" and then leave it to come up with motivations of its
own. This is a common science fiction error: it is always assumed that
the thinking part would develop its own mitivations. Not so: it has to
have some motivations built into it. What happens when we imagine
science fiction robots is that we automatically insert the same
motivation set as is found in human beings, without realising that this
is a choice, not something that comes as part and parcel, along with
pure intelligence.
The $64,000 question then becomes what *kind* of motivations we give it.
I have discussed that before, and it does not directly bear on your
question, so I'll stop here. Okay, I'll stop after this paragraph ;-).
I believe that we will eventually have to getting very sophisticated
about how we design the motivational/emotional system (because this is a
very primitive aspect of AI at the moment), and that when we do, we will
realise that it is going to be very much easier to build a simple and
benign motivational system than to build a malevolent one (because the
latter will be unstable), and as a result of this the first AGI systems
will be benevolent. After that, the first systems will supply all the
other systems, and ensure (peacefully, and with grace) that no systems
are built that have malevolent motivations. Because of this, I believe
that we will quickly get onto an "upward spiral" toward a state in which
int is impossible for these systems to become anything other than
benevolent. This is extremely counterintuitive, of course, but only
because 100% of our experience in this world has been with intelligent
systems that have a particular (and particularly violent) set of
motivations. We need to explore this question in depth, because it is
fantastically important for the viability of the singularity idea.
Alas, at the moment there is no sign of rational discussion of this
issue, because as soon as the idea is mentioned, people come rusing
forward with nightmare scenarios, and appeal to people's gut instincts
and raw fears. (And worst of all, the Singualrity Institute for
Artificial Intelligence (SIAI) is dominated by people who have invested
their egos in a view of the world in which the only way to guarantee the
safety of AI systems is through their own mathematical proofs.)
Hope that helps, but please ask questions if it does not.
Richard Loosemore.
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?member_id=4007604&id_secret=56650779-cde055
