Kevin wrote:
!!!!!!!!!!!!!!!
*************
In practice, it seems that an AGI is likely to have an
"owner" or a handful of them, who will have the kind of power you
describe. For instance, if my team should succeed in creating a true
Novamente AGI, then even if others participate in teaching the system, we will
have overriding power to make the changes we want. This goes along with
the fact that artificial minds are not initially going to be given any "legal
rights" in our society (whereas children have some legal rights, though not as
many as adults).
************
Would this overriding occur because the person carries
more weight with Novamente, or would they need to go in and altar the
structure\links\nodes directly to affect the change?
!!!!!!!!!!!!!
Either case could occur. In Novamente, it is possible to assign
default "confidence levels" to information sources, so one could actually tell
the system to assign more confidence to information from certain
individuals. However, there is a lot of flexibility in the design, so
the system could definitely evolve into a configuration where it worked around
these default confidence levels and decided NOT to assign more confidence to
what its teachers told it.
"Going in and altering the structure/links/nodes directly" isn't always
difficult, it may just mean loading a script containing some new (or
reweighted) nodes and links.
!!!!!!!!!!!
*********************************
At least two
questions come up then, right?
1) Depending on the AGI architecture, enforcing one's opinion on the AGI
may be very easy or very difficult. [In Novamente, I guess it will be
"moderately difficult"]
***********************************
That's the crux of the matter
isn't it? Wouldn't it be easy to enforce an opinion while Novamente is
in its formative stages, versus when a large foundation of knowledge is in
place?
!!!!!!!!!!
Yes, that's correct.
**********************************
!!!!!!
Suppose I am overtaken by greed, and I happen to get my hands
on a baby Novamente. I teach it that it should listen to me above
others. I also teach it that it is very desirable for me to have alot of
money. Novamente begins to form goal nodes geared towards fulfilling my
desire for wealth. I direct it to spread itself on the internet, and
determine ways to make me money, preferably without detection. Perhaps
it could manipulate markets, I don't know. Or perhaps it could crack
into electronic accounts and transfer the money to yours truly.
What's to stop\prevent this? In a real sci fi scenario, perhaps for
your next book, could we have NOvamentes "fighting" Novamente's?
!!!!!!!!
There is nothing in the Novamente architecture preventing this kind of
unfortunate occurence. This has to do with the particular system of
goals, beliefs and habits inside a given Novamente system, rather than with
the AI architecture itself.
!!!!!
This all goes to my concern regarding morality. I know you
resist the idea of hard coding morality into the Novamentes for various
reasons. Perhaps as an alternative, the first Novamente could be trained
over a period of time with a strong basis of moral rules(not encoded, but
trained). Then any new Novamentes would be trained by that Novamente
before being released to the public domain, making it nearly impossible for
the new Novamentes to be taught otherwise.
!!!!!!
This is something close to what we have planned.
Several others have asked me about this, and I have promised to write a
systematic (probably brief) document on Novamente Friendliness sometime in
early 2003, shortly after finishing my work on the current draft of the
Novamente book.
!!!
I know some of this stuff is a bit out there, but shouldn't we
be considering this stuff now instead of later??
!!!
It definitely needs to be thought about very hard before Novamente
reaches chimp-level intelligence. And in fact I *have* thought about it
pretty hard, though I haven't written up my thoughts much (as I've prioritized
writing up the actual design, which is taking longer than I'd hoped as it's so
damn big...).
Right now Novamente is just a software core plus a bunch of
modules-being-tested-but-not-yet-integrated, running on top of the core.
So we have a whole bunch of coding and (mostly) testing and tuning to do
before we have a system with animal-level intelligence. Admittedly,
though, if our design is right, the transition from animal-level to
human-level intelligence will be a matter of getting more machines and doing
more parameter-tuning, it won't require introduction of significant new code
or ideas.
Having said that I've thought about and will write about it, however, I
have a big caveat...
My strong feeling is that any theorizing we do about AI morality in
advance, is probably going to go out the window once we have a chimp-level AGI
to experiment with. the important thing is that we go into that phase of
experimentation with the right attitude -- with a realization that training
the system for morality is as important as training it for intelligence -- and
with a careful approach.
-- Ben
