Vladimir Nesov wrote:
So this "hackability" is a technical question about possibility of
closed-source deployment that would provide functional copies of the
system but would prevent users from modifying its goal system. Is it
really important? Source/technology will eventually get away, and from
it any goal system can be forged.
Susceptibility to being tricked into different goal system through
normal communication is not a question of strength of goal system, but
a question of intelligence of system, so that it will be able to
recognize the intent of such forged communication and refuse to act on
it.
No, that wasn't the scenario I was talking about.
After a very early stage the only person that would be able to tamper
with an AGI would be another AGI, not a human being. Since the AGIs are
all built to be friendly, the chance of one of them deciding to tamper
with some other, or with its own code, would be neglibible (as I have
already argued).
BUT, to assure people that they are even more secure than *that* we have
to consider scenarios whose origins I cannot even imagine.
So, would an AGI be able to quietly tamper with an AGI to change its
goal system? That is the main situation I addressed with the hacking
comment: could it make just a few changes, and as a result give one AGI
a completely different motivation system? The answer to that would be
no, because the distributed nature of the semantics and the goal system
would mean that a new goal system would require doing a whole bunch of
experiments, building a fresh AGI and educating it from scratch. That
would be a huge undertaking, and would be noticed by the other
(watching) AGIs.
Your suggestion above had more to do with an extremely early stage in
the development of AGI, and I do not think there will be a stage when
copies of the software are just floating around, waiting for people to
reverse engineer them and modify them.
Richard Loosemore
On 10/2/07, Mark Waser <[EMAIL PROTECTED]> wrote:
Interesting. I believe that we have a fundamental disagreement. I
would argue that the semantics *don't* have to be distributed. My
argument/proof would be that I believe that *anything* can be described in
words -- and that I believe that previous narrow AI are brittle because they
don't have both a) closure over the terms that they use and b) the ability
to learn the meaning if *any* new term (traits that I believe that humans
have -- and I'm not sure at all that the "intelligent" part of humans have
distributed semantics). Of course, I'm also pretty sure that my belief is
in the minority on this list as well.
I believe that an English system with closure and learning *is* going to
be a complex system and can be grounded (via the closure and interaction
with the real world). And scalable looks less problematic to me with
symbols than without.
We may be different enough in (hopefully educated) opinions that this
e-mail may not allow for a response other than "We shall see" but I would be
interested, if you would, in hearing more as to why you believe that
semantics *must* be distributed (though I will immediately concede that it
will make them less hackable).
Mark
----- Original Message -----
From: "Richard Loosemore" <[EMAIL PROTECTED]>
To: <agi@v2.listbox.com>
Sent: Monday, October 01, 2007 8:36 PM
Subject: **SPAM** Re: [agi] Religion-free technical content
Mark Waser wrote:
And apart from the global differences between the two types of AGI, it
would be no good to try to guarantee friendliness using the kind of
conventional AI system that is Novamente, because inasmuch as general
goals would be encoded in such a system, they are explicitly coded as
"statement" which are then interpreted by something else. To put it
crudely (and oversimplify slightly) if the goal "Be empathic to the
needs of human beings" were represented just like that, as some kind of
proposition, and stored at a particular location, it wouldn't take much
for a hacker to get inside and change the statement to "Make [hacker's
name] rich and sacrifice as much of humanity as necessary". If that
were to become the AGI's top level goal, we would then be in deep
doodoo. In the system I propose, such events could not happen.
I think that this focuses on the wrong aspect. It is not the fact that
the goal is explicitly encoded as a statement that is a problem -- it is
the fact that it is in only one place that is dangerous. My assumption
is that your system basically build it's base constraints from a huge
number of examples and that it is distributed enough that that it would
be difficult if not impossible to maliciously change enough to cause a
problem. The fact that you're envisioning your system as not having
easy-to-read statements is really orthogonal to your argument and a
system that explicitly codes all of it's constraints as readable
statements but still builds it's base constraints from a huge number of
examples should be virtually as incorruptible as your system (with the
difference being security by obscurity -- which is not a good thing to
rely upon and also means that your system is less comprehensible).
Mark,
You have put your finger on one aspect of the proposal that came up, in a
slightly different way, when Jef Allbright started talking about
pragmatics: the "semantics" of the system. This is the hardest feature
to explain in a short space.
I really did consciously mean to have both things, not just distributed
representation of the constraints, but also the fact that the semantics of
the system is distributed. This distributed, semi-opaque semantics is
what I meant by talking about the propositions not being explicitly
encoded, above, and what I also was referring to in my comment to Jef.
If the basic knowledge units ("atoms") of the system develop as a result
of learning mechanisms + real world interaction (which together make them
grounded), then the meaning of any given atom is encoded in the whole web
of connections between it and the other atoms, and also by the mechanisms
that browse on (/use, /modify) these atoms. It is not easy to point to an
atom and say exactly what it does.
This is not an optional part of the framework: it is crucial. It is the
main reason why the system has some complexity. It is also the reason why
the system can be properly grounded and is scalable (which is what, with
an ordinary, conventional AI system, cannot be done because of the complex
systems problem).
In a sense the system is less comprehensible, but this is only a matter of
degree. I don't think it makes any practical difference to our attempts
to govern its behavior. It is going to be comprehensible enouigh that we
can put hooks in for monitoring purposes.
The great benefit of this way of doing things is that, once the system has
matured to adulthood, it cannot be hacked: you cannot just write a worm
to go around hunting for constraints and modifying them in a regular way
(as you might be able to do with ordinary distributed constraints, where
the semantics of each individual atom is well defined enough that you can
make a clean edit), because if you tried to do this you would destabilize
the whole thing and turn it into a gibbering wreck. It would stop working
... and the effect would be so dramatic that we (and it) could easily set
up automatic shutdown mechanisms to intervene in such a case.
Richard Loosemore
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?&;
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?&;
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?member_id=8660244&id_secret=48865640-5a2450
