So this "hackability" is a technical question about possibility of closed-source deployment that would provide functional copies of the system but would prevent users from modifying its goal system. Is it really important? Source/technology will eventually get away, and from it any goal system can be forged.
Susceptibility to being tricked into different goal system through normal communication is not a question of strength of goal system, but a question of intelligence of system, so that it will be able to recognize the intent of such forged communication and refuse to act on it. On 10/2/07, Mark Waser <[EMAIL PROTECTED]> wrote: > Interesting. I believe that we have a fundamental disagreement. I > would argue that the semantics *don't* have to be distributed. My > argument/proof would be that I believe that *anything* can be described in > words -- and that I believe that previous narrow AI are brittle because they > don't have both a) closure over the terms that they use and b) the ability > to learn the meaning if *any* new term (traits that I believe that humans > have -- and I'm not sure at all that the "intelligent" part of humans have > distributed semantics). Of course, I'm also pretty sure that my belief is > in the minority on this list as well. > > I believe that an English system with closure and learning *is* going to > be a complex system and can be grounded (via the closure and interaction > with the real world). And scalable looks less problematic to me with > symbols than without. > > We may be different enough in (hopefully educated) opinions that this > e-mail may not allow for a response other than "We shall see" but I would be > interested, if you would, in hearing more as to why you believe that > semantics *must* be distributed (though I will immediately concede that it > will make them less hackable). > > Mark > > ----- Original Message ----- > From: "Richard Loosemore" <[EMAIL PROTECTED]> > To: <[email protected]> > Sent: Monday, October 01, 2007 8:36 PM > Subject: **SPAM** Re: [agi] Religion-free technical content > > > > Mark Waser wrote: > >>> And apart from the global differences between the two types of AGI, it > >>> would be no good to try to guarantee friendliness using the kind of > >>> conventional AI system that is Novamente, because inasmuch as general > >>> goals would be encoded in such a system, they are explicitly coded as > >>> "statement" which are then interpreted by something else. To put it > >>> crudely (and oversimplify slightly) if the goal "Be empathic to the > >>> needs of human beings" were represented just like that, as some kind of > >>> proposition, and stored at a particular location, it wouldn't take much > >>> for a hacker to get inside and change the statement to "Make [hacker's > >>> name] rich and sacrifice as much of humanity as necessary". If that > >>> were to become the AGI's top level goal, we would then be in deep > >>> doodoo. In the system I propose, such events could not happen. > >> > >> I think that this focuses on the wrong aspect. It is not the fact that > >> the goal is explicitly encoded as a statement that is a problem -- it is > >> the fact that it is in only one place that is dangerous. My assumption > >> is that your system basically build it's base constraints from a huge > >> number of examples and that it is distributed enough that that it would > >> be difficult if not impossible to maliciously change enough to cause a > >> problem. The fact that you're envisioning your system as not having > >> easy-to-read statements is really orthogonal to your argument and a > >> system that explicitly codes all of it's constraints as readable > >> statements but still builds it's base constraints from a huge number of > >> examples should be virtually as incorruptible as your system (with the > >> difference being security by obscurity -- which is not a good thing to > >> rely upon and also means that your system is less comprehensible). > > > > Mark, > > > > You have put your finger on one aspect of the proposal that came up, in a > > slightly different way, when Jef Allbright started talking about > > pragmatics: the "semantics" of the system. This is the hardest feature > > to explain in a short space. > > > > I really did consciously mean to have both things, not just distributed > > representation of the constraints, but also the fact that the semantics of > > the system is distributed. This distributed, semi-opaque semantics is > > what I meant by talking about the propositions not being explicitly > > encoded, above, and what I also was referring to in my comment to Jef. > > > > If the basic knowledge units ("atoms") of the system develop as a result > > of learning mechanisms + real world interaction (which together make them > > grounded), then the meaning of any given atom is encoded in the whole web > > of connections between it and the other atoms, and also by the mechanisms > > that browse on (/use, /modify) these atoms. It is not easy to point to an > > atom and say exactly what it does. > > > > This is not an optional part of the framework: it is crucial. It is the > > main reason why the system has some complexity. It is also the reason why > > the system can be properly grounded and is scalable (which is what, with > > an ordinary, conventional AI system, cannot be done because of the complex > > systems problem). > > > > In a sense the system is less comprehensible, but this is only a matter of > > degree. I don't think it makes any practical difference to our attempts > > to govern its behavior. It is going to be comprehensible enouigh that we > > can put hooks in for monitoring purposes. > > > > The great benefit of this way of doing things is that, once the system has > > matured to adulthood, it cannot be hacked: you cannot just write a worm > > to go around hunting for constraints and modifying them in a regular way > > (as you might be able to do with ordinary distributed constraints, where > > the semantics of each individual atom is well defined enough that you can > > make a clean edit), because if you tried to do this you would destabilize > > the whole thing and turn it into a gibbering wreck. It would stop working > > ... and the effect would be so dramatic that we (and it) could easily set > > up automatic shutdown mechanisms to intervene in such a case. > > > > > > > > > > Richard Loosemore > > > > > > > > ----- > > This list is sponsored by AGIRI: http://www.agiri.org/email > > To unsubscribe or change your options, please go to: > > http://v2.listbox.com/member/?&; > > > > > ----- > This list is sponsored by AGIRI: http://www.agiri.org/email > To unsubscribe or change your options, please go to: > http://v2.listbox.com/member/?&; > -- Vladimir Nesov mailto:[EMAIL PROTECTED] ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244&id_secret=48736327-159eb6
