Hank Conn wrote:
[snip...]
> I'm not asserting any specific AI design. And I don't see how > a motivational system based on "large numbers of diffuse constrains" > inherently prohibits RSI, or really has any relevance to this. "A > motivation system based on large numbers of diffuse constraints" does > not, by itself, solve the problem- if the particular constraints do not > form a congruent mapping to the concerns of humanity, regardless of > their number or level of diffuseness, then we are likely facing an > Unfriendly outcome of the Singularity, at some point in the future.Richard Loosemore wrote: The point I am heading towards, in all of this, is that we need to unpack some of these ideas in great detail in order to come to sensible conclusions. I think the best way would be in a full length paper, although I did talk about some of that detail in my recent lengthy post on motivational systems. Let me try to bring out just one point, so you can see where I am going when I suggest it needs much more detail. In the above, you really are asserting one specific AI design, because you talk about the goal stack as if this could be so simple that the programmer would be able to insert the "make paperclips" goal and the machine would go right ahead and do that. That type of AI design is very, very different from the Motivational System AI that I discussed before (the one with the diffuse set of constraints driving it). Here is one of many differences between the two approaches. The goal-stack AI might very well turn out simply not to be a workable design at all! I really do mean that: it won't become intelligent enough to be a threat. Specifically, we may find that the kind of system that drives itself using only a goal stack never makes it up to full human level intelligence because it simply cannot do the kind of general, broad-spectrum learning that a Motivational System AI would do. Why? Many reasons, but one is that the system could never learn autonomously from a low level of knowledge *because* it is using goals that are articulated using the system's own knowledge base. Put simply, when the system is in its child phase it cannot have the goal "acquire new knowledge" because it cannot understand the meaning of the words "acquire" or "new" or "knowledge"! It isn't due to learn those words until it becomes more mature (develops more mature concepts), so how can it put "acquire new knowledge" on its goal stack and then unpack that goal into subgoals, etc? Try the same question with any goal that the system might have when it is in its infancy, and you'll see what I mean. The whole concept of a system driven only by a goal stack with statements that resolve on its knowledge base is that it needs to be already very intelligent before it can use them.If your system is intelligent, it has some goal(s) (or "motivation(s)"). For most really complex goals (or motivations), RSI is an extremely useful subgoal (sub-...motivation). This makes no further assumptions about the intelligence in question, including those relating to the design of the goal (motivation) system. Would you agree? -hank
Recursive Self Inmprovement? The answer is yes, but with some qualifications.In general RSI would be useful to the system IF it were done in such a way as to preserve its existing motivational priorities.
That means: the system would *not* choose to do any RSI if the RSI could not be done in such a way as to preserve its current motivational priorities: to do so would be to risk subverting its own most important desires. (Note carefully that the system itself would put this constraint on its own development, it would not have anything to do with us controlling it).
There is a bit of a problem with the term "RSI" here: to answer your question fully we might have to get more specific about what that would entail.
Finally: the usefulness of RSI would not necessarily be indefinite. The system could well get to a situation where further RSI was not particularly consistent with its goals. It could live without it.
Richard Loosemore ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
