[agi] Re: Motivational Systems of an AI
Matt, I have to argue against most of your conclusions (though some do make sense). On 12/2/06, Matt Mahoney [EMAIL PROTECTED] wrote: I argue this has to be the case because an intelligent system cannot be allowed to modify its motivational system. Disagree, if motivational system means the CONTENTS of goals, motives, and drives. See the following. Our most fundamental models of intelligent agents require this (e.g. AIXI -- the reward signal is computed by the environment). That to me is a major problem of AIXI. I would rather say that we interpret some of our experience of the environment as reward signals --- they are not objective, and our interpretation can be wrong. You cannot turn off hunger or pain. To a degree we can --- by shifting attention to other things. You cannot control your emotions. I can, to a degree. ;-) Since the synaptic weights cannot be altered by training (classical or operant conditioning), they must be hardwired as determined by your DNA. We don't know that as a fact. Even if it is, I don't think it means that much for AI. Do you agree? If not, what part of this argument do you disagree with? So I don't. Let's see. That reward and punishment exist and result in learning in humans? Of course reward and punishment exist, but not as a given. Instead, these signals are produced by the system itself, based on experience. The difference from your opinion is that certain experience may be interpreted differently in different context, with respect to its nature (reward or punishment), strength, and object (to which operation or goal). That there are neurons dedicated to computing reinforcement signals? I don't know, and don't care too much. That the human motivational system (by which I mean the logic of computing the reinforcement signals from sensory input) is not trainable? Of course the system must have innate mechanism for motivation, but we need to be careful about what we mean by logic. In the case of NARS, all initial goals are given to the system (via its interaction with the environment), but all derived goals depend on the system's beliefs, which comes from the system's own experience. The derivation rules and functions are not trainable, but the contents and strength of goals and motives are trainable. In general, the system often needs to learn whether a signal is a reward. That the motivational system is completely specified by DNA? Again, it depends on what you mean by motivational system --- the mechanism is innate, but the content is not. That all human learning can be reduced to classical and operant conditioning? In principle and at a very general level, I agree, but it won't help us in designing AGI. As soon as the restriction of knowledge and resources are taken into consideration, we cannot treat all learning as conditioning. One concrete reason is that we cannot assume the availability of reliable and immediate feedback on each decision/behavior. That humans are animals that differ only in the ability to learn language? More than that. Even the ability to learn language may not be a independent capability, but is based on other capabilities. That models of goal seeking agents like AIXI are realistic models of intelligence? Yes for goal seeking agents, but no for like AIXI. To me, AIXI is an interesting idealized model of intelligence, but by no means realistic. Do you object to behavioralism because of their view that consciousness and free will do not exist, except as beliefs? We don't need to go that far in this discussion. Do you object to the assertion that the brain is a computer with finite memory and speed? That your life consists of running a program? Is this wrong, or just uncomfortable? That is right in principle, but not useful for the design of AGI, similar to assertions like An intelligent system consists of atoms. Pei - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
[agi] Re: Motivational Systems of an AI
Matt Mahoney wrote: --- Richard Loosemore [EMAIL PROTECTED] wrote: I am disputing the very idea that monkeys (or rats or pigeons or humans) have a part of the brain which generates the reward/punishment signal for operant conditioning. This is behaviorism. I find myself completely at a loss to know where to start, if I have to explain what is wrong with behaviorism. Call it what you want. I am arguing that there are parts of the brain (e.g. the nucleus accumbens) responsible for reinforcement learning, and furthermore, that the synapses along the input paths to these regions are not trainable. I argue this has to be the case because an intelligent system cannot be allowed to modify its motivational system. Our most fundamental models of intelligent agents require this (e.g. AIXI -- the reward signal is computed by the environment). You cannot turn off hunger or pain. You cannot control your emotions. Since the synaptic weights cannot be altered by training (classical or operant conditioning), they must be hardwired as determined by your DNA. Pei has already spoken eloquently on many of these questions. Do you agree? If not, what part of this argument do you disagree with? That reward and punishment exist and result in learning in humans? Reward and punishment at what level? Your use of behaviorist phrasing implies that you mean one particular interpretation of these terms, but there are others. If it is the former, then the terms are so incoherent as mechanisms that there is no answer: there simply is nothing as crude as behaviorist style reward and punishment going on. As an idea bout mechanism it is bankrupt. If you mean it at some other level, or if you mean the terms to be interpreted so generally that they could mean, for example, that there are mechanisms responsible for relaxation pressures that go in a particular direction, then of course they result in learning. That there are neurons dedicated to computing reinforcement signals? Similar answer to the previous. Reinforcement signals could mean just about anything, but if you mean in the behaviorist sense, then there is no such thing as reinforcement learning going on. And to understand *that* statement (the one I just made) you meed to understand a long story about why behaviorism is wrong. That the human motivational system (by which I mean the logic of computing the reinforcement signals from sensory input) is not trainable? Now you are asking a question based on terms that (see above) are either ambiguous or incoherent. If I back off from you interpretation of the motivational system, I can answer that the latter is probably a complicated entity, with many components, so some parts of it are trainable, and some others are not. That the motivational system is completely specified by DNA? This is a meaningless question. Do you mean *directly* specified by the DNA? Or do you mean that the DNA specifies a generator that builds the motivational system? Or that the DNA specifies a generator that eventually builds the motivational system, after morphing through several intermediate mechanisms? Are you allowing or excluding the interaction of the generators with the environment when they build the motivational system? In all except the first case, the DNA only specifies things indirectly, so the phrase completely specified by DNA is ambivalent at best. That all human learning can be reduced to classical and operant conditioning? Of course I am disputing this. This is the behaviorist idea that has been completely rejected by the cognitive science community since 1956. If you are willing to bend the meaning of the terms classical and operant conditioning sufficiently far from their origins, you might be able to make the idea more plausible, but that kind of redefinition is a little silly, and I don't see you trying to do that. That humans are animals that differ only in the ability to learn language? Do I disagree with this? Of course. Humans are not moluscs that talk. That models of goal seeking agents like AIXI are realistic models of intelligence? AIXI is not a model of a goal seeking agent, it is a mathematical abstraction of a goal seeking agent. Of course it has no value as a realistic model of intelligence. Do you object to behavioralism because of their view that consciousness and free will do not exist, except as beliefs? I assume you mean behaviorism. My objection to behaviorism has nothing to do with any of their claims about free will or consciousness. I happen to think that their opinions on such matters were generally incoherent (though not entirely so), but they were in good company on that one, so no matter. Do you object to the assertion that the brain is a computer with finite memory and speed? That your life consists of running a program? Is this wrong, or just uncomfortable? Well, I'm glad you ended on a
[agi] Re: Motivational Systems of an AI
J. Storrs Hall, PhD. wrote: On Friday 01 December 2006 23:42, Richard Loosemore wrote: It's a lot easier than you suppose. The system would be built in two parts: the motivational system, which would not change substantially during RSI, and the thinking part (for want of a better term), which is where you do all the improvement. For concreteness, I have called these the Utility Function and World Model in my writings on the subject... Well I am avoiding Utility Function precisely because it has a specific meaning in the context of the type of AI that I have been lambasting as the goal stack approach to motivation. A plan that says Let RSI consist of growing the WM and not the UF suffers from the problem that the sophistication of the WM's understanding soon makes the UF look crude and stupid. Human babies want food, proximity to their mothers, and are frightened of strangers. That's good for babies but a person with greater understanding and capabilities is better off (and the rest of us are better off if the person has) a more sophisticated UF as well. I don't want to take the bait on your baby-motivation analogy because I do not believe the difference between human baby and adult is the same as the difference between adult AI and even-smarter-adult-AI. Some, including myself, are of the opinion that there is a threshold of sentience above which things settle down a lot, so the AI would never look back on its earlier motivational system and call it crude and stupid. Also, implicit in your description of the UF and WM are some ideas that I have been explicitly avoiding in my discussion of diffuse motivational systems. That would make some of your points not applicable. No time to spell it out right now. If you look back at the root of this thread you might see why, or you can wait until I get the thing written up properly. It is not quite a contradiction, but certainly this would be impossible: deciding to make a modification that clearly was going to leave it wanting something that, if it wanted that thing today, would contradict its current priorities. Do you see why? The motivational mechanism IS what the system wants, it is not what the system is considering wanting. This is a good first cut at the problem, and is taken by e.g. Nick Bostrom in a widely cited paper at http://www.nickbostrom.com/ethics/ai.html Well, Nick Bostrum is not the origin of this idea: it is kind of obvious. The system is not protecting current beliefs, it is believing its current beliefs. Becoming more capable of understanding the reality it is immersed in? You have implicitly put a motivational priority in your system when you suggest that that is important to it ... does that rank higher than its empathy with the human race? You see where I am going: there is nothing god-given about the desire to understand reality in a better way. That is just one more candidate for a motivational priority. Ah, but consider: knowing more about how the world works is often a valuable asset to the attempt to increase the utility of the world, *no matter* what else the utility function might specify. Whoa: increase the utility of the world? Again, your terms do not map onto a viewpoint of motivation that dumps the idea of a crude UF. In essence, you have restated the idea that I was attacking: that increase the utility of the world is a motivation that trumps others. It is not necessarily the case that this is the system's primary motivation. Thus, a system's self-modification (or evolution in general) is unlikely to remove curiosity / thirst for knowledge / desire to improve one's WM as a high utility even as it changes other things. Yes and no. I am going to have to get back to you on this. Here is an idea to try to fit into that worldview. After the Singularity, I would love to go into a closed domain in which I get to live in a replica of 17th century England, growing up there from childhood with my memories put on ice for the duration of a (then normal) lifetime, and with the goal of experiencing what it would have been like to be a Natural Philosopher discovering the wonder of science for the first time. I want to discover things that are known in this era, after temporarily removing them from my mind. So I would be what? Contradicting my utility function by deliberately removing knowledge? Seeking to do what? Get the knowledge back a different way? Am I seeking knowledge, or just seeking a new experience? I claim the latter: but that idea of seeking new experience just does not map onto the kind of silly :-) utility functions that AI people play games with today. They cannot even represent the goal of having interesting subjective experiences, as far as I can see. Richard Loosemore There are several such properties of a utility function that are likely to be invariant under self-improvement or evolution. It is by
Re: [agi] Re: Motivational Systems of an AI
--- Richard Loosemore [EMAIL PROTECTED] wrote: Matt Mahoney wrote: --- Richard Loosemore [EMAIL PROTECTED] wrote: I am disputing the very idea that monkeys (or rats or pigeons or humans) have a part of the brain which generates the reward/punishment signal for operant conditioning. This is behaviorism. I find myself completely at a loss to know where to start, if I have to explain what is wrong with behaviorism. Call it what you want. I am arguing that there are parts of the brain (e.g. the nucleus accumbens) responsible for reinforcement learning, and furthermore, that the synapses along the input paths to these regions are not trainable. I argue this has to be the case because an intelligent system cannot be allowed to modify its motivational system. Our most fundamental models of intelligent agents require this (e.g. AIXI -- the reward signal is computed by the environment). You cannot turn off hunger or pain. You cannot control your emotions. Since the synaptic weights cannot be altered by training (classical or operant conditioning), they must be hardwired as determined by your DNA. Pei has already spoken eloquently on many of these questions. Yes, and I agree with most of his comments. I need to clarify that the part of the motivational system that is not trainable is the one that computes top level goals such as hunger, thirst, pain, the need for sleep, reproductive drive, etc. I think we can agree on this. Regardless of training, everyone will get hungry if they don't eat. You can temporarily distract yourself from hunger, but a healthy person can't change this top level goal. If this were not true, obesity would not be such a problem, and instead you would see a lot of people starving themselves to death. I think the confusion is over learned secondary goals, such as seeking money to buy food, or education to get a better job. So in that context, I agree with most of your comments too. That all human learning can be reduced to classical and operant conditioning? Of course I am disputing this. This is the behaviorist idea that has been completely rejected by the cognitive science community since 1956. If you are willing to bend the meaning of the terms classical and operant conditioning sufficiently far from their origins, you might be able to make the idea more plausible, but that kind of redefinition is a little silly, and I don't see you trying to do that. How about if I call them supervised and unsupervised learning? Of course this is not helpful. What I am trying to do is understand how learning works in humans so it can be modeled in AGI. Classical conditioning (e.g. Pavlov) has a simple model proposed by Hebb in 1949. If neuron A fires followed by B after time t, then the weight from A to B is increased in proportion to AB/t (where A and B are activation levels). The dependence on A and B has been used in neural models long before synaptic weight changes were observed in animal brains. The factor 1/t (for t greater than a few hundred milliseconds) is supported by animal experiments. The model for reinforcement learning is not so clear. We can imagine several possibilities. 1. The weights of a neural network are randomly and temporarily varied. After a positive reinforcement, the changes become permanent. If negative, the changes are undone or made in the opposite direction. 2. The neuron activation level of B is varied by adding random noise, dB. After reinforcment r after time t, the weight change from A to B is proportional to A(dB)r/t. 3. There is no noise. Let dB be the rate of increase of B. The weight change is proportional to A(dB)r/t. 4. (as pointed out by Philip Goetz) http://www.iro.umontreal.ca/~lisa/pointeurs/RivestNIPS2004.pdf The weight change is proportional to AB(r-p), where p is the predicted reinforcement (trained by classical conditioning) and r is the actual reinforcement (tri-Hebbian model). And many other possibilities. We don't know what the brain uses. It might be a combination of these. From animal experiments we know that the learning rate is proportional to r/t, but not much else. From computer simulations, we know there is no best solution because it depends on the problem. So I would like to see an answer to this question. How does it work in the brain? How should it be done in AGI? -- Matt Mahoney, [EMAIL PROTECTED] - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
