RE: [agi] Symbols in search of meaning - what is the meaning of B31-58-DFT?
Ben, OK - so Novamente has a system for handling 'importance' already and there is an importance updating function that feeds back to other aspects of Attention Value. That's good in terms of Novamente having an internal architecture capable of supporting and ethical system. You're asking the AGI to solve the inverse problem: Find the concept that is consistent with these descriptions and associations, and then embody that concept in your own behavior. I think this is a very hard learning problem which presumably means that it will be put off until the AGI has the capacity to undertake the leaning process. So why is this a problem? and the AGI might will come up with something subtly but dangerously twisted I don't trust this 1/1000 at much as experience-based learning. But it's not either/or - under the approach that I've suggested, Novamente would have an itch to learn about certain ethical concepts AND it would gain experience-based learning - so if experience-based learning is so good why wouldn't it help Novamente to handle it internal itch-driven learning without the subtle but dangerous twisting that you fear? And anyway why would your pure experience-based learning approach be any less likely to lead to subtly but dangerously warped ethical systems? The trainers could make errors and a Novamente's self-learning could be skewed by the limits of its experience and the modelling it observes. H. I am not certain, but I don't have a good feeling about it. I think it's fine to stimulate things related to compassion and inhibit things opposed to it, but, I think this will be useful as a *guide* to an experience-based learning process, not as a primary means of stimulating the development of a sense of compassion. Your approach to ethics seems to be based almost 100% on learning and you seem to thing that your own team will be training all the Novamentes before they leave the sandbox. How can you guarantee that your team will always be the trainers and the quality standards will always be maintained? For example, why couldn't someone outside your group get a copy of a Novamente and just strip out the learned data and then retrain the new copy of Novamente themselves? --- Getting back to your basic preferred concept of using expience-based learning to build a Novamente's ethical beliefs - this means that every Novamente has to start as a tabula rasa and in effect learn all the lessons of evolution all by itself. With anything less intelligent than a human-equivalent Novamente this would be a highly inefficient approach. But with something as intelligent as human-equivalent Novamente this a hugely dangerous strategy. Given that ethics were not hard wired into early animals - you have to ask why this hard wiring eventually emerged. My guess is that as animals became more powerful and potentially dangerous to their own kind it was only the ones with inbuilt ethics that could be activated soon after birth that were safe enough to survive and pass on their genes. In other word the lesson of evolution was that evolutionary recapitulation could not be relied on to get each animal to a point where it was safe for its fellows. Just as an aside, it seems that autism is a condition caused by problems with a human's pre-wired empathy system. According to your preferred approach to GI training it should only be a matter of training human GI in ethics and empathy. Why then does autism exist as a problem since 99% of autistic kids are put through a major training program by parents and others to get them to relate socially? I simply can't see why a Novamente that is without a modicum of ethical hardwiring will not end up being autisitic - no matter how good the training program you might give it. Why will your Novamentes not be autistic - despite the training regime that you intend? At this stage in the discussion on the AGI list I haven't heard anything to convinve me that a certain amount of ethical pre-wiring is certain to cause problems that are any greater than the problems that could be caused by NOT having a modicum of carefully designed ethical hardwiring. You have said many times that we need to suck it an see through experiment - that the theory of AGI psychological development is too underdeveloped because we don't know what we are dealing with. So why not proceed to develop Novamente's down two different paths simultaneously - the path you have already designed - where experience- based learning is virtually the only strategy, and a variant where some Novamentes have a modicum of carefully designed pre-wiring for ethics? Then you've got some experiential basis for comparing the two proposed strategies - and quick corrective action will be easier if one or other strategy shows signs of running into problems. And less
RE: [agi] Symbols in search of meaning - what is the meaning of B31-58-DFT?
Ben, I don't have a good argument on this point, just an intuition, based on the fact that generally speaking in narrow AI, inductive learning based rules based on a very broad range of experience, are much more robust than expert-encoded rules. The key is a broad range of experience, otherwise inductive learning can indeed lead to rules that are overfit to their training situations and don't generalize well to fundamentally novel situations. I've played around with expert systems years ago (I designed one to interpret a legal framework I was working on) and I'm familiar with the notion of inductive learning - using computers to generate algorithms representing patterns in large data sets. And I can see why the fuzzier system might be more robust in the face of partial novely. But I'm not proposing that AGIs rely only on pre-wired ethical drivers - a major program of experience-based learning would also be needed - just as you are planning. And in any case I didn't propose that the modicum of hard-wiring take the form a deductive 'expert system'-style rule-base. That would be very inflexible as the sole basis for ethical judgement formation (and in any case the AGI itself would be capable of developing very good deductive rule-bases and inductive expert system 'rule' bases without the need for these to be preloaded). If there need to be multiple Novamentes (not clear -- one might be enough), they could be produced through cloning rather than raising each one from scratch. Ok - I hadn't thought of cloning as a way to avoid having to directly train every Novamente. But the idea of having just one Novamente seems somewhat unrealistic and quite risky to me. If the Novamente design is going to enable boostraping as you plan then your one Novamente is going to end up being very powerful. If you try to be the gatekeeper to this one powerful AGI then (a) the rest of the world will end up considering your organisation as worse than Microsoft and many of your clients are not going to want to be held to ranson by being dependent on your one AGI for their mission critical work and (b) the one super-Novamente might develop ideas if it own that might not include you or anyone else being the gatekeeper. The idea of one super-Novamente is also dangerous because this one AGI will develop its own perspecitive on things and given its growing power that perpective or bias could become very dangerous for any one or anything that didn't fit in with that perspective. I think an AGI needs other AGIs to relate to as a community so that a community of leaning develops with multiple perspectives available. This I think is the only way that the accelerating bootstraping of AGIs can be handled with any possibility of being safe. The engineering/teaching of ethics in an AI system is pretty different from its evolution in natural systems... Of course. But that is not to say that there is nothing to be learned from evolution about the value of building in ethics in creatures that are very intelligent and very powerful. You didn't respond to one part of my last message: Philip: So why not proceed to develop Novamentes down two different paths simultaneously - the path you have already designed - where experience-based learning is virtually the only strategy, and a variant where some Novamentes have a modicum of carefully designed pre-wiring for ethics. (coupled with a major program of experience-based learning)? On reflection I can well imagine that you are not ready to make any commitment to my suggestion to give the dual (simultaneous) development path approach a go. But would you be prepared to explore the possibility of dual (simultaneous) development path approach? I think there would be much to be learned from at least examining the dual approach prior to making any commitment. What do you think? Cheers, Philip
RE: [agi] Symbols in search of meaning - what is the meaning of B31-58-DFT?
* But the idea of having just one Novamente seems somewhat unrealistic and quite risky to me. If the Novamente design is going to enable boostraping as you plan then your one Novamente is going to end up being very powerful. If you try to be the gatekeeper to this one powerful AGI then (a) the rest of the world will end up considering your organisation as worse than Microsoft and many of your clients are not going to want to be held to ranson by being dependent on your one AGI for their mission critical work and (b) the one super-Novamente might develop ideas if it own that might not include you or anyone else being the gatekeeper. The idea of one super-Novamente is also dangerous because this one AGI will develop its own perspecitive on things and given its growing power that perpective or bias could become very dangerous for any one or anything that didn't fit in with that perspective. I think an AGI needs other AGIs to relate to as a community so that a community of leaning develops with multiple perspectives available. This I think is the only way that the accelerating bootstraping of AGIs can be handled with any possibility of being safe. ** That feels to me like a lot of anthropomorphizing... Clearly there are going to be a fair number of commercial partial-Novamente software systems in use before we finish the real uber-Novamente But, I don't see why there necessarily has to be more than one Novamente taught to be a true AGI. To me, it's an unanswered question whether it'sa better use of, say,10^5 computers to make them all one Novamente, or to partition them into a society of Novamente's Philip: So why not proceed to develop Novamentes down two different paths simultaneously - the path you have already designed - where experience-based learning is virtually the only strategy, and a variant where some Novamentes have a modicum of carefully designed pre-wiring for ethics. (coupled with a major program of experience-based learning)? On reflection I can well imagine that you are not ready to make any commitment to my suggestion to give the dual (simultaneous) development path approach a go. But would you be prepared to explore the possibility of dual (simultaneous) development path approach? I think there would be much to be learned from at least examining the dual approach prior to making any commitment. What do you think? *** I guess I'm accustomed to working in a limited-resources situation, where you just have to make an intuitive call as to the best way to do something and then go with it ... and then try the next way on the list, if one's first way didn't work... Of course, if there are a lot of resources available, one can explore parallel paths simultaneously and do more of a breadth-first rather than a depth-first search through design space ! -- Ben G
RE: [agi] Symbols in search of meaning - what is the meaning of B31-58-DFT?
Ben, Philip: I think an AGI needs other AGIs to relate to as a community so that a community of learning develops with multiple perspectives available. This I think is the only way that the accelerating bootstraping of AGIs can be handled with any possibility of being safe. ** Ben: That feels to me like a lot of anthropomorphizing... Why? Why would humans be the only super-intelligent GI to have perspectives or points of view? I would have thought it was inevitable for any resource limited/experience limited GI system. And any AGI in the real world is going to be resource and experience limited. To me, it's an unanswered question whether it's a better use of, say, 10^5 computers to make them all one Novamente, or to partition them into a society of Novamente's This was the argument that raged over mainframe vs mini/PC computers. The question is only partly technical - there are many other issues that will determine the outcome. If for no other reason, the monopolies regulators are probably not going to allow all the work requiring an AGI to go through one company. Also users of AGI services are not going to want to have to deal with a monopolist - most big companies will want to have at the very least least 2-3 AGI service companies in the market place.And its unlikely that these service companies are going to want to have to buy all their AGI grunt from just one company. Even in the CPU market there's still AMD serving up a bit of competition to Intel. And Windows isn't the only OS in the market. And then there's the wider community - if there are going to be AGIs at all will the community rest easier if they think there is just one super AGI?? What do people think of Oracle's plan to have one big government database? In any case it's clearly not safe to have just one AGI in existance - if the one AGI goes feral the rest of us are going to need to access the power of some pretty powerful AGIs to contain/manage the feral one. Humans have the advantage of numbers but in the end we may not have the intellectual power or speed to counter an AGI that is actively setting out to threaten humans. Philip: So why not proceed to develop Novamentes down two different paths simultaneously - the path you have already designed - where experience-based learning is virtually the only strategy, and a variant where some Novamentes have a modicum of carefully designed pre-wiring for ethics. (coupled with a major program of experience-based learning)? Ben: I guess I'm accustomed to working in a limited-resources situation, where you just have to make an intuitive call as to the best way to do something and then go with it ... and then try the next way on the list, if one's first way didn't work... Of course, if there are a lot of resources available, one can explore parallel paths simultaneously and do more of a breadth-first rather than a depth-first search through design space ! There is at least one other option that you haven't mentioned and that is to take longer to create the AGI via the 100% experience-based learning route so you can free some resources to devote to following the 'hard-wiring plus experiential learning' route as well. It's not going to be the end of the world if we take a little longer to create a safe AGI but it could be the end of the line for all humans or at least those humans not allied with the AGI if we get a feral or dangerous AGI by mistake. And maybe by pursuing both routes simulaneously you might generate more goodwill that might increase the resourcing levels a bit further down the track. Cheers, Philip
RE: [agi] Symbols in search of meaning - what is the meaning of B31-58-DFT?
Philip, What would help me to understand this idea would be to understand in more detail what kinds of rules you want to hardwire. Do you want to hardwire, for instance, a rule like "Don'tkill people." And then give it rough rule-based definitions of "don't", "kill" and "people", and count on learning to augment/replace these definitions? Or do you want to hardwire rules at a finer-grained level? ben And maybe by pursuing both routes simulaneously you might generate more goodwill that might increase the resourcing levels a bit further down the track. *** Well, the bottom line is that the hard-wiring approach doesn't make that much intuitive sense to me. But I could be wrong, I've been wrong plenty of times before. We're going to have the Novamente book published before we have a super-smart Novamente ready. So, hopefully someone will read the book and formulate a good approach to hard-wiring ethical rules, in detail If it makes enough sense I'll be convinced that it's the way to do things I'm not closed-minded about it, i just don't see why it's a good idea yet, and I don't have enough intuition for your idea to design something in detail based on it myself... -- Ben+
RE: [agi] Symbols in search of meaning - what is the meaning of B31-58-DFT?
*** At the moment you have truth and attention values attached to nodes and links. I'm wondering whether you need to have a third numerical value type relating to 'importance'. Attention has a temporal implication - it's intended to focus significant mental resources on a key issue in the here and now. And truth values indicate the reliability of the data. Neither of these concepts capture the notion of importance. *** Actually, the AttentionValue has multiple components. One is called LongTermImportance and it controls which things are retained memory versus being paged to disk or outright deleted. For instance, if you're overseas for three years cut off from all contact with your home country, your mom's name may not be worthy of attention during that period, but you won't forget it... in Novamente this means it has a lot of LTI Clearly, ethical truths should have a lot of LTI... There is also an AV component called "importance," which is controlled by a number of factors including LTI, the recent utility of thinking about something, and the amount of recentactivity involving that thing... *** I guess the next question is, what would an AGI do with data on importance. I'm just thinking off the top of my head, but my guess is that if the nodes and links had high importance values but low truth values that this should set up an 'itch' in the system driving the AGI to engage in learning and contemplation that would lift the truth values. Maybe the higher the dissonance between the importance values and the truth values, the more this would stimulate high attention values for the related nodes and links. *** The dynamics are not as simple as you suggest, but this is the basic character of what the nonlinear Importance Updating Function does. * Let's take the hardest case first. Let's take the most arcane abstract concept that you can think of or the one that has the most intricate and complex implications/shades of meaning for living. Lets label the concept B31-58-DFT. We create a register in the AGI machinery to store important ethical concepts. We load in the label B31- 58-DFT and we give it a high importance value. We also load in a set of words in quite a few major languages into two other registers - one set of words are considered to have meaning very close to the concept that we have prelabelled as B31-58-DFT. We also load in words that are not the descriptive *meaning* of the B31-58-DFT concept but are often associated with it. We then set the truth value of B31-58-DFT to, say, zero. We also create a GoalNode associated to B31-58-DFT that indicates whether the AGI should link B31-58-DFT to its positive goal structure or to its negative goal structure ie. is B31-58-DFT more of an attractor or a repeller concept? (BTW, most likely there would need to be some system for ensuring that the urge to contemplate concept B31-58-DFT didn't get so strong that the AGI was incapable of doing anything else.) We could also load in some body-language patterns often observed in association with the concept if there are such things in this case eg. smiles on human faces, wagging tails on dogs, purring in cats, etc. (or some other pattern, eg. (1) bared teeth, growling hissing, frowns, red faces; (2) pricked ears, lifted eye brows, quite alterness; and so on). We make sure that the words we load in to the language registers include words that the AGI in the infantile stages of development might most likely associate with concept B31-58-DFT - so that the assocation between the prebuilt info about B31-58-DFT and what the AGI learns early in its life can be linked early and easily. With this structure in place the itch to understand and elaborate B31-58- DFT should be pretty strong and the AGI would probably get a real kick about thinking about B31-58-DFT. In the earliest of stages the AGI will scan its register of preloaded ethically important concepts and find that it has no learned information to understand or link the words and body language patterns to B31-58-DFT because it has not learned to recognise these words and body language patterns in the real world. ie. there is no bridge yet between what it truly has learned for itself and the preloaded linked words/body language patterns. So the itch will still be strong to learn more (from experience and taught learning) until the bridge can be created. Once one or bridges are created then the AGI should engage in accellerated self-directed learning to create more bridges to concept B31-58-DFT. From there on the AGI will probably work on several fronts - filling out the concept, working out what it means for action, improving the truth value of the concept and retuning the importance value. How would this system work if B31-58-DFT was the code for "compassion
RE: [agi] Symbols in search of meaning - what is the meaning of B31-58-DFT?
Ben, One question is whether it's enough to create general pattern-recognition functionality, and let it deal with seeking meaning for symbols as a subcase of its general behavior. Or does one need to create special heuristics/algorithms/structures just for guiding this particular process? Bit of both I think. Its a bit like there's a search for 'meaning' and a search for 'Meaning'. I think all AGIs need to search for meaning behind patterns to be able to work out useful cause/effect webs. And when AGIs work with symbols this general 'seeking the meaning of patterns' process can be applied as the first level of contemplation. But in the ethical context I think we are after 'Meaning' where this relates to to some notion of the importance of the pattern or symbol for some significant entity - for the AGI, the AGIs mentors, other sentient beings and other life. At the moment you have truth and attention values attached to nodes and links. I'm wondering whether you need to have a third numerical value type relating to 'importance'. Attention has a temporal implication - it's intended to focus significant mental resources on a key issue in the here and now. And truth values indicate the reliability of the data. Neither of these concepts capture the notion of importance. I guess the next question is, what would an AGI do with data on importance. I'm just thinking off the top of my head, but my guess is that if the nodes and links had high importance values but low truth values that this should set up an 'itch' in the system driving the AGI to engage in learning and contemplation that would lift the truth values. Maybe the higher the dissonance between the importance values and the truth values, the more this would stimulate high attention values for the related nodes and links. Then there's the question of what would generate the importance values. I think these values would ultimately be derived from the perceived importance values conveyed by 'significant others' for the AGI and by the AGIs own ethical goal structure. I don't think that preloading symbols and behavior models for something as complex as *ethical issues* is really going to be possible. I think ethical issues and associated behavior models are full of nuances that really need to be learned. Of course ethical issues and associated behavior models are full of nuances that really need to be learned to make much deep sense. Even NGIs like us, with presumably loads of hardwired predisposition to ethical behaviour, can spend their whole life in ethical learning and contemplation! :) So I guess the issues are (a) whether it's worth preload ethical concepts and (b) whether it's possible to do it. I'll start with (b) first and then cosider (a) (since lots of people have a pragmatic tendency not to bother about issues till the means for acting on them are available). (Please bear in mind that I'm not experienced or expert in any of the domains I'm riding rough shod over.everything I say will be intuitive generalist ideas...) Let's take the hardest case first. Let's take the most arcane abstract concept that you can think of or the one that has the most intricate and complex implications/shades of meaning for living. Lets label the concept B31-58-DFT. We create a register in the AGI machinery to store important ethical concepts. We load in the label B31- 58-DFT and we give it a high importance value. We also load in a set of words in quite a few major languages into two other registers - one set of words are considered to have meaning very close to the concept that we have prelabelled as B31-58-DFT. We also load in words that are not the descriptive *meaning* of the B31-58-DFT concept but are often associated with it. We then set the truth value of B31-58-DFT to, say, zero. We also create a GoalNode associated to B31-58-DFT that indicates whether the AGI should link B31-58-DFT to its positive goal structure or to its negative goal structure ie. is B31-58-DFT more of an attractor or a repeller concept? (BTW, most likely there would need to be some system for ensuring that the urge to contemplate concept B31-58-DFT didn't get so strong that the AGI was incapable of doing anything else.) We could also load in some body-language patterns often observed in association with the concept if there are such things in this case eg. smiles on human faces, wagging tails on dogs, purring in cats, etc. (or some other pattern, eg. (1) bared teeth, growling hissing, frowns, red faces; (2) pricked ears, lifted eye brows, quite alterness; and so on). We make sure that the words we load in to the language registers include words that the AGI in the infantile stages of development might most likely associate with concept B31-58-DFT - so that the assocation between the prebuilt info about B31-58-DFT and what the AGI learns early in its life can
