Re: [agi] Moore's law data - defining HEC
Ilkka Tuomi questions the existence, speed, and regularity of Moore's Law: http://firstmonday.org/issues/issue7_11/tuomi/index.html SL4 discussion of memory bandwidth (not speed) as the limiting factor in human-equivalent computing: http://sl4.org/archive/0104/1063.html http://www.google.com/search?q=+site:sl4.org+crossover+bandwidth -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Friendliness toward humans
maitri wrote: I agree with your ultimate objective, the big question is *how* to do it. What is clear is that no one has any idea that seems to be guaranteed to work in creating an AGI with these qualities. We are currently resigned to let's build it and see what happens. Which is quite scary for some, although not me(this is subject to change) Not everyone, maitri. Though it does seem, sadly, to be a popular sentiment. I would also strongly caution you against use of the word guarantee, as under many theories of rationality, including mine, there is simply no such thing as a justified confidence of 1.0 - not even in mathematical truths. Your phrasing suggests that if Friendliness cannot be absolutely guaranteed, then one might as well go ahead and build an AI with no theory of Friendliness at all. Anyway, it seems to be the implication of your post that nobody is working out a detailed theory in advance. This is not correct. Friendly AI was originally coined in a 900KB publication on the subject that laid out fundamental architectural considerations and tried to describe at least of the content and development strategies. I haven't yet written up the last one-and-a-half year's worth of enormous improvements on that original document, but I've gone on developing the model. With luck, by the time anyone needs it, the theory will be there. Of course, realizing that you need a theory is a separate problem. Around all I can do on that score is hope that anyone who hasn't gotten far enough theoretically to realize this also won't get very far on AGI implementation. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] The Metamorphosis of Prime Intellect
Eliezer S. Yudkowsky wrote: There may be additional rationalization mechanisms I haven't identified yet which are needed to explain anosognosia and similar disorders. Mechanism (4) is the only one deep enough to explain why, for example, the left hemisphere automatically and unconsciously rationalizes the actions of the left hemisphere; and mechanism (4) doesn't necessarily explain that, it only looks like it might someday do so. That is, the left hemisphere automatically and unconsciously rationalizes the actions of the right hemisphere in split-brain patients. Sorry. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] The Metamorphosis of Prime Intellect
Ben Goertzel wrote: This is exactly why I keep trying to emphasize that we all should forsake those endlessly fascinating, instinctively attractive political arguments over our favorite moralities, and instead focus on the much harder problem of defining an AI architecture which can understand that its morality is wrong in various ways; wrong definitions, wrong reinforcement procedures, wrong source code, wrong Friendliness architecture, wrong definition of wrongness, and many others. These are nontrivial problems! Each turns out to require nonobvious structural qualities in the architecture of the goal system. Hmmm. It seems to me the ability to recognize one's own potential wrongness comes along automatically with general intelligence... Ben, I've been there, 1996-2000, and that turned out to be the WRONG ANSWER. There's an enormous amount of moral complexity that does *not* come along with asymptotically increasing intelligence. Thankfully, despite the tremendous emotional energy I put into believing that superintelligences are inevitably moral, and despite the amount of published reasoning I had staked on it, I managed to spot this mistake before I pulled a Lawrence on the human species. Please, please, please don't continue where I left off. The problem here is the imprecision of words. *One* form of wrongness, such as factual error, or wrong source code which is wrong because it is inefficient or introduces factual errors, is readily conceivable by a general intelligence without extra moral complexity. You do, indeed, get recognition of *that particular* kind of wrongness for free. It does not follow that all the things we recognize as wrong, in moral domains especially, can be recognized by a general intelligence without extra moral complexity. If it is the case that a general intelligence necessarily has the ability to conceive of a wrongness in a top-level goal definition and has a mechanism for correcting it, this is not obvious to me - not for any definition of wrongness at all. Prime Intellect, with its total inability to ask any moral question except how desirable is X, under the Three Laws as presently defined, seems to me quite realistic. Note also that the ability to identify *a* kind of wrongness, does not necessarily mean the ability to see - as a human would - the specific wrongness of your own programmer standing by and screaming That's not what I meant! Stop! Stop! If this realization is a necessary ability of all minds-in-general it is certainly not clear why. Recognizing wrong source code requires a codic modality, of course, and recognizing wrong Friendliness architecture requires an intellectual knowledge of philosophy and software design. What is there about recognizing one's wrongness in the ways you mention, that doesn't come for free with general cognition and appropriate perception? So... you think a real-life Prime Intellect would have, for free, recognized that it should not lock Lawrence out? But why? I guess there is an attitude needed to recognize one's own wrongness: a lack of egoistic self-defensive certainty in one's own correctness A skeptical attitude even about one's own most deeply-held beliefs. In Novamente, this skeptical attitude has two aspects: 1) very high level schemata that must be taught not programmed 2) some basic parameter settings that will statistically tend to incline the system toward skepticism of its own conclusions [but you can't turn the dial too far in the skeptical direction either...] That's for skepticism about facts. I agree you get that for free with general intelligence. If *all* questions of morality, means and ends and ultimate goals, were reducible to facts and deducible by logic or observation, then the issue would end right there. That was my position 1996-2000. Is this your current position? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] C-T Thesis (or a version thereof) - Is it useable as anin-principle argument for strong AI?
The C-T Thesis is an argument for Strong AI because it requires that any opponent of Strong AI posit a nonstandard model of physics to allow for noncomputable processes which, through a nonstandard neurological mechanism, have a noticeable macroscopic effect on brain processes. This is not a knockdown argument but it is a strong one; only Penrose and Hameroff have had the courage to face it down openly - postulate, and search for, both the new physics and the new neurology required. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Cosmodelia's posts: Music and Artificial Intelligence;Jane;
[EMAIL PROTECTED] wrote: I propose that *any* successful AGI design will be a design, instead of in reason, *in the technology of spirit*. Now, by spirit, I don't mean the Let's go pray to lunar crystals snake oil-kind. I mean a real, all-encompasing, perspective on what it means to be. No doubt many AI'ers will claim spirit is irrelevant. No doubt some will claim that spirit is an emergent property of general intelligence. But I'm inclined (largely for rational reasons and partly for intuition) to disagree. Regarding the first point, I don't believe that spirit is irrelevant. Rather, I believe it is essential to intelligence. You can't have AGI without human-GI's ability to percieve spirit. Regarding the second point, I really doubt that spirit could be an emergent property of AGI. How can the whole emerge from the part unless the part already contains it? The problem, as I'm sure you realized, is that you can't go up to an AGI researcher and say: Build me an AI that includes spirit. Well, you can take that physical action. But if you say it one of the usual run of AGI researchers, one of two things will happen: 1) The AGI researcher will say: Spirit is poorly defined, not realizing that humans start out with an extremely poorly defined intuitive view of things like intelligence, spirit, emotion, reality, truth, et cetera, and that it is a necessary part of the job of creating AI to create clearly defined naturalistic views that completely, clearly, and satisfyingly encompass all these realms. 2) The AGI researcher will invent a spur-of-the moment definition for spirit which doesn't really match what you're interested in - it doesn't provide a detailed naturalistic view which, when you see it, is fully and satisfyingly identifiable with what you were interested in. But the AGI researcher will insist that the definition must embrace spirit for some reason or other, just as earlier AGI researchers insisted that search trees, three-layer neural networks, semantic nets, frames, agents, et cetera, carried within them all the complexity of the mind. What you have is an intuition that something has been left undone in traditional models of AGI. You don't know what's missing - you just know that it is. This intuition tends to be missing in most AGI researchers. If they had the ability to tell when their theories were missing something critical they would not be launching AGI projects based on incomplete theories. There is a selection effect at work; people who know they don't understand the mind don't become AI researchers. But nonetheless, although you know something has been left undone, you don't really know *what* has been left undone. You can't describe it in enough detail to create it; if you could do that you would be a (true) AGI creator yourself. All you can do is insist that something is missing. And it is a sad fact that you will not get very far with this insistence - even though you are, for the record, correct. I think I understand what you're calling spirit. I think I understand it well enough to deliberately transfer it from humans, where it exists now, into another kind of rational-moral empirical regularity called a Friendly AI. I would say, from within that understanding, that you are correct in that spirit is not emergent... very little is, really. Emergence in AGI is mostly an excuse for not understanding things. If you don't understand something, you can't create it, and you certainly can't know in advance that it will emerge. In the physical natural sciences, where emergence really is the most useful paradigm for understanding most phenomena, you would be laughed out of the lecture hall if you showed a picture of the Sun's corona and insisted that you'd finally got the corona all figured out: It's an emergent phenomenon of the Sun! But of course this explanation tells us nothing about *how* the corona emerges from the Sun and whether we're likely to see an analogous phenomenon appear in a glass of water or a Bose-Einstein condensate. All that emergence says is that the explanation doesn't invoke an optimization process such as cognitive design or evolutionary adaptation. (Or to be specific, to say that A is naturally emergent from B is to say that given B, one does not need to postulate further information, or settings of physical variables, deriving from a cognitive or evolutionary optimization process, for A to result.) Spirit isn't emergent, and isn't everywhere, and isn't a figment of the imagination, and isn't supernatural. Spirit refers to a real thing, with a real explanation; it's just that the explanation is very, very difficult. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member
Re: [agi] AGI morality
Ben Goertzel wrote: However, it's to be expected that an AGI's ethics will be different than any human's ethics, even if closely related. What do a Goertzelian AGI's ethics and a human's ethics have in common that makes it a humanly ethical act to construct a Goertzelian AGI? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] AGI morality - goals and reinforcement values
Bill Hibbard wrote: On Wed, 12 Feb 2003, Philip Sutton wrote: Ben/Bill, My feeling is that goals and ethics are not identical concepts. And I would think that goals would only make an intentional ethical contribution if they related to the empathetic consideration of others. Absolutely goals (I prefer the word values) and ethics are not identical. Values are a means to express ethics. Words goin' in circles... in my account there's morality, metamorality, ethics, goals, subgoals, supergoals, child goals, parent goals, desirability, ethical heuristics, moral ethical heuristics, metamoral ethical heuristics, and honor. Roughly speaking you could consider ethics as describing regularities in subgoals, morality as describing regularities in supergoals, and metamorality as defining the computational pattern to which the current goal system is a successive approximation and which the current philosophy is an interim step in computing. In all these cases I am overriding existing terminology to serve as a term of art. In discussions like these, common usage is simply not adequate to define what the words mean. (Those who find my definitions inadequate can find substantially more thorough definitions in Creating Friendly AI.) -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] unFriendly AIXI
Eliezer S. Yudkowsky wrote: I recently read through Marcus Hutter's AIXI paper, and while Marcus Hutter has done valuable work on a formal definition of intelligence, it is not a solution of Friendliness (nor do I have any reason to believe Marcus Hutter intended it as one). In fact, as one who specializes in AI morality, I was immediately struck by two obvious-seeming conclusions on reading Marcus Hutter's formal definition of intelligence: 1) There is a class of physically realizable problems, which humans can solve easily for maximum reward, but which - as far as I can tell - AIXI cannot solve even in principle; 2) While an AIXI-tl of limited physical and cognitive capabilities might serve as a useful tool, AIXI is unFriendly and cannot be made Friendly regardless of *any* pattern of reinforcement delivered during childhood. Before I post further, is there *anyone* who sees this besides me? Also, let me make clear why I'm asking this. AIXI and AIXI-tl are formal definitions; they are *provably* unFriendly. There is no margin for handwaving about future revisions of the system, emergent properties of the system, and so on. A physically realized AIXI or AIXI-tl will, provably, appear to be compliant up until the point where it reaches a certain level of intelligence, then take actions which wipe out the human species as a side effect. The most critical theoretical problems in Friendliness are nonobvious, silent, catastrophic, and not inherently fun for humans to argue about; they tend to be structural properties of a computational process rather than anything analogous to human moral disputes. If you are working on any AGI project that you believe has the potential for real intelligence, you are obliged to develop professional competence in spotting these kinds of problems. AIXI is a formally complete definition, with no margin for handwaving about future revisions. If you can spot catastrophic problems in AI morality you should be able to spot the problem in AIXI. Period. If you cannot *in advance* see the problem as it exists in the formally complete definition of AIXI, then there is no reason anyone should believe you if you afterward claim that your system won't behave like AIXI due to unspecified future features. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] unFriendly AIXI
Ben Goertzel wrote: AIXI and AIXItl are systems that are designed to operate with an initial fixed goal. As defined, they don't modify the overall goal they try to achieve, they just try to achieve this fixed goal as well as possible through adaptively determining their actions. Basically, at each time step, AIXI searches through the space of all programs to find the program that, based on its experience, will best fulfill its given goal. It then lets this best program run and determine its next action. Based on that next action, it has a new program space search program... etc. AIXItl does the same thing but it restricts the search to a finite space of programs, hence it's a computationally possible (but totally impractical) algorithm. The harmfulness or benevolence of an AIXI system is therefore closely tied to the definition of the goal that is given to the system in advance. Actually, Ben, AIXI and AIXI-tl are both formal systems; there is no internal component in that formal system corresponding to a goal definition, only an algorithm that humans use to determine when and how hard they will press the reward button. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] unFriendly AIXI
more. The question is whether Hutter's adaptive reality-and-reward algorithm encapsulates the behaviors you want... do you think it does? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] unFriendly AIXI
Ben Goertzel wrote: Huh. We may not be on the same page. Using: http://www.idsia.ch/~marcus/ai/aixigentle.pdf Page 5: The general framework for AI might be viewed as the design and study of intelligent agents [RN95]. An agent is a cybernetic system with some internal state, which acts with output yk on some environment in cycle k, perceives some input xk from the environment and updates its internal state. Then the next cycle follows. We split the input xk into a regular part x0k and a reward rk, often called reinforcement feedback. From time to time the environment provides non-zero reward to the agent. The task of the agent is to maximize its utility, defined as the sum of future rewards. I didn't see any reward function V defined for AIXI in any of the Hutter papers I read, nor is it at all clear how such a V could be defined, given that the internal representation of reality produced by Solomonoff induction is not fixed enough for any reward function to operate on it in the same way that, e.g., our emotions bind to our own standardized cognitive representations. Quite literally, we are not on the same page ;) Thought so... Look at page 23, Definition 10 of the intelligence ordering relation (which says what it means for one system to be more intelligent than another). And look at the start of Section 4.1, which Definition 10 lives within. The reward function V is defined there, basically as cumulative reward over a period of time. It's used all thru Section 4.1, and following that, it's used mostly implicitly inside the intelligence ordering relation. The reward function V however is *not* part of AIXI's structure; it is rather a test *applied to* AIXI from outside as part of Hutter's optimality proof. AIXI itself is not given V; it induces V via Solomonoff induction on past rewards. V can be at least as flexible as any criterion a (computable) human uses to determine when and how hard to press the reward button, nor is AIXI's approximation of V fixed at the start. Given this, would you regard AIXI as formally approximating the kind of goal learning that Novamente is supposed to do? As Definition 10 makes clear, intelligence is defined relative to a fixed reward function. A fixed reward function *outside* AIXI, so that the intelligence of AIXI can be defined relative to it... or am I wrong? What the theorems about AIXItl state is that, given a fixed reward function, the AIXItl can do as well as any other algorithm at achieving this reward function, if you give it computational resources equal to those that the other algorithm got, plus a constant. But the constant is fucking HUGE. Actually, I think AIXItl is supposed to do as well as a tl-bounded algorithm given t2^l resources... though again perhaps I am wrong. Whether you specify the fixed reward function in its cumulative version or not doesn't really matter... Actually, AIXI's fixed horizon looks to me like it could give rise to some strange behaviors, but I think Hutter's already aware that this is probably AIXI's weakest link. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] unFriendly AIXI
observer. Would it be fair to say that AIXI's spontaneous behavior is formally superior to Novamente's spontaneous behavior? Yeah, AIXI is formally superior if one distinguishes any fixed goal and asks whether Novamente or AIXI can better achieve that goal. But so what? AIXI assumes you have infinite computing power!! If I assumed infinite computing power, I would have designed Novamente rather differently... and much more simply... As a side point, I'm not sure the best way to compare systems is to assume a fixed formal goal and ask who can achieve it better. This is the way Hutter's theorems do the comparison, but... But no matter HOW you want to compare systems, if you let me assume infinite computing power, I can design a system that will outperform a Novamente ... I'm not trying to compare amounts of computing power but fundamental *kinds* of intelligence, as is AIXI's purpose as a formal definition. Again, AIXI as a formal system has no goal definition. [Note: I may be wrong about this; Ben Goertzel and I seem to have acquired different models of AIXI and it is very possible that mine is the wrong one.] Well, the purpose of AIXI and AIXItl is to have theorems proved about them. These theorems are of the form: Given a fixed reward function (a fixed goal), That's an interesting paraphrase. * AIXI is maximally intelligent at achieving the goal * AIXItl is as intelligent as any other finite-resource program at achieving the goal, so long as AIXItl is given C more computing power than the other program, where C is very big But you are right that AIXI and AIXItl could also be run without a fixed reward function /goal. No, that's not what I said. I said AIXI could be run with only a reward channel and no separate input channel, or with a reward predicate that is a deterministic function of the input channel. You cannot run AIXI in the absence of a reward channel. In that case you cannot prove any of Hutter's theorems about them. And if you can't prove theorems about them then they are nothing more than useless abstractions. Since AIXI can never be implemented and AIXItl is so inefficient it could never do anything useful in practice. But they are very useful tools for talking about fundamental kinds of intelligence. If the humans see that AIXI seems to be dangerously inclined toward just proving math theorems, they might decide to press the reward button when AIXI provides cures for cancer, or otherwise helps people. AIXI would then modify its combined reality-and-reward representation accordingly to embrace the new simplest explanation that accounted for *all* the data, i.e., its reward function would then have to account for mathematical theorems *and* cancer cures *and* any other kind of help that humans had, in the past, pressed the reward button for. Would you say this is roughly analogous to the kind of learning you intend Novamente to perform? Or perhaps even an ideal form of such learning? Well, sure ... it's *roughly analogous*, in the sense that it's experiential reinforcement learning, sure. Is it roughly analogous, but not really analogous, in the sense that Novamente can do something AIXI can't? Self-modification in any form completely breaks Hutter's definition, and you no longer have an AIXI any more. The question is whether Hutter's adaptive reality-and-reward algorithm encapsulates the behaviors you want... do you think it does? Not really. There is certainly a significant similarity between Hutter's stuff and the foundations of Novamente, but there are significant differences too. To sort out the exact relationship would take me more than a few minutes' thought. There are indeed major differences in the foundations. Is there something useful or important that Novamente does, given its foundations, that you could not do if you had a physically realized infinitely powerful computer running Hutter's stuff? One major difference, as I mentioned above, is that Hutter's systems are purely concerned with goal-satisfaction, whereas Novamente is not entirely driven by goal-satisfaction. Is this reflected in a useful or important behavior of Novamente, in its intelligence or the way it interacts with humans, that is not possible to AIXI? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] unFriendly AIXI
Eliezer S. Yudkowsky wrote: Not really. There is certainly a significant similarity between Hutter's stuff and the foundations of Novamente, but there are significant differences too. To sort out the exact relationship would take me more than a few minutes' thought. There are indeed major differences in the foundations. Is there something useful or important that Novamente does, given its foundations, that you could not do if you had a physically realized infinitely powerful computer running Hutter's stuff? Actually, you said that it would take you more than a few minutes thought to sort it all out, so let me ask a question which you can hopefully answer more quickly... Do you *feel intuitively* that there is something useful or important Novamente does, given its foundations, that you could not do if you had a physically realized AIXI? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] unFriendly AIXI
Bill Hibbard wrote: On Tue, 11 Feb 2003, Ben Goertzel wrote: Eliezer wrote: Interesting you should mention that. I recently read through Marcus Hutter's AIXI paper, and while Marcus Hutter has done valuable work on a formal definition of intelligence, it is not a solution of Friendliness (nor do I have any reason to believe Marcus Hutter intended it as one). In fact, as one who specializes in AI morality, I was immediately struck by two obvious-seeming conclusions on reading Marcus Hutter's formal definition of intelligence: 1) There is a class of physically realizable problems, which humans can solve easily for maximum reward, but which - as far as I can tell - AIXI cannot solve even in principle; I don't see this, nor do I believe it... I don't believe it either. Is this a reference to Penrose's argument based on Goedel's Incompleteness Theorem (which is wrong)? Oh, well, in that case, I'll make my statement more formal: There exists a physically realizable, humanly understandable challenge C on which a tl-bounded human outperforms AIXI-tl for humanly understandable reasons. Or even more formally, there exists a computable process P which, given either a tl-bounded uploaded human or an AIXI-tl, supplies the uploaded human with a greater reward as the result of strategically superior actions taken by the uploaded human. :) -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] unFriendly AIXI
Ben Goertzel wrote: Oh, well, in that case, I'll make my statement more formal: There exists a physically realizable, humanly understandable challenge C on which a tl-bounded human outperforms AIXI-tl for humanly understandable reasons. Or even more formally, there exists a computable process P which, given either a tl-bounded uploaded human or an AIXI-tl, supplies the uploaded human with a greater reward as the result of strategically superior actions taken by the uploaded human. :) -- Eliezer S. Yudkowsky Hmmm. Are you saying that given a specific reward function and a specific environment, the t1-bounded uploaded human with resources (t,l) will act so as to maximize the reward function better than AIXI-tl with resources (T,l) with T as specified by Hutter's theorem of AIXI-tl optimality? Presumably you're not saying that, because it would contradict his theorem? Indeed. I would never presume to contradict Hutter's theorem. So what clever loophole are you invoking?? ;-) An intuitively fair, physically realizable challenge with important real-world analogues, solvable by the use of rational cognitive reasoning inaccessible to AIXI-tl, with success strictly defined by reward (not a Friendliness-related issue). It wouldn't be interesting otherwise. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] unFriendly AIXI
Ben Goertzel wrote: It's right as mathematics... I don't think his definition of intelligence is the maximally useful one, though I think it's a reasonably OK one. I have proposed a different but related definition of intelligence, before, and have not been entirely satisfied with my own definition, either. I like mine better than Hutter's... but I have not proved any cool theorems about mine... Can Hutter's AIXI satisfy your definition? If Novamente can do something AIXI cannot, then Hutter's work is very highly valuable because it provides a benchmark against which this becomes clear. If you intuitively feel that Novamente has something AIXI doesn't, then Hutter's work is very highly valuable whether your feeling proves correct or not, because it's by comparing Novamente against AIXI that you'll learn what this valuable thing really *is*. This holds true whether the answer turns out to be It's capability X that I didn't previously really know how to build, and hence didn't see as obviously lacking in AIXI or It's capability X that I didn't previously really know how to build, and hence didn't see as obviously emerging from AIXI. So do you still feel that Hutter's work tells you nothing of any use? Well, it hasn't so far. It may in the future. If it does I'll say so ;-) The thing is, I (like many others) thought of algorithms equivalent to AIXI years ago, and dismissed them as useless. What I didn't do is prove anything about these algorithms, I just thought of them and ignored them Partly because I didn't see how to prove the theorems, and partly because I thought even once I proved the theorems, I wouldn't have anything pragmatically useful... It's not *about* the theorems. It's about whether the assumptions **underlying** the theorems are good assumptions to use in AI work. If Novamente can outdo AIXI then AIXI's assumptions must be 'off' in some way and knowing this *explicitly*, as opposed to having a vague intuition about it, cannot help but be valuable. Again, it sounds to me like, in this message, you're taking for *granted* that AIXI and Novamente have the same theoretical foundations, and that hence the only issue is design and how much computing power is needed, in which case I can see why it would be intuitively straightforward to you that (a) Novamente is a better approach than AIXI and (b) AIXI has nothing to say to you about the pragmatic problem of designing Novamente, nor are its theorems relevant in building Novamente, etc. But that's exactly the question I'm asking you. *Do* you believe that Novamente and AIXI rest on the same foundations? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
[agi] unFriendly AIXI... and Novamente?
Ben, you and I have a long-standing disagreement on a certain issue which impacts the survival of all life on Earth. I know you're probably bored with it by now, but I hope you can understand why, given my views, I keep returning to it, and find a little tolerance for my doing so. The issue is our two differing views on the difficulty of AI morality. Your intuitions say... I am trying to summarize my impression of your viewpoint, please feel free to correct me... AI morality is a matter of experiential learning, not just for the AI, but for the programmers. To teach an AI morality you must give it the right feedback on moral questions and reinforce the right behaviors... and you must also learn *about* the deep issues of AI morality by raising a young AI. It isn't pragmatically realistic to work out elaborate theories of AI morality in advance; you must learn what you need to know as you go along. Moreover, learning what you need to know, as you go along, is a good strategy for creating a superintelligence... or at least, the rational estimate of the goodness of that strategy is sufficient to make it a good idea to try and create a superintelligence, and there aren't any realistic strategies that are better. An informal, intuitive theory of AI morality is good enough to spark experiential learning in the *programmer* that carries you all the way to the finish line. You'll learn what you need to know as you go along. The most fundamental theoretical and design challenge is making AI happen, at all; that's the really difficult part that's defeated everyone else so far. Focus on making AI happen. If you can make AI happen, you'll learn how to create moral AI from the experience. In contrast, I felt that it was a good idea to develop a theory of AI morality in advance, and have developed this theory to the point where it currently predicts, counter to my initial intuitions and to my considerable dismay: 1) AI morality is an extremely deep and nonobvious challenge which has no significant probability of going right by accident. 2) If you get the deep theory wrong, there is a strong possibility of a silent catastrophic failure: the AI appears to be learning everything just fine, and both you and the AI are apparently making all kinds of fascinating discoveries about AI morality, and everything seems to be going pretty much like your intuitions predict above, but when the AI crosses the cognitive threshold of superintelligence it takes actions which wipe out the human species as a side effect. AIXI, which is a completely defined formal system, definitely undergoes a failure of exactly this type. Ben, you need to be able to spot this. Think of it as a practice run for building a real transhuman AI. If you can't spot the critical structural property of AIXI's foundations that causes AIXI to undergo silent catastrophic failure, then a real-world reprise of that situation with Novamente would mean you don't have the deep theory to choose good foundations deliberately, you can't spot bad foundations deductively, and because the problems only show up when the AI reaches superintelligence, you won't get experiential feedback on the failure of your theory until it's too late. Exploratory research on AI morality doesn't work for AIXI - it doesn't even visibly fail. It *appears* to work until it's too late. If you don't spot the problem in advance, you lose. If I can demonstrate that your current strategy for AI development would undergo silent catastrophic failure in AIXI - that your stated strategy, practiced on AIXI, would wipe out the human species, and you didn't spot it - will you acknowledge that as a practice loss? A practice loss isn't the end of the world. I have one practice loss on my record too. But when that happened I took it seriously; I changed my behavior as a result. If you can't spot the silent failure in AIXI, would you then *please* admit that your current strategy on AI morality development is not adequate for building a transhuman AI? You don't have to halt work on Novamente, just accept that you're not ready to try and create a transhuman AI *yet*. I can spot the problem in AIXI because I have practice looking for silent failures, because I have an underlying theory that makes it immediately obvious which useful properties are formally missing from AIXI, and because I have a specific fleshed-out idea for how to create moral systems and I can see AIXI doesn't work that way. Is it really all that implausible that you'd need to reach that point before being able to create a transhuman Novamente? Is it really so implausible that AI morality is difficult enough to require at least one completely dedicated specialist? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily
[agi] Breaking AIXI-tl
Okay, let's see, I promised: An intuitively fair, physically realizable challenge, with important real-world analogues, formalizable as a computation which can be fed either a tl-bounded uploaded human or an AIXI-tl, for which the human enjoys greater success measured strictly by total reward over time, due to the superior strategy employed by that human as the result of rational reasoning of a type not accessible to AIXI-tl. Roughly speaking: A (selfish) human upload can engage in complex cooperative strategies with an exact (selfish) clone, and this ability is not accessible to AIXI-tl, since AIXI-tl itself is not tl-bounded and therefore cannot be simulated by AIXI-tl, nor does AIXI-tl have any means of abstractly representing the concept a copy of myself. Similarly, AIXI is not computable and therefore cannot be simulated by AIXI. Thus both AIXI and AIXI-tl break down in dealing with a physical environment that contains one or more copies of them. You might say that AIXI and AIXI-tl can both do anything except recognize themselves in a mirror. The simplest case is the one-shot Prisoner's Dilemna against your own exact clone. It's pretty easy to formalize this challenge as a computation that accepts either a human upload or an AIXI-tl. This obviously breaks the AIXI-tl formalism. Does it break AIXI-tl? This question is more complex than you might think. For simple problems, there's a nonobvious way for AIXI-tl to stumble onto incorrect hypotheses which imply cooperative strategies, such that these hypotheses are stable under the further evidence then received. I would expect there to be classes of complex cooperative problems in which the chaotic attractor AIXI-tl converges to is suboptimal, but I have not proved it. It is definitely true that the physical problem breaks the AIXI formalism and that a human upload can straightforwardly converge to optimal cooperative strategies based on a model of reality which is more correct than any AIXI-tl is capable of achieving. Ultimately AIXI's decision process breaks down in our physical universe because AIXI models an environmental reality with which it interacts, instead of modeling a naturalistic reality within which it is embedded. It's one of two major formal differences between AIXI's foundations and Novamente's. Unfortunately there is a third foundational difference between AIXI and a Friendly AI. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] AI Morality -- a hopeless quest
Brad Wyble wrote: Tell me this, have you ever killed an insect because it bothered you? In other words, posthumanity doesn't change the goal posts. Being human should still confer human rights, including the right not to be enslaved, eaten, etc.. But perhaps being posthuman will confer posthuman rights that we understand as little as a dog understands the right to vote. -- James Hughes -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] unFriendly AIXI... and Novamente?
deeply than I understand AIXI, and which we are focusing a lot of our time on. I know, I know, I know. The problem is that just because you really need to win the lottery, it doesn't follow that you will. And just because you really don't have the time, pragmatically speaking, to spend on figuring out certain things, doesn't make ignorance of them any less dangerous. I can't detect anything in the structure of the universe which says that genuinely tired and overworked AI researchers are exempted from the consequences imposed by bitch Nature. I was feeling really tired that day is, when you think about it, one of the most likely final epitaphs for the human species. Remember that what I'm trying to convince you is that you're not ready *yet*. I'm really tired right now, can't find time to discuss it is evidence in favor of this proposition, not against it. I'm really tired is only evidence in favor of pushing ahead anyway if you accept the Mom Logic that things you're too tired to think about can't hurt you. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Breaking AIXI-tl
Shane Legg wrote: Eliezer, Yes, this is a clever argument. This problem with AIXI has been thought up before but only appears, at least as far as I know, in material that is currently unpublished. I don't know if anybody has analysed the problem in detail as yet... but it certainly is a very interesting question to think about: What happens when two super intelligent AIXI's meet? SI-AIXI is redundant; all AIXIs are enormously far beyond superintelligent. As for the problem, the obvious answer is that no matter what strange things happen, an AIXI^2 which performs Solomonoff^2 induction, using the universal prior of strings output by first-order Oracle machines, will come up with the best possible strategy for handling it... Has the problem been thought up just in the sense of What happens when two AIXIs meet? or in the formalizable sense of Here's a computational challenge C on which a tl-bounded human upload outperforms AIXI-tl? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Reply to Bill Hubbard's post: Mon, 10 Feb 2003
C. David Noziglia wrote: The problem with the issue we are discussing here is that the worst-case scenario for handing power to unrestricted, super-capable AI entities is very bad, indeed. So what we are looking for is not really building an ethical structure or moral sense at all. Failure is not an option. The only way to prevent the worst-case scenarios that have been mentioned by discussants is not to design moral values and hope, but to build in hard-wired, Three Laws-type rules that cannot be overridden. And then, on top of that, build in social, competitive systems that use the presence of mulitple AIs, dependence on humans as suppliers or intermediaries, ethical, legal, and even game-theory (remember the movie /War Games/?) strictures, and even punishment systems up to and incuding shut-down capabilities. That *still* doesn't work. 1) Hard-wired rules are a pipe dream. It consists of mixing mechanomorphism (machines only do what they're told to do) with anthropomorphism (I wish those slaves down on the plantation would stop rebelling). The only hard-wired level of organization is code, or, in a seed AI, physics. Once cognition exists it can no longer be usefully described using the adjective hard-wired. This is like saying you can make Windows XP stable by hard-wiring it not to crash, presumably by including the precompilation statement #define BUGS 0. 2) Any individual ethic that cannot be overridden - if we are speaking about a successfully implemented design property of the system, and not a mythical hardwiring - will never be any stronger, smarter, or more reliable than the frozen goal system of its creator as it existed at the time of producing that ethic. In particular, odd things start happening when you take an intelligence of order X and try to control it using goal patterns that were produced by an intelligence of order X. You say cannot be overridden, I hear cannot be renormalized. 3) A society of selfish AIs may develop certain (not really primatelike) rules for enforcing cooperative interactions among themselves; but you cannot prove for any entropic specification, and I will undertake to *disprove* for any clear specification, that this creates any rational reason to assign a greater probability to the proposition that the AI society will protect human beings. 4) As for dependence on human suppliers, if you're talking about transhumans of any kind, AIs, uploads, what-have-you, transhumans dependent on a human economy is a pipe dream. (Order custom proteins from an online DNA synthesis and peptide sequencer; build nanotech; total time of dependence on human economy, 48 hours.) -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Reply to Bill Hubbard's post: Mon, 10 Feb 2003
the dice in creating AGI's and the best we can do is load them. I'm not providing a surefire guarantee. I'm... loading the dice is a good enough description if taken in the fully general sense of biasing possible futures. *But* to load the dice effectively, you have to actually *understand* the dice. Vague arguments about what humans do in situation XYZ are not going to work in the absence of an understanding of what the dependencies are. Or to put it another way, you see Friendliness in AIs as pretty likely regardless, and you think I'm going to all these lengths to provide a guarantee. I'm not. I'm going to all these lengths to create a *significant probability* of Friendliness. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Reply to Bill Hubbard's post: Mon, 10 Feb 2003
Brad Wyble wrote: There are simple external conditions that provoke protective tendencies in humans following chains of logic that seem entirely natural to us. Our intuition that reproducing these simple external conditions serve to provoke protective tendencies in AIs is knowably wrong, failing an unsupported specific complex miracle. Well said. Or to put it another way, you see Friendliness in AIs as pretty likely regardless, and you think I'm going to all these lengths to provide a guarantee. I'm not. I'm going to all these lengths to create a *significant probability* of Friendliness. You're mischaracterizing my position. I'm certainly not saying we'll get friendliness for free, but was trying to reason by analogy (perhaps in a flawed way), that our best chance of success may be to model AGI's based on our innate tendencies wherever possible. Human behavior is a knowable quality. Okay... what I'm saying, basically, is that to connect AI morality to human morality turns out to be a very complex problem that is not solved by saying let's copy human nature. You need a very specific description of what you have to copy, how you do the copying, and so on, and this involves all sorts of complex nonobvious concepts within a complex nonobvious theory that completely changes the way you see morality. It would even be fair to say, dismayingly, that in saying let's build AGI's which reproduce certain human behaviors, you have not even succeeded in stating the problem, let alone the solution. This isn't intended in any personal way, btw. It's just that, like, the fate of the world *does* actually depend on it and all, so I have to be very precise about how much progress has occurred at a given point of theoretical development, rather than offering encouragement. I perceived, based on the character of your discussion, that you would be unsatisfied with anything short of a formal, mathetmatical proof that any given AGI would not destroy us before giving the assent to turning it on. If that characterization was incorrect, the fault is mine. No! It's *my* fault! You can't have any! Anyhow, I don't think such a formal proof is possible. The problem with the proposals I see is not that they are not *provably* Friendly but that a rational extrapolation of them shows that they are *unFriendly* barring a miracle. I'll take a proposal whose rational extrapolation is to Friendliness and which seems to lie at a local optimum relative to the improvements I can imagine; proof is impossible. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Breaking AIXI-tl
a tl-bounded seed AI can outperform AIXI-tl on the ordinary (non-quined) problem of cooperation with a superintelligence. The environment can't ever *really* be constant and completely separated as Hutter requires. A physical environment that gives rise to an AIXI-tl is different from the environment that gives rise to a tl-bounded seed AI, and the different material implementations of these entities (Lord knows how you'd implement the AIXI-tl) will have different side effects, and so on. All real world problems break the Cartesian assumption. The questions But are there any kinds of problems for which that makes a real difference? and Does any conceivable kind of mind do any better? can both be answered affirmatively. Welll I agree with only some of this. The thing is, an AIXI-tl-driven AI embedded in the real world would have a richer environment to draw on than the impoverished data provided by PD2. This AI would eventually learn how to model itself and reflect in a rich way (by learning the right operating program). However, AIXI-tl is a horribly bad AI algorithm, so it would take a VERY VERY long time to carry out this learning, of course... Measured in computing cycles, yes. Measured in rounds of information required, no. AIXI-tl is defined to run on a very VERY fast computer. Marcus Hutter has formally proved your intutions about the requirement of a rich environment or prior training to be wrong; I am trying to show that your intuitions about what AIXI-tl is capable of learning are wrong. But to follow either Hutter's argument or my own requires mentally reproducing more of the abstract properties of AIXI-tl, given its abstract specification, than your intuitions currently seem to be providing. Do you have a non-intuitive mental simulation mode? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Breaking AIXI-tl
Bill Hibbard wrote:
On Fri, 14 Feb 2003, Eliezer S. Yudkowsky wrote:
It *could* do this but it *doesn't* do this. Its control process is such
that it follows an iterative trajectory through chaos which is forbidden
to arrive at a truthful solution, though it may converge to a stable
attractor.
This is the heart of the fallacy. Neither a human nor an AIXI
can know that his synchronized other self - whichever one
he is - is doing the same. All a human or an AIXI can know is
its observations. They can estimate but not know the intentions
of other minds.
The halting problem establishes that you can never perfectly understand
your own decision process well enough to predict its decision in advance,
because you'd have to take into account the decision process including the
prediction, et cetera, establishing an infinite regress.
However, Corbin doesn't need to know absolutely that his other self is
synchronized, nor does he need to know his other self's decision in
advance. Corbin only needs to establish a probabilistic estimate, good
enough to guide his actions, that his other self's decision is correlated
with his *after* the fact. (I.e., it's not a halting problem where you
need to predict yourself in advance; you only need to know your own
decision after the fact.)
AIXI-tl is incapable of doing this for complex cooperative problems
because its decision process only models tl-bounded things and AIXI-tl is
not *remotely close* to being tl-bounded. Humans can model minds much
closer to their own size than AIXI-tl can. Humans can recognize when
their policies, not just their actions, are reproduced. We can put
ourselves in another human's shoes imperfectly; AIXI-tl can't put itself
in another AIXI-tl's shoes to the extent of being able to recognize the
actions of an AIXI-tl computed using a process that is inherently 2t^l
large. Humans can't recognize their other selves perfectly but the gap in
the case of AIXI-tl is enormously greater. (Humans also have a reflective
control process on which they can perform inductive and deductive
generalizations and jump over a limited class of infinite regresses in
decision processes, but that's a separate issue. Suffice it to say that a
subprocess which generalizes over its own infinite regress does not
obviously suffice for AIXI-tl to generalize over the top-level infinite
regress in AIXI-tl's control process.)
Let's say that AIXI-tl takes action A in round 1, action B in round 2, and
action C in round 3, and so on up to action Z in round 26. There's no
obvious reason for the sequence {A...Z} to be predictable *even
approximately* by any of the tl-bounded processes AIXI-tl uses for
prediction. Any given action is the result of a tl-bounded policy but the
*sequence* of *different* tl-bounded policies was chosen by a t2^l process.
A human in the same situation has a mnemonic record of the sequence of
policies used to compute their strategies, and can recognize correlations
between the sequence of policies and the other agent's sequence of
actions, which can then be confirmed by directing O(other-agent) strategic
processing power at the challenge of seeing the problem from the opposite
perspective. AIXI-tl is physically incapable of doing this directly and
computationally incapable of doing it indirectly. This is not an attack
on the computability of intelligence; the human is doing something
perfectly computable which AIXI-tl does not do.
--
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
---
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] unFriendly Hibbard SIs
Bill Hibbard wrote: Hey Eliezer, my name is Hibbard, not Hubbard. *Argh* sound of hand whapping forehead sorry. On Fri, 14 Feb 2003, Eliezer S. Yudkowsky wrote: *takes deep breath* This is probably the third time you've sent a message to me over the past few months where you make some remark like this to indicate that you are talking down to me. No, that's the sound of a lone overworked person taking on yet another simultaneous conversation. But I digress. But then you seem to chicken out on the exchange. For example, this morning I pointed out the fallacy in your AIXI argument and got no reply (you were assuming that humans have some way of knowing, rather than just estimating, the intention of other minds). Okay. I shall reply to that as well. You're thinking of the logical entailment approach and the problem with that, as it appears to you, is that no simple set of built-in principles can entail everything the SI needs to know about ethics - right? Yes. Laws (logical constraints) are inevitably ambiguous. Does that include the logical constraints governing the reinforcement process itself? Like, the complexity of everything the SI needs to do is some very high quantity, while the complexity of the principles that are supposed to entail it is small, right? As wonderfully demonstrated by Eric Baum's papers, complex behaviors are learned via simple values. *Some* complex behaviors can be learned via *some* simple values. The question is understanding *which* simple values result in the learning of which complex behaviors; for example, Eric Baum's system had to be created with simple values that behave in a very precise way in order to achieve its current level of learning ability. That's why Eric Baum had to write the paper, instead of just saying Aha, I can produce complex behaviors via simple values. If SIs have behaviors that are reinforced by a set of values V, what is the internal mechanism that an SI uses to determine the amount of V? Let's say that the SI contains an internal model of the environment, which I think is what you mean by temporal credit assignment, et cetera, and the SI has some predicate P that applies to this internal model and predicts the amount of human happiness that exists. Or perhaps you weren't thinking of a system that complex; perhaps you just want a predicate P that applies to immediate sense data, like the human sense of pleasurable tastes. What is the complexity of the predicate P? I mean, I'm sure it seems very straightforward to you to determine when human happiness is occurring... There are already people developing special AI programs to recognize emotions in human facial expresssions and voices. And emotional expressions in body language shouldn't be much harder. I'm not claiming that these problems are totally solved, just that there are much easier than AGI, and can serve as reinforcement values for an AGI. The value V or predicate P for reinforcement values are immediate, and relatively simple. Reinforcement learning generates very complex behaviors from these. Credit assignment, including temporal credit assignment, is the problem of understanding cause and effect relations between multiple behaviors and future values. Yes, reinforcement learning generates very complex behaviors from there. The question is *which* complex behaviors - whether you see all the complex behaviors you want to see, and none of the complex behaviors you don't want to see. Can I take it from the above that you believe that AI morality can be created by reinforcing behaviors using a predicate P that acts on incoming video sensory information to recognize smiles and laughter and generate a reward signal? Is this adequate for a superintelligence too? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] who is this Bill Hubbard I keep reading about?
Bill Hibbard wrote: Strange that there would be someone on this list with a name so similar to mine. I apologize, dammit! I whack myself over the head with a ballpeen hammer! Now let me ask you this: Do you want to trade names? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Breaking AIXI-tl
similar enough - and it's not a symmetrical situation; AIXI-tl can't handle correlation with even the most similar possible mind. The Clone challenge is just a very extreme case of that. I'd worked out the structure for the general case previously. Encountering AIXI, I immediately saw that AIXI didn't handle the general case, then I devised the Clone challenge as the clearest illustration from a human perspective. So what happens if you have a (tabula rasa) AIXI-tl trying to impersonate a superintelligence in real life? If AIXI-tl fails to spoof a reflective SI perfectly on the first round, everyone in the universe will soon know it's an AIXI-tl or something similar, which violates Hutter's separability condition. If the environment gives rise to an AIXI-tl in a way that lets an SI reason about AIXI-tl's internals from its initial conditions, it breaks the Cartesian theatre. The interesting part is that these little natural breakages in the formalism create an inability to take part in what I think might be a fundamental SI social idiom, conducting binding negotiations by convergence to goal processes that are guaranteed to have a correlated output, which relies on (a) Bayesian-inferred initial similarity between goal systems, and (b) the ability to create a top-level reflective choice that wasn't there before, that (c) was abstracted over an infinite recursion in your top-level predictive process. But if this isn't immediately obvious to you, it doesn't seem like a top priority to try and discuss it... -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Breaking AIXI-tl
Eliezer S. Yudkowsky wrote: But if this isn't immediately obvious to you, it doesn't seem like a top priority to try and discuss it... Argh. That came out really, really wrong and I apologize for how it sounded. I'm not very good at agreeing to disagree. Must... sleep... -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Breaking AIXI-tl
Ben Goertzel wrote:
hi,
No, the challenge can be posed in a way that refers to an arbitrary agent
A which a constant challenge C accepts as input.
But the problem with saying it this way, is that the constant challenge
has to have an infinite memory capacity.
So in a sense, it's an infinite constant ;)
Infinite Turing tapes are a pretty routine assumption in operations like
these. I think Hutter's AIXI-tl is supposed to be able to handle constant
environments (as opposed to constant challenges, a significant formal
difference) that contain infinite Turing tapes. Though maybe that'd
violate separability? Come to think of it, the Clone challenge might
violate separability as well, since AIXI-tl (and hence its Clone) builds
up state.
No, the charm of the physical challenge is exactly that there exists a
physically constant cavern which defeats any AIXI-tl that walks into it,
while being tractable for wandering tl-Corbins.
No, this isn't quite right.
If the cavern is physically constant, then there must be an upper limit to
the t and l for which it can clone AIXItl's.
Hm, this doesn't strike me as a fair qualifier. One, if an AIXItl exists
in the physical universe at all, there are probably infinitely powerful
processors lying around like sunflower seeds. And two, if you apply this
same principle to any other physically realized challenge, it means that
people could start saying Oh, well, AIXItl can't handle *this* challenge
because there's an upper bound on how much computing power you're allowed
to use. If Hutter's theorem is allowed to assume infinite computing
power inside the Cartesian theatre, then the magician's castle should be
allowed to assume infinite computing power outside the Cartesian theatre.
Anyway, a constant cave with an infinite tape seems like a constant
challenge to me, and a finite cave that breaks any {AIXI-tl, tl-human}
contest up to l=googlebyte also still seems interesting, especially as
AIXI-tl is supposed to work for any tl, not just sufficiently high tl.
Well, yes, as a special case of AIXI-tl's being unable to carry out
reasoning where their internal processes are correlated with the
environment.
Agreed...
(See, it IS actually possible to convince me of something, when it's
correct; I'm actually not *hopelessly* stubborn ;)
Yes, but it takes t2^l operations.
(Sorry, you didn't deserve it, but a straight line like that only comes
along once.)
--
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
---
To unsubscribe, change your address, or temporarily deactivate your subscription,
please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Breaking AIXI-tl
Ben Goertzel wrote: In a naturalistic universe, where there is no sharp boundary between the physics of you and the physics of the rest of the world, the capability to invent new top-level internal reflective choices can be very important, pragmatically, in terms of properties of distant reality that directly correlate with your choice to your benefit, if there's any breakage at all of the Cartesian boundary - any correlation between your mindstate and the rest of the environment. Unless, you are vastly smarter than the rest of the universe. Then you can proceed like an AIXItl and there is no need for top-level internal reflective choices ;) Actually, even if you are vastly smarter than the rest of the entire universe, you may still be stuck dealing with lesser entities (though not humans; superintelligences at least) who have any information at all about your initial conditions, unless you can make top-level internal reflective choices. The chance that environmental superintelligences will cooperate with you in PD situations may depend on *their* estimate of *your* ability to generalize over the choice to defect and realize that a similar temptation exists on both sides. In other words, it takes a top-level internal reflective choice to adopt a cooperative ethic on the one-shot complex PD rather than blindly trying to predict and outwit the environment for maximum gain, which is built into the definition of AIXI-tl's control process. A superintelligence may cooperate with a comparatively small, tl-bounded AI, but be unable to cooperate with an AIXI-tl, provided there is any inferrable information about initial conditions. In one sense AIXI-tl wins; it always defects, which formally is a better choice than cooperating on the oneshot PD, regardless of what the opponent does - assuming that the environment is not correlated with your decisionmaking process. But anyone who knows that assumption is built into AIXI-tl's initial conditions will always defect against AIXI-tl. A small, tl-bounded AI that can make reflective choices has the capability of adopting a cooperative ethic; provided that both entities know or infer something about the other's initial conditions, they can arrive at a knowably correlated reflective choice to adopt cooperative ethics. AIXI-tl can learn the iterated PD, of course; just not the oneshot complex PD. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Breaking AIXI-tl
Ben Goertzel wrote: AIXI-tl can learn the iterated PD, of course; just not the oneshot complex PD. But if it's had the right prior experience, it may have an operating program that is able to deal with the oneshot complex PD... ;-) Ben, I'm not sure AIXI is capable of this. AIXI may inexorably predict the environment and then inexorably try to maximize reward given environment. The reflective realization that *your own choice* to follow that control procedure is correlated with a distant entity's choice not to cooperate with you may be beyond AIXI. If it was the iterated PD, AIXI would learn how a defection fails to maximize reward over time. But can AIXI understand, even in theory, regardless of what its internal programs simulate, that its top-level control function fails to maximize the a priori propensity of other minds with information about AIXI's internal state to cooperate with it, on the *one* shot PD? AIXI can't take the action it needs to learn the utility of... -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] doubling time revisted.
Faster computers make AI easier. They do not make Friendly AI easier in the least. Once there's enough computing power around that someone could create AI if they knew exactly what they were doing, Moore's Law is no longer your friend. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
[agi] Low-hanging fruits for true AGIs
Okay, hypothetical question... and yes, it's really hypothetical... if you had a true AGI, say with very powerful pattern-recognition intelligence but perhaps not with much in the way of natural human interaction yet, what would be the simplest and least effortful way to make money with it? Stock markets and commodities markets are obvious targets, and trading itself is relatively simple to set up through an online broker; but is there any freely available online source of data rich enough to perform prediction on, assuming that stock markets and commodities markets contain at least some AGI-recognizable, previously unrecognized, usefully predictable and reliably exploitable empirical regularities? Or does any attempt to generate money via AGI require launching at least a small specialized company to do so? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: AGI Complexity (WAS: RE: [agi] doubling time watcher.)
Ben Goertzel wrote: But of course, none of us *really know*. Technically, I believe you mean that you *think* none of us really know, but you don't *know* that none of us really know. To *know* that none of us really know, you would have to really know. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] doubling time watcher.
Brad Wyble wrote: I'm uncomfortable with the phrase Human Equivalent because I think we are very far from understanding what that phrase even means. We don't yet know the relevant computational units of brain function. It's not just spikes, it's not just EEG rhythms. I understand we'll never know for certain, but at the moment, the possibility of guesstimating within even an order of magnitude seems premature. See also Human-level software crossover date from the human crossover metathread on SL4: http://sl4.org/archive/0104/1057.html -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Breaking AIXI-tl
Wei Dai wrote: Eliezer S. Yudkowsky wrote: Important, because I strongly suspect Hofstadterian superrationality is a *lot* more ubiquitous among transhumans than among us... It's my understanding that Hofstadterian superrationality is not generally accepted within the game theory research community as a valid principle of decision making. Do you have any information to the contrary, or some other reason to think that it will be commonly used by transhumans? You yourself articulated, very precisely, the structure underlying Hofstadterian superrationality: Expected utility of a course of action is defined as the average of the utility function evaluated on each possible state of the multiverse, weighted by the probability of that state being the actual state if the course was chosen. The key precise phrasing is weighted by the probability of that state being the actual state if the course was chosen. This view of decisionmaking is applicable to a timeless universe; it provides clear recommendations in the case of, e.g., Newcomb's Paradox. The mathematical pattern of a goal system or decision may be instantiated in many distant locations simultaneously. Mathematical patterns are constant, and physical processes may produce knowably correlated outputs given knowably correlated initial conditions. For non-deterministic systems, or cases where the initial conditions are not completely known (where there exists a degree of subjective entropy in the specification of the initial conditions), the correlation estimated will be imperfect, but nonetheless nonzero. What I call the Golden Law, by analogy with the Golden Rule, states descriptively that a local decision is correlated with the decision of all mathematically similar goal processes, and states prescriptively that the utility of an action should be calculated given that the action is the output of the mathematical pattern represented by the decision process, not just the output of a particular physical system instantiating that process - that the utility of an action is the utility given that all sufficiently similar instantiations of a decision process within the multiverse do, already have, or someday will produce that action as an output. Similarity in this case is a purely descriptive argument with no prescriptive parameters. Golden decisionmaking does not imply altruism - your goal system might evaluate the utility of only your local process. The Golden Law does, however, descriptively and prescriptively produce Hofstadterian superrationality as a special case; if you are facing a sufficiently similar mind across the Prisoner's Dilemna, your decisions will be correlated and that correlation affects your local utility. Given that the output of the mathematical pattern instantiated by your physical decision process is C, the state of the multiverse is C, C; given that the output of the mathematical pattern instantiated by your physical decision process is D, the state of the multiverse is D, D. Thus, given sufficient rationality and a sufficient degree of known correlation between the two processes, the mathematical pattern that is the decision process will output C. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Breaking AIXI-tl
Wei Dai wrote: Ok, I see. I think I agree with this. I was confused by your phrase Hofstadterian superrationality because if I recall correctly, Hofstadter suggested that one should always cooperate in one-shot PD, whereas you're saying only cooperate if you have sufficient evidence that the other side is running the same decision algorithm as you are. Similarity in this case may be (formally) emergent, in the sense that a most or all plausible initial conditions for a bootstrapping superintelligence - even extremely exotic conditions like the birth of a Friendly AI - exhibit convergence to decision processes that are correlated with each other with respect to the oneshot PD. If you have sufficient evidence that the other entity is a superintelligence, that alone may be sufficient correlation. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Breaking AIXI-tl
Billy Brown wrote: Ben Goertzel wrote: I think this line of thinking makes way too many assumptions about the technologies this uber-AI might discover. It could discover a truly impenetrable shield, for example. It could project itself into an entirely different universe... It might decide we pose so little threat to it, with its shield up, that fighting with us isn't worthwhile. By opening its shield perhaps it would expose itself to .0001% chance of not getting rewarded, whereas by leaving its shield up and leaving us alone, it might have .1% chance of not getting rewarded. Now, it is certainly conceivable that the laws of physics just happen to be such that a sufficiently good technology can create a provably impenetrable defense in a short time span, using very modest resources. If that happens to be the case, the runaway AI isn't a problem. But in just about any other case we all end up dead, either because wiping out humanity now is far easier that creating a defense against our distant descendants, or because the best defensive measures the AI can think of require engineering projects that would wipe us out as a side effect. It should also be pointed out that we are describing a state of AI such that: a) it provides no conceivable benefit to humanity b) a straightforward extrapolation shows it wiping out humanity c) it requires the postulation of a specific unsupported complex miracle to prevent the AI from wiping out humanity c1) these miracles are unstable when subjected to further examination c2) the AI still provides no benefit to humanity even given the miracle When a branch of an AI extrapolation ends in such a scenario it may legitimately be labeled a complete failure. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/?[EMAIL PROTECTED]
Re: [agi] Hard Wired Switch
Ben Goertzel wrote: However, the society approach does not prevent a whole society of AGI's from drifting into evil. How good is our understanding of AGI sociodynamics??? ;-) This approach just replaces one hard problem with another... which may or may not be even harder... Indeed; if one cannot describe the moral dynamics of that physical process which looks to human eyes like one AI, how would one describe the physical process which looks to human eyes like many AI? A good theory should be able to describe the moral dynamics of both, including important empirical differences (if any). As far as I'm concerned, physically implemented morality is physically implemented morality whether it's a human, an AI, an AI society, or a human society. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Why is multiple superintelligent AGI's safer than a singleAGI?
Ben Goertzel wrote: Yes, I see your point now. If an AI has a percentage p chance of going feral, then in the case of a society of AI's, only p percent of them will go feral, and the odds are that other AI's will be able to stop it from doing anything bad. But in the case of only one AI, then there's just a p% chance of it going feral, without much to do about it... Unknowns are the odds of AI's going feral and supersmart at the same time, and the effects of society-size on the probability of ferality... But you do have a reasonable point, I'll admit, and I'll think about it more... This does not follow. If an AI has a P chance of going feral, then a society of AIs may have P chance of all simultaneously going feral - it depends on how much of the probability is independent among different AIs. Actually, for the most worrisome factors, such as theoretical flaws in the theory of AI morality, I would expect the risk factor to be almost completely shared among all AIs. Furthermore, risk factors stemming from divergent rates of self-enhancement or critical thresholds in self-enhancement may not be at all improved by multiply copying or multiply diverging AIs if AI improvement is less than perfectly synchronized. Remember that humans are evolved and trained to pay attention to variances among humans - the things we all have in common are not noticed. Humans exist in an evolved state where everyone has certain variances and societies are built around the interaction of those variances. That evolution emergently produced this kind of society following social selection pressures on individual fitness does not show that it is the best method given intelligent design, or even a good method at all. I am very suspicious of proposals to construct AI societies. What we regard as beneficial social properties are very contingent on our evolved individual designs. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
[agi] Singletons and multiplicities
BAD IDEA because it greatly increases the chance of at least one catastrophic error occurring, while not increasing by much the chance of recovering from the catastrophic error. If there is */any /*benefit in having more than one AGI around in the case where an AGI does go feral then your comment I'm just not so sure that there's _any _benefit to the society of AGI as opposed to one big AGI approach no longer holds as an absolute. Ya know, this is why (no offense, Ben) I really dislike statements like I'm just not so sure that there's any benefit to X. It doesn't encourage analysis the way that I think X's disadvantages outweigh its advantages does. If you're not so sure, you might as well try anyway, or not try, or whatever. If you phrase it as a comparison, that encourages enumeration of the advantages and disadvantages. It then gets back to having a society of AGIs might be an advantage in certain cercumstances, but having more than one AGI might have the following down sides. At this point a balanced risk/benefit assessment can be made (not definitive of course since we haven't seen super-intelligent AGIs operation yet). But at least we've got some relevant issues on the table to think about. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Why is multiple superintelligent AGI's safer than a singleAGI?
Philip Sutton wrote: Hi Eliezer, This does not follow. If an AI has a P chance of going feral, then a society of AIs may have P chance of all simultaneously going feral I can see you point but I don't agree with it. If General Motors churns out 100,000 identical cars with all the same charcteristics and potiential flaws, they will */not /*all fail at the same instant in time. Each of them will be placed in a different operating environment and the failures will probably spread over a bell curve style distribution. That's because your view of this problem has automatically factored out all the common variables. All GM cars fail when dropped off a cliff. All GM cars fail when crashed at 120 mph. All GM cars fail on the moon, in space, underwater, in a five-dimensional universe. All GM cars are, under certain circumstances, inferior to telecommuting. How much of the risk factor in AI morality is concentrated into such universals? As far as I can tell, practically all of it. Every AI morality failure I have ever spotted has been of a kind where a society of such AIs would fail in the same way. The bell-curve failures to which you refer stem from GM making a cost-performance tradeoff. The bell-curve distributed failures, like the fuel filter being clogged or whatever, are *acceptable* failures, not existential risks. It therefore makes sense to accept a probability X of failure, for component Q, which can be repaired at cost C when it fails; and when you add up all those probability factors you end up with a bell curve. But if the car absolutely had to work, you would be minimizing X like hell, to the greatest degree allowed by your *design ability and imagination*. You'd use a diamondoid fuel filter. You'd use three of them. You wouldn't design a car that had a single point of failure at the fuel filter. You would start seriously questioning whether what you really wanted should be described as a car. Which in turn would shift the most probable cause of catastrophic failure away from bell-curve probabilistic failures and into outside-context failures of imagination. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Web Consciousness and self consciousness
Ben Goertzel wrote: I see physics as a collection of patterns in the experienced world. It's a very, very powerful and intense collection of patterns. But nevertheless, it's not totally comprehensive, in the sense that there are some patterns in the experienced world that are not part of physics, but are rather complementary to physics (in the Niels Bohr sense of complementarity). Qualia, perhaps, are in this set of patterns that are complementary to the patterns comprising physics. Bohr's needlessly complex metaphysics of the quantum-classical split became unnecessary once I understood the Everett-Wheeler-DeWitt many-worlds interpretation, in which there is not and never was a classical universe, and measuring instruments are a continuous part of the wavefunction. There is no observation, only the appearance of observation from inside the continuous part of the wavefunction that happens to be yourself. The wavefunction does not collapse; you split along with the wavefunction, and each of you thinks the wavefunction has collapsed. The illusion of complementarity; the illusion of solipsism; underneath all of it is one continuous equation that does not treat observation as a special case. If one understood qualia one would have no need to invoke complementarity for them. Now, physics sees my experienced world as the product of some physical system (my brain, etc.) From this point of view, the qualia that are patterns in my experienced world are patterns generated by a physical system (my brain, etc.) Physics and experience are thus two perspectives, each of which claims to contain and generate the other! In my own work on AGI, I have drawn on both of these perspectives: both the mechanistic, scientific perspective AND the subjective, experiential perspective. In theory one could create a digital mind based purely the mechanistic, scientific perspective, but given the currently somewhat primitive state of mind science, obtaining some guidance from the experiential side (via drawing analogies between subjective human experience and the likely subjective experience of an AI) has seemed prudent. If one hasn't understood the subjective in terms of physics, then one will need to go on thinking about cognition in terms of phenomena that one finds to be subjective as well as objective. However, in my experience, learning how to create chunks of cognition requires crossing the gap between the subjective and the objective with respect to that problem. While you may be able to make incremental progress in understanding cognition by using a dual perspective to work on subproblems, I would predict that any part of your wanted AI design that you do not yet know how to describe in purely physical terms, will fail to work. That's part of what makes AI hard. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Discovering the Capacity of Human Memory
James Rogers wrote: I was wondering about that. It seems that the number represents the size of the phase space, when a more useful metric would be the size (Kolmogorov complexity) of the average point *in* the phase space. There is a world of difference between the number of patterns that can be encoded and the size of the biggest pattern that can be encoded; the former isn't terribly important, but the latter is very important. Are you talking about the average point in the phase space in the sense of an average empirical human brain, or in the sense of a randomly selected point in the phase space? I assume you mean the former, since, for the latter question, if you have a simple program P that produces a phase space of size 2^X, the average size of a random point in the phase space must be roughly X (plus the size of P?) according to both Shannon and Kolmogorov. (Incidentally, I'll join in expressing my astonishment and dismay at the level of sheer mathematical and physical and computational ignorance on the part of authors and reviewers that must have been necessary for even the abstract of this paper to make it past the peer review process, and add that the result violates the Susskind holographic bound for an object that can be contained in a 1-meter sphere - no more than 10^70 bits of information.) -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Discovering the Capacity of Human Memory
The Tao is the set of truths that can be stored in zero bits. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
[agi] HUMOR: Friendly AI Critical Failure Table
is suffering from a number of diseases and medical conditions, but they would, if informed of the AI's capabilities, suffer from an extreme fear that appearing on the AI's video cameras would result in their souls being stolen. The tribe has not currently heard of any such thing as video cameras, so their fear is extrapolated by the AI; and the tribe members would, with almost absolute certainty, eventually come to understand that video cameras are not harmful, especially since the human eye is itself essentially a camera. But it is also almost certain that, if flatly informed of the video cameras, the !Kung would suffer from extreme fear and prefer death to their presence. Meanwhile the AI is almost powerless to help them, since no bots at all can be sent into the area until the moral issue of photography is resolved. The AI wants your advice: is the humane action rendering medical assistance, despite the !Kung's (subjunctive) fear of photography? If you say Yes you are quietly, seamlessly, invisibly uploaded. 25: The AI informs you - yes, *you* - that you are the only genuinely conscious person in the world. The rest are zombies. What do you wish done with them? 26: The AI does not inflict pain, injury, or death on any human, regardless of their past sins or present behavior. To the AI's thinking, nobody ever deserves pain; pain is always a negative utility, and nothing ever flips that negative to a positive. Socially disruptive behavior is punished by tickling and extra homework. 27: The AI's user interface appears to our world in the form of a new bureaucracy. Making a wish requires mailing forms C-100, K-2210, and T-12 (along with a $25 application fee) to a P.O. Box in Minnesota, and waiting through a 30-day review period. 28: The programmers and anyone else capable of explaining subsequent events are sent into temporal stasis, or a vantage point from which they can observe but not intervene. The rest of the world remains as before, except that psychic powers, ritual magic, alchemy, et cetera, begin to operate. All role-playing gamers gain special abilities corresponding to those of their favorite character. 29: Everyone wakes up. 30: Roll twice again on this table, disregarding this result. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] HUMOR: Friendly AI Critical Failure Table
deering wrote: 31. The best of all possible worlds: the AI decides that the world, just as it is, is the best possible combination of challenge, reward, serendipity, irony, tragedy, mystery, romance, and beauty. Although the AI claims to have ultimate power and is open to any request that would improve the world, all requests are met by explanations of why everyone would be better off leaving it just the way it is. The explanations are logically consistent and convincing yet emotionally unsatisfying. That sounds like one *heck* of a critical failure. I think you'd have to roll *two* critical failures in a row before the AI was *that* badly screwed up. Maybe we should have a Double Critical Failure Table with all the screwups that are too mind-numbingly wrong to be listed next to mere ordinary critical failures like rebuilding the solar system as a hentai anime. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Bayes rule in the brain
Ben Goertzel wrote: BTW, to me, the psychological work on human bias, heuristics, and fallacy (including the well known work by Tversky and Kahneman) contains many wrong results --- the phenomena are correctly documented, but their analysis and conclusions are often based on implicit assumptions that are not justified. Yes, I agree with you there. An example? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Within-cell computation in biological neural systems??
Philip Sutton wrote: Does anyone have an up-to-date fix on how much computation occurs (if any) within-cells (as opposed to the traditional neural net level) that are part of biolgical brain systems? especially in the case of animals that have a premium placed on the number of neurones they can support (eg. limited by size, weight or energy supply compared to the need for computational capacity). Let's put it this way: The idea that neurons are simple little integrating units is an urban legend propagated by computer scientists. Flipping through my bookmarks, I found The role of single neurons in information processing by Christof Koch and Idan Segev, which I recall is a decent quick intro: http://icnc.huji.ac.il/Files/Koch_Segev_NN2000.pdf Abstract: Neurons carry out the many operations that extract meaningful information from sensory receptor arrays at the organisms periphery and translate these into action, imagery and memory. Within todays dominant computational paradigm, these operations, involving synapses, membrane ionic channels and changes in membrane potential, are thought of as steps in an algorithm or as computations. The role of neurons in these computations has evolved conceptually from that of a simple inte-grator of synaptic inputs until a threshold is reached and an output pulse is initiated, to a much more sophisticated processor with mixed analog-digital logic and highly adaptive synaptic elements. I think Koch has a book about this, but I don't recall the title offhand. In any case, people are still trying to break, e.g., the giant squid axon (the largest nerve known, and hence an oft-studied one) down into enough compartments that it can be computationally simulated with some degree of accuracy. Long-term potentiation is another aspect that is only beginning to be understood. Lots of mysteries. All very expensive to simulate until we figure out what the higher-level functions are, since you have to get every detail, not knowing which details are important. Uploading without understanding is *not* cheap, if that's what you're thinking of. You could easily end up having to go down to the molecular level. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Within-cell computation in biological neural systems??
Yan King Yin wrote: I agree, but there'll still be limitations (NP-hardness, computational and physical complexity etc). So what, if the limitations are far, far above human level? The only reason I've been going on about RSI is to make the point that, from our perspective, you can have what looks like a harmless little infrahuman AI, and the next (day, hour, year) it's a god, just like what happened with the multicellular organisms that invented agriculture. I agree with you that there will be no limits to the above 2 processes. What I'm skeptical about is how do we exploit this possibility. I cannot imagine how an AI can impose (can't think of better word) a morality on all human beings on earth, even given intergalactic computing resources. If this cannot be done, then we *must* default to self-organization of the free market economy. That means you have to specify what your AI will do, instead of relying on idealistic descriptions that have no bearing on reality. This seems to me like a sequence of complete nonsequiturs. Of course you have to specify exactly what an engineered set of dynamics do, including the dynamics that make up what is, from our perspective, a mind. Who ever said otherwise? Well, me. But I now fully acknowledge my ancient position to have been incredibly, suicidally stupid. As for the rest of it... I have no idea what you're visualizing here. To give a simple and silly counterexample, someone could roll Friendly AI Critical Failure #4 and transport the human species into a world based on Super Mario Bros - a well-specified task for an SI by comparison to most of the philosophical gibberish I've seen - in which case we would not be defaulting to self-organization of the free market economy. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
[agi] Integrating uncertainty about computation into Bayesian causal networks?
Have any of you seen a paper out there that integrates uncertainty about the results of deterministic computations into Bayesian networks? E.g., being unsure whether 2 + 2 = 4 or 2 + 2 = 5. In particular I'm wondering whether anyone's tried to integrate axiomatic proof systems into causal networks a la Judea Pearl, i.e., if we prove A implies B (but haven't yet proven B implies A) then we draw a directed arc from A to B in our Bayesian causal diagram. I was recently trying to write a paper on a different subject that required me to introduce uncertainties about computation into causal networks, and I noticed that this subtopic was interesting enough to perhaps deserve a paper in its own right. I'm wondering whether anyone on the list has seen such integration attempted yet, by way of avoiding duplication of effort. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] estimated cost of Seed AI
Alexander E. Richter wrote: At 20:54 11.06.05 -0400, you wrote: What is the estimated cost of Seed AI? Dan one person 10 hours/day IQ 180+ very good memory (photographic memory) high frustration-tolerance 1000-2000 $/month (To keep mind free from waste and unnecessary thought) hardware 5000 $/year it will take 5-10 years (starting now) it will take 1-7 years (someone working on it already) Imho, its more like the development of H1-H4 sea clocks (John Harrison) cu Alex More or less me too. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] intuition [was: ... [was: ...]]
Pei Wang wrote: Definitions like the Wikipedia one have the problem of only talking about the good/right intuitions, while there are at least as many intuitions that are bad/wrong. To call them by another name would make things worse, because they are produced by the same mechanism, therefore you couldn't get the good ones without the bad ones. Instead of using positive terms like understanding and insight, I define intuition neutrally as a belief a system has, but cannot explain where it came from. I suggest that this definition defines a continuum of intuitiveness rather than a clustered binary category, where larger leaps that are less explainable are more intuitive. I specify large leaps, because even explainable reasoning is only explained as a series of obvious-seeming atomic steps. Only a specialist in the cognitive sciences could try to explain why the atomic steps are obvious; and even then, could only offer a high-level explanation, in terms of work performed by cognition and evolutionary selection pressures, rather than a neurological stack trace. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
[agi] Re: Superrationality
Ben Goertzel wrote: Hi, A few years ago there was a long discussion on this list or the AGI list about Hofstadterian superrationality... http://en.wikipedia.org/wiki/Superrationality I wonder if anyone knows of any mathematical analysis of superrationality. I worked out an analysis based on correlated computational processes - you treat your own decision system as a special case of computation and decide as if your decision determines the output of all computations that are similar to the decision. Or to put it another way, you don't choose as if you believe that multiple instantiations of an identical abstract computation can have different outputs. This can be formalized by extending Judea Pearl's causal graphs to include uncertainty about abstract computations, and reworking Pearl's surgical formalism for acts accordingly, which in turn is justified by considerations that these margins are too small to include. I haven't published this, but I believe I mentioned it on AGI during a discussion of AIXI. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
[agi] Re: Superrationality
Ben Goertzel wrote: Hi, I'll be offline for the rest of the day, but I'll write up something clear on my very simple decision theory approach to superrationality, tomorrow or over the weekend, and post it for criticism ;-) It deals simply with Newcomb's Paradox [taking the one box] but I did not think about Solomon's Problem, in fact I am not sure what this problem is (I assume it is related to King Solomon and the baby-halving threat, but if you could point to a webpage giving a crisp formulation of the problem I'd appreciate it). Not the baby-halving threat, actually. http://www.geocities.com/eganamit/NoCDT.pdf Here Solomon's Problem is referred to as The Smoking Lesion, but the formulation is equivalent. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Re: Superrationality
John Oh wrote: Eliezer, Under your decision theory that you believe to be superior to CDT and EDT, please answer the yes/no questions below: Do you believe the formulation of the Smoking Lesion as stated in the paper you cited below to be sufficiently complete to enable a definite answer? Looks like it. If so, does your theory conclude that Susan should smoke? (This should be trivial to answer, even if you are unwilling to explain how or why your theory provides the answer that it does.) Yes, Susan should smoke under the given conditions. She should also take one box in Newcomb's Problem. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Re: Superrationality
John Oh wrote: And did I hear you correctly that you also believe Susan should cooperate in a standard one-shot Prisoner's Dilemma (assuming she believes there is a high enough probability that the opposing player is sufficiently similar to her)? Correct. For example, if Susan is facing her synchronized Copy, she should definitely cooperate. Against her desynchronized Copy she will be unable to prove anything if she is a human-level intelligence, but she should still intuitively cooperate. A superintelligence can prove similarity of decision systems that are not copies but which implement an identical algorithm at a deep level, *or* prove similarity of a decision system that contains an explicit dependency on an accurate model of the superintelligence. In other words, the superintelligence SI-A can decide to cooperate with SI-B because SI-B decided to cooperate if it modeled SI-A as cooperating and SI-A knows this. This gives SI-B a motive to decide to cooperate if it models SI-A as cooperating. That, roughly speaking, is how two dissimilar SIs would cooperate on the oneshot PD if they can obtain knowably reliable information about each other - through random sampling of computing elements, historical modeling, or even a sufficiently strong prior probability. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
[agi] Re: Superrationality
Ben Goertzel wrote: Thanks for the reference. The paper is entertaining, in that both the theories presented (evidential decision theory and causal decision theory) are patently very stupid right from the outset ;-) EDT and CDT have been the two dominant decision theories, with CDT having the upper hand, for decades. I agree that both are wrong, but it is an audacious assertion. I haven't written up my own mathematical analysis because it would require on the order of a book to put forth an alternative theory in academia. I just did the analysis for myself because I needed to know if I had to do any special work in setting up the initial conditions of an FAI. EDT's foolishness is more mathematical in nature (via setting up the problem mathematically in a way that ignores relevant information) whereas CDT's foolishness is more philosophical in nature EDT and CDT are precisely symmetrical except in how they compute counterfactual probabilities. (essentially, via introducing the folk-psychology notion of causality which has no role in rigorous formal analyses of events). Causality a folk-psychology notion? Judea Pearl begs to disagree with you, and I beg to agree with Judea Pearl. My own theory is causal in nature - that is, it uses Pearl's graphs. I really think this stuff is not that complicated; but people seem to be misled in thinking about it via commonplace illusions related to free will ... The answer itself is simple. Justifying it is not. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] AGIRI Summit
Philip Goetz wrote: There are many such accounts. I've yet to come across an account of a patient who lost their procedural memory but ratained declarative memory. Perhaps such a patient would be diagnosed with motor problems rather than with memory loss. If you lose your procedural memory, you forget how to move, how to talk, and how to operate your brain. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Two draft papers: AI and existential risk; heuristics and biases
to high standards, work out detailed consequences, expose themselves to criticism, witness other people doing things that they think will fail and so beginning to appreciate the danger of the problem... What counts in the long run is whether your initial mistaken approach forces you to go out and learn many fields, work out consequences in detail, appreciate the scope of the problem, hold yourself to high standards, fear mental errors, and, above all, keep working on the problem. The mistake I made in 1996, when I thought any SI would automatically be Friendly, was a very bad mistake because it meant I didn't have to devote further thought to the problem. The mistake I made in 2001 was just as much a mistake, but it was a mistake that got me to spend much more time thinking about the problem and study related fields and hold myself to a higher standard. If I saw a detailed experimental plan, expressed mathematically, that drew on concepts from four different fields or whatever... I wouldn't credit you with succeeding, but I'd see that you were actually changing your strategy, making Friendliness a first-line requirement in the sense of devoting actual work-hours to it, showing willingness to shoulder safety considerations even when they seem bothersome and inconvenient, etc. etc. It would count as *trying*. It might get you to the point where you said, Whoa! Now that I've studied just this one specific problem for a couple of years, I realize that what I had previously planned to do won't work! But in terms of how you spend your work-hours, which code you write, your development plans, how you allocate your limited reading time to particular fields, then this business of First experiment with AGI has the fascinating and not-very-coincidental-looking property of having given rise to a plan that looks exactly like the plan one would pursue if Friendly AI were not, in fact, an issue. Because this is a young field, how much mileage you get out will be determined in large part by how much sweat you put in. That's the simple practical truth. The reasons why you do X are irrelevant given that you do X; they're screened off, in Pearl's terminology. It doesn't matter how good your excuse is for putting off work on Friendly AI, or for not building emergency shutdown features, given that that's what you actually do. And this is the complaint of IT security professionals the world over; that people would rather not think about IT security, that they would rather do the minimum possible and just get it over with and go back to their day jobs. Who can blame them for such human frailty? But the result is poor IT security. What kind of project might be able to make me believe that they had a serious chance of achieving Friendly AI? They would have to show that, rather than needing to be *argued into* spending effort and taking safety precautions, they enthusiastically ran out and did as much work on safety as they could. That would not be sufficient but it would certainly be necessary. They would have to show that they did *not* give in to the natural human tendency to put things off until tomorrow, nor come up with clever excuses why inconvenient things do not need to be done today, nor invent reasons why they are almost certainly safe for the moment. For whoso does this today will in all probability do the same tomorrow and tomorrow and tomorrow. I do think there's a place for experiment in Friendly AI development work, which is as follows: One is attempting to make an experimental prediction of posthuman friendliness and staking the world on this prediction; there is no chance for trial and error; so, as you're building the AI, you make experimental predictions about what it should do. You check those predictions, by observation. And if an experimental prediction is wrong, you halt, melt, catch fire, and start over, either with a better theory, or holding yourself to a stricter standard for what you dare to predict. Maybe that is one way an adolescent could confront an adult task. There are deeper theoretical reasons (I'm working on a paper about this) why you could not possibly expect an AI to be Friendly unless you had enough evidence to *know* it was Friendly; roughly, you could not expect *any* complex behavior that was a small point in the space of possibilities, unless you had enough evidence to single out that small point in the large space. So, although it sounds absurd, you *should* be able to know in advance what you can and can't predict, and test those predictions you dare make, and use that same strict standard to predict the AI will be Friendly. You should be able to win in that way if you can win at all, which is the point of the requirement. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change
Re: [agi] How the Brain Represents Abstract Knowledge
Eric Baum wrote: It is demonstrably untrue that the ability to predict the effects of any action, suffices to decide what actions one should take to reach one's goals. For example, given a specification of a Turing machine, one can predict its sequence of states if one feeds in any particular program, but one can not necessarily say what program to feed in to put it in a specific state :^) Given a specification of a Turing machine, we can predict (given unbounded computing power) its output for any particular input, supposing that the machine halts at all. But similarly, given unbounded computing power, we can find an input that produces any particular output, supposing that at least one such input exists. Perhaps you mean to say that, even given a *fast* algorithm for predicting the results of particular inputs, there may exist no *fast* algorithm for finding an input that corresponds to a particular output. Thus we can solve the real-world prediction problem without solving the real-world manipulation problem. There is more to planning than just prediction. Of course I agree. This is my chief criticism of Hawkins. Rationality as conventionally defined has two components, probability theory and decision theory. If you leave out the decision theory, you can't even decide which information to gather. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] singularity humor
I think this one was the granddaddy: http://yudkowsky.net/humor/signs-singularity.txt -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Processing speed for core intelligence in human brain
Ben Goertzel wrote: Hi, On a related subject, I argued in What is Thought? that the hard problem was not processor speed for running the AI, but coding the software, This is definitely true. Agreed. However, Warren has recently done some digging on the subject, and come up with what seems to be a better estimate that 10^44 bacteria have lived on Earth. However, evolution is not doing software design using anywhere near the same process that we human scientists are. So I don't think these sorts of calculations are very directly relevant... Also very much agreed. Transistor speeds and neural speeds, evolutionary search and intelligent search, are not convertible quantities; it is like trying to convert temperature to mass, or writing an equation that says E = MC^3. See e.g. http://dspace.dial.pipex.com/jcollie/sle/index.htm -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
[agi] Re: strong and weakly self improving processes
and passing on partial progress, for example in the mode of bed-time stories and fiction. Both for language itself and things like theory of mind, one can imagine some evolutionary improvements in ability to use it through the Baldwin effect, but the main point here seems to be the use of external storage in culture in developing the algorithms and passing them on. Other examples of modules that directly effect thinking prowess would be the axiomatic method, and recursion, which are specific human discoveries of modes of thinking, that are passed on using language and improve intelligence in a core way. Considering the infinitesimal amount of information that evolution can store in the genome per generation, on the order of one bit, it's certainly plausible that a lot of our software is cultural. This proposition, if true to a sufficiently extreme degree, strongly impacts my AI ethics because it means we can't read ethics off of generic human brainware. But it has very little to do with my AGI theory as such. Programs are programs. But try to teach the human operating system to a chimp, and you realize that firmware counts for *a lot*. Kanzi seems to have picked up some interesting parts of the human operating system - but Kanzi won't be entering college anytime soon. The instructions that human beings communicate to one another are instructions for pulling sequences of levers on an enormously complex system, the brain, which we never built. If the machine is not there, the levers have nothing to activate. When the first explorers of AI tried to write down their accessible knowledge of How to be a scientist as code, they failed to create a scientist. They could not introspect on, and did not see, the vast machine their levers controlled. If you tell a human, Try to falsify your theories, rather than trying to prove them, they may learn something important about how to think. If you inscribe the same words on a rock, nothing happens. Don't get me wrong - those lever-pulling sequences are important. But the true power, I think, lies in the firmware. Could a human culture with a sufficiently different operating system be more alien to us than bonobos? More alien than a species that evolved on another planet? If most of the critical complexity is in the OS, then you'd expect this to be the case. Maybe I just lack imagination, but I have difficulty seeing it. Another ramification of this layered picture are all the ways that evolution evolves to evolve better, including finding meaningful chunks that can then be put together into programs in novel ways. These are analogous to adding or improving lower layers on an intelligent system, which may make it as intelligent as modifying the top layers would in any conceivable way. Evolution, which constructed our ability to rationally design, may apply very much the same processes on itself. I don't understand any real distinction between weakly self improving processes and strongly self improving processes, and hence, if there is such a distinction, I would be happy for clarification. The cheap shot reply is: Try thinking your neurons into running at 200MHz instead of 200Hz. Try thinking your neurons into performing noiseless arithmetic operations. Try thinking your mind onto a hundred times as much brain, the way you get a hard drive a hundred times as large every 10 years or so. Now that's just hardware, of course. But evolution, the same designer, wrote the hardware and the firmware. Why shouldn't there be equally huge improvements waiting in firmware? We understand human hardware better than human firmware, so we can clearly see how restricted we are by not being able to modify the hardware level. Being unable to reach down to firmware may be less visibly annoying, but it's a good bet that the design idiom is just as powerful. The further down you reach, the more power. This is the idiom of strong self-improvement and I think the hardware reply is a valid illustration of this. It seems so simple that it sounds like a cheap shot, but I think it's a valid cheap shot. We were born onto badly designed processors and we can't fix that by pulling on the few levers exposed by our introspective API. The firmware is probably even more important; it's just harder to explain. And merely the potential hardware improvements still imply I. J. Good's intelligence explosion. So is there a practical difference? Eric Baum http://whatisthought.com -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
[agi] Re: strong and weakly self improving processes
powerful generator produce knowledge orders of magnitude faster? Obviously yes, because human neurons run at speeds that are at least six orders of magnitude short of what we know to be physically possible. (Drexler's _Nanosystems_ describes sensory inputs and motor outputs that operate at a similar speedup.) What about better firmware? Would that buy us many additional orders of magnitude? If most of the generator complexity lay in a culturally transmitted human operating system that was open to introspection, then further improvements to firmware might be trivial. But then scientists would have a much better understanding of how science works; but most scientists proceed mostly by instinct, and they don't have to learn rituals on anything remotely approaching the complexity of a human brain. Most people would find learning the workings of the human brain a hugely intimidating endeavor - rather than being an easier and simpler version of something they did unwittingly as children, in the course of absorbing the larger and more important human operating system you postulate. This human operating system, this modular theory of mind that gets transmitted - where is it written down? There's a sharp limit on how much information you can accumulate without digital fidelity of transmission between generations. The vast majority of human evolution took place long before the invention of writing. I don't believe in a culturally transmitted operating system, that existed over evolutionary periods, which contains greater total useful complexity than that specified in the brain-constructing portions of the human genome itself. And even if such a thing existed, the fact that we haven't written it down implies that it is largely inaccessible to introspection and hence to deliberative, intelligent self-modification. I don't understand any real distinction between weakly self improving processes and strongly self improving processes, and hence, if there is such a distinction, I would be happy for clarification. Eliezer The cheap shot reply is: Try thinking your neurons into Eliezer running at 200MHz instead of 200Hz. Try thinking your Eliezer neurons into performing noiseless arithmetic operations. Try Eliezer thinking your mind onto a hundred times as much brain, the Eliezer way you get a hard drive a hundred times as large every 10 Eliezer years or so. Eliezer Now that's just hardware, of course. But evolution, the same Eliezer designer, wrote the hardware and the firmware. Why shouldn't Eliezer there be equally huge improvements waiting in firmware? We Eliezer understand human hardware better than human firmware, so we Eliezer can clearly see how restricted we are by not being able to Eliezer modify the hardware level. Being unable to reach down to Eliezer firmware may be less visibly annoying, but it's a good bet Eliezer that the design idiom is just as powerful. Eliezer The further down you reach, the more power. This is the Eliezer idiom of strong self-improvement and I think the hardware Eliezer reply is a valid illustration of this. It seems so simple Eliezer that it sounds like a cheap shot, but I think it's a valid Eliezer cheap shot. We were born onto badly designed processors and Eliezer we can't fix that by pulling on the few levers exposed by our Eliezer introspective API. The firmware is probably even more Eliezer important; it's just harder to explain. Eliezer And merely the potential hardware improvements still imply Eliezer I. J. Good's intelligence explosion. So is there a practical Eliezer difference? The cheapshot reply to your cheapshot reply, is that if we construct an AI, that AI is just another part of the lower level in the weakly self-improving process, its part of our culture, so we can indeed realize the hardware improvement. This may sound cheap, but it shows there is no real difference between the 2 layered system and the entirely self-recursive one. The cheap-cheap-cheap-reply is that if a self-improving AI goes off and builds a Dyson Sphere, and that is no real difference, I'm not sure I want to see what a real difference looks like. Again, the cheap^3 reply seems to me valid because it asks what difference of experience we anticipate. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
[agi] Re: strong and weakly self improving processes
Eliezer S. Yudkowsky wrote: Eric Baum wrote: Eliezer Considering the infinitesimal amount of information that Eliezer evolution can store in the genome per generation, on the Eliezer order of one bit, Actually, with sex its theoretically possible to gain something like sqrt(P) bits per generation (where P is population size), cf Baum, Boneh paper could be found on whatisthought.com and also Mackay paper. (This is a digression, since I'm not claiming huge evolution since chimps). I furthermore note that gaining one standard deviation per generation, which is what your paper describes, is not obviously like gaining sqrt(P) bits of Shannon information per generation. Yes, the standard deviation is proportional to sqrt(N), but it's not clear how you're going from that to gaining sqrt(N) bits of Shannon information in the gene pool per generation. It would seem heuristically obvious that if your algorithm eliminates roughly half the population on each round, it can produce at most one bit of negentropy per round in allele frequencies. I only skimmed the referenced paper, though; so if there's a particular paragraph I ought to read, feel free to direct me to it. Yeah, so, I reread my paragraph above and it doesn't make any sense. Standard deviations are not proportional to the square root of the population size (duh). N in Baum, Boneh, and Garrett (1995) is the length of the string, not the population size. Why http://citeseer.ist.psu.edu/82728.html shows that you can gain anything proportional to sqrt(P) per generation, I have no idea. As for gaining sqrt(P) bits of negentropy in the gene pool by eliminating merely half the population, I just don't see how that would work. Maybe you're thinking of sqrt(log(P)) which would be how much you could gain in one generation by culling all but the uppermost part of a Gaussian distribution? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] [META] Is there anything we can do to keep junk out of the AGI Forum?
Richard Loosemore wrote: I am beginning to wonder if this forum would be better off with a restricted membership policy. SL4 uses the List Sniper technique. Anyone can join, and if they don't seem suitable, they're removed. The bane of mailing lists is well-intentioned but stupid people, and list moderators who can't bring themselves to say anything so impolite as Goodbye. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] fuzzy logic necessary?
When you think something is more likely or less likely, you're translating a feeling into English. The English translation doesn't involve verbal probabilities like 0.6 or 0.8 - the syllables probability zero point eight don't flow through your auditory workspace. But that doesn't rule out the possibility that, somewhere in your head, neurons are spiking at .733 of their maximum rate, which you feel as a strong probability, and which you translate into English as pretty likely. Now there are other ways that neurons might quantitatively represent feelings of anticipation; and whatever the quantities, they won't obey Bayesian rules except qualitatively, because the brain is an evolved hack. But you also shouldn't suppose that an AI can operate on LISP tokens labeled more likely and less likely; such English phrases merely serialize your feelings of anticipation into acoustic vibrations so that you can transmit them to another human who translates them back into internal quantities. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
[agi] Re: On proofs of correctness
Charles D Hixson wrote: An excellent warning on proofs of correctness is at: http://googleresearch.blogspot.com/2006/06/extra-extra-read-all-about-it-nearly.html Hear, hear. But I don't take away the lesson that not even formal methods are powerful enough. They would've caught this if, for example, they'd assumed and proven from transistor diagrams of the underlying chip. Formal methods break when their axioms break, and the proof of correctness for this code presumably used axioms that were only true most of the time, axioms that could be broken even without cosmic rays. I don't think this is mere argument-in-hindsight; it occurred to me long ago not to trust integer addition, just transistors. And even then, shielded hardware and reproducible software would not be out of order. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Marcus Hutter's lossless compression of human knowledge prize
As long as we're talking about fantasy applications that require superhuman AGI, I'd be impressed by a lossy compression of Wikipedia that decompressed to a non-identical version carrying the same semantic information. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Lossy ** lossless compression
Matt Mahoney wrote: DEL has a lossy model, and nothing compresses smaller. Is it smarter than PKZip? Let me state one more time why a lossless model has more knowledge. If x and x' have the same meaning to a lossy compressor (they compress to identical codes), then the lossy model only knows p(x)+p(x'). A lossless model also knows p(x) and p(x'). You can argue that if x and x' are not distinguishable then this extra knowledge is not important. But all text strings are distinguishable to humans. Suppose I give you a USB drive that contains a lossless model of the entire universe excluding the USB drive - a bitwise copy of all quark positions and field strengths. (Because deep in your heart, you know that underneath the atoms, underneath the quarks, at the uttermost bottom of reality, are tiny little XML files...) Let's say that you've got the entire database, and a Python interpreter that can process it at any finite speed you care to specify. Now write a program that looks at those endless fields of numbers, and says how many fingers I'm holding up behind my back. Looks like you'll have to compress that data first. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Why so few AGI projects?
Shane Legg wrote: Funding is certainly a problem. I'd like to work on my own AGI ideas after my PhD is over next year... but can I get money to do that? Probably not. So as a compromise I'll have to work on something else in AI during the day, and spend my weekends doing the stuff I'd really like to be doing. Currently I code my AI at nights and weekends. Shane, what would you do if you had your headway? Say, you won the lottery tomorrow (ignoring the fact that no rational person would buy a ticket). Not just AGI - what specifically would you sit down and do all day? If there's somewhere online that already answers this or a previous AGI message I should read, just point. Pressure to publish is also a problem. I need results on a regular basis that I can publish otherwise my career is over. AGI is not really short term results friendly. Indeed not. It takes your first five years simply to figure out which way is up. But Shane, if you restrict yourself to results you can regularly publish, you couldn't work on what you really wanted to do, even if you had a million dollars. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Natural versus formal AI interface languages
Pei Wang wrote: Let's don't confuse two statements: (1) To be able to use a natural language (so as to passing Turing Test) is not a necessary condition for a system to be intelligent. (2) A true AGI should have the potential to learn any natural language (though not necessarily to the level of native speakers). I agree with both of them, and I don't think they contradict to each other. Natural language isn't. Humans have one specific idiosyncratic built-in grammar, and we might have serious trouble learning to communicate in anything else - especially if the language was being used by a mind quite unlike our own. Even a programming language is still something that humans made, and how many people do you know who can *seriously*, not-jokingly, think in syntactical C++ the way they can think in English? I certainly think that something could be humanish-level intelligent in terms of optimization ability, and not be able to learn English, if it had a sufficiently alien cognitive architecture - nor would we be able to learn its languge. Of course you can't be superintelligent and unable to speak English - *that* wouldn't make any sense. I assume that's what you mean by true AGI above. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Natural versus formal AI interface languages
Eric Baum wrote: (Why should producing a human-level AI be cheaper than decoding the genome?) Because the genome is encrypted even worse than natural language. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
Re: [agi] Natural versus formal AI interface languages
Eric Baum wrote: Eliezer Eric Baum wrote: (Why should producing a human-level AI be cheaper than decoding the genome?) Eliezer Because the genome is encrypted even worse than natural Eliezer language. (a) By decoding the genome, I meant merely finding the sequence (should have been clear in context), which didn't involve any decryption at all. (b) why do you think so? (a) Sorry, didn't pick up on that. Possibly, more money has already been spent on failed AGI projects than on the human genome. (b) Relative to an AI built by aliens, it's possible that the human proteome annotated by the corresponding selection pressures (= the decrypted genome), is easier to reverse-engineer than the causal graph of human language. Human language, after all, takes place in the context of a complicated human mind. But relative to humans, human language is certainly a lot easier for us to understand than the human proteome! -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
Re: [agi] SOTA
Philip Goetz wrote: Haven't been googling. But the fact is that I've never actually /seen/ one in the wild. My point is that the market demand for such simple and useful and cheap items is low enough that I've never actually seen one. Check any hardware store, there's a whole shelf. I bought one for my last apartment. I see them all over the place. They're really not rare. Moral: in AI, the state of the art is often advanced far beyond what people think it is. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
[agi] Optimality of using probability
harder, because I don't actually *know* that it's a lot harder, because I don't know exactly how to do it, and therefore I don't know yet how hard or easy it will be. I suspect it's more complicated than the simple case, at least. I tried to solve this problem in 2006, just in case it was easier than it looked (it wasn't). I concluded that the problem required a fairly sophisticated mind-system to carry out the reasoning that would justify probabilities, so I was blocking on subparts of this mind-system that I didn't know how to specify yet. Thus I put the problem on hold and decided to come back to it later. As a research program, the difficulty would be getting a researcher to see that a nontrivial problem exists, and come up with some non-totally-ad-hoc interesting solution, without their taking on a problem so large that they can't solve it. One decent-sized research problem would be scenarios in which you the programmer could expect utility from a program that used probabilities, in a state of programmer knowledge that *didn't* let you calculate those probabilities yourself. One conceptually simple problem, that would still be well worth a publication if no one has done it yet, would be calculating the expected utilities of using well-known uninformative priors in plausible problems. But the real goal would be to justify using probability in cases of structural uncertainty. A simple case of this more difficult problem would be calculating the expected utility of inducting a Bayesian network with unknown latent structure, known node behaviors (like noisy-or), known priors for network structures, and uninformative priors for the parameters. One might in this way work up to Boolean formulas, and maybe even some classes of arbitrary machines, that might be in the environment. I don't think you can do a similar calculation for Solomonoff induction, even in principle, because Solomonoff is uncomputable and therefore ill-defined. For, say, Levin search, it might be doable; but I would be VERY impressed if anyone could actually pull off a calculation of expected utility. In general, I would suggest starting with the expected utility of simple uninformative priors, and working up to more structural forms of uncertainty. Thus, strictly justifying more and more abstract uses of probabilistic reasoning, as your knowledge about the environment becomes ever more vague. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
Re: [agi] Betting and multiple-component truth values
Ben Goertzel wrote: However, this doesn't solve the problem of finite resources making true probabilistic accuracy impossible, of course. AGI systems with finite resources will in fact not be ideally rational betting machines; they will not fully obey Cox's axioms; an ideal supermind would be able to defeat them via clever betting taking advantage of their weaknesses. Any entity not logically omniscient can be trivially bilked by a logically omniscient bookie, because the non-logically-omniscient player assigns positive probabilities to events that are logically impossible. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
Re: [agi] Betting and multiple-component truth values
Pei Wang wrote: Ben, Sorry that now I don't have the time for long discussions, but a brief scan of your message remind me of Ellsberg paradox (see http://en.wikipedia.org/wiki/Ellsberg_paradox). He used betting examples to show that a probability is not enough, and a second number is necessary. The Ellsberg paradox is a descriptive observation about human psychology. At most it only shows that probability does not suffice to describe human reasoning, saying nothing about normativeness; and of course, we know from many descriptive examples that humans do not use Kolmogorov/Cox consistent probabilities. This says nothing about what kind of mind we would *want* to build, though. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
[agi] Re: Optimality of using probability
Mitchell Porter wrote:
If you the programmer ('you' being an AI, I assume) already have the
concept of probability, and you can prove that a possible program will
estimate probabilities more accurately than you do, you should be able
to prove that it would provide an increase in utility, to a degree
depending on the superiority of its estimates and the structure of
your utility function. (A trivial observation, but that's usually where
you have to start.)
Mitch, I haven't found that problem to be trivial if one seeks a precise
demonstration. I say precise demonstration, rather than formal
proof, because formal proof often carries the connotation of
first-order logic, which is not necessarily what I'm looking for. But a
line of reasoning that an AI itself carries out will have some exact
particular representation and this is what I mean by precise. What
exactly does it mean for an AI to believe that a program, a collection
of ones and zeroes, estimates probabilities more accurately than
does the AI? And how does the AI use this belief to choose that the
expected utility of running its program is ordinally greater than the
expected utility of the AI exerting direct control? For simple cases -
where the statistical structure of the environment is known, so that you
could calculate the probabilities yourself given the same sensory
observations as the program - this can be argued precisely by summing
over all probable observations. What if you can't do the exact sum?
How would you make the demonstration precise enough for an AI to walk
through it, let alone independently discover it?
*Intuitively* the argument is clear enough, I agree.
--
Eliezer S. Yudkowsky http://singinst.org/
Research Fellow, Singularity Institute for Artificial Intelligence
-
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?list_id=303
Re: [agi] Betting and multiple-component truth values
Pei Wang wrote: On 2/8/07, gts [EMAIL PROTECTED] wrote: I gave an example of a Dutch book in a post to Russell in which an incoherent thinker assigns a higher probability to intelligent life on Mars than to mere life on Mars. Since the first hypothesis can be true only if the second is true, it is incoherent to assign a higher probability to the first than to the second. Coherence is basically just common sense applied to probabilistic reasoning. I'm dismayed to learn from Ben that coherence is so difficult to achieve in AGI. In simple cases like the above one, an AGI should achieve coherence with little difficulty. What an AGI cannot do is to guarantee coherence in all situations, which is impossible for human beings, neither --- think about situations where the incoherence of a bet setting needs many steps of inference, as well as necessary domain knowledge, to reveal. Actually, conjunction fallacy is probably going to be one of the most difficult of all biases to eliminate; it may even be provably impossible for entities using any complexity-based variant of Occam's Razor, such as Kolmogorov complexity. If you ask for P(A) at time T and then P(AB) at time T+1, you should get a higher answer for P(AB) wherever A is a complex set of variable values that are insufficiently supported by direct evidence, and B is a non-obvious compact explanation for A. Thus, seeing B reduces the apparent Kolmogorov complexity of A, raising A's prior. You cannot always see B directly from A because this amounts to always being able to find the most compact explanation, which amounts to finding the shortest Turing machine that reproduces the data, which is unsolvable by the halting problem. I have sometimes thought that Levin search might yield provably consistent probabilities - after all, a supposed explanation doesn't do you any good if you can't derive data from it or prove that it halts. Even so, seeing B directly from A might require an exponential search too costly to perform. Thus, conjunction fallacy - cases where being told about the hypothesis B raises the subjective probability of P(AB) over that you previously gave to P(A) - is probably with us to stay, even unto the furthest stars. It may greatly diminish but not be utterly defeated. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
Re: [agi] Betting and multiple-component truth values
Ben Goertzel wrote: However, it shouldn't be hard for AGIs to avoid the particularly simple and glaring examples of conjunction fallacy that have been made famous in the cognitive psychology literature... Some of them, but not others. For an example of the more difficult case: ** Two independent sets of professional analysts at the Second International Congress on Forecasting were asked to rate, respectively, the probability of A complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983 or A Russian invasion of Poland, and a complete suspension of diplomatic relations between the USA and the Soviet Union, sometime in 1983. The second set of analysts responded with significantly higher probabilities. ** This is a type of conjunction fallacy where, arguably, an AI can beat a human in this specific case, but only by expending more computing power to search through many possible pathways from previous beliefs to the conclusion. In which case, given a more complex scenario, one that defeated the AI's search capabilities, the AI would fail in a way essentially analogous to the human who conducts almost no search. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
Re: [agi] Priors and indefinite probabilities
Benjamin Goertzel wrote: Tying together recent threads on indefinite probabilities and prior distributions (PI, maxent, Occam), I thought I'd make a note on the relation between the two topics. In the indefinite probability approach, one assigns a statement S a truth value L,U,b,k denoting one's attachment of probability b to the statement that: after k more observations have been made, one's best guess regarding the probability of S will lie in [L,U]. Ben, is the indefinite probability approach compatible with local propagation in graphical models? -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
Re: Languages for AGI [WAS Re: [agi] Priors and indefinite probabilities]
Chuck Esterbrook wrote: On 2/18/07, Eliezer S. Yudkowsky [EMAIL PROTECTED] wrote: Heh. Why not work in C++, then, and write your own machine language? No need to write files to disk, just coerce a pointer to a function pointer. I'm no Lisp fanatic, but this sounds more like a case of Greenspun's Tenth Rule to me. I find C++ overly complex while simultaneously lacking well known productivity boosters including: * garbage collection * language level bounds checking * contracts * reflection / introspection (complete and portable) * dynamic loading (portable) * dynamic invocation I was being sarcastic, not advocating C++ as the One True AI language. Eliezer, do write code at the institute? What language do you use and for what reasons? What do you like and dislike about it with respect to your project? Just curious. I'm currently a theoretician. My language-of-choice is Python for programs that are allowed to be slow. C++ for number-crunching. Incidentally, back when I did more programming in C++, I wrote my own reflection package for it. (In my defense, I was rather young at the time.) B. Sheil once suggested that LISP excels primarily at letting you change your code after you realize that you wrote the wrong thing, and this is why LISP is the language of choice for AI work. Strongly typed languages enforce boundaries between modules, and provide redundant constraints for catching bugs, which is helpful for coding conceptually straightforward programs. But this same enforcement and redundancy makes it difficult to change the design of the program in midstream, for things that are not conceptually straightforward. Sheil wrote in the 1980s, but it still seems to me like a very sharp observation. If you know in advance what code you plan on writing, choosing a language should not be a big deal. This is as true of AI as any other programming task. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?list_id=303
[agi] Gödel's theorem for intelligence
Shane Legg wrote: Thus I think that the analogue of Gödel's theorem here would be something more like: For any formal definition of intelligence there will exist a form of intelligence that cannot be proven to be intelligent even though it is intelligent. With unlimited computing power this is obvious. Take a computation that halts if it finds an even number that is not the sum of two primes. Append AIXItl. QED. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=231415user_secret=fabd7936
Re: [agi] definitions of intelligence, again?!
Shane Legg wrote: Would the following be possible with your notion of intelligence: There is a computer system that does a reasonable job of solving some optimization problem. We go along and keep on plugging more and more RAM and CPUs into the computer. At some point the algorithm sees that it has enough resources to always solve the problem perfectly through brute force search and thus drops its more efficient but less accurate search strategy. As the system is now solving the optimization problem in a much simpler way (brute force search), according to your perspective it has actually become less intelligent? It has become more powerful and less intelligent, in the same way that natural selection is very powerful and extremely stupid. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=231415user_secret=fabd7936
Re: [agi] Write a doctoral dissertation, trigger a Singularity
Why is Murray allowed to remain on this mailing list, anyway? As a warning to others? The others don't appear to be taking the hint. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=231415user_secret=fabd7936
Re: [agi] Opensource Business Model
Russell Wallace wrote: *sigh* I thought you knew better than this. The idea that the bottleneck in AGI was the discovery of secret sauce algorithms was excusable back in the 80s. We've known better for a good many years now. Ditto across the board: sure there is this and that algorithm that isn't known that would make life somewhat easier, but in practice the limiting factor is almost entirely in putting the pieces together the right way, and hardly at all in missing any particular piece. Belldandy preserve us. You think you know everything you need to know, have every insight you require, to put a sentient being together? That the rest is just an implementation detail? That, moreover, *any* modern computer scientist knows it? What can I say, but: ... -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=231415user_secret=e9e40a7e
Re: [agi] Opensource Business Model
Russell Wallace wrote: On 6/1/07, *Eliezer S. Yudkowsky* [EMAIL PROTECTED] mailto:[EMAIL PROTECTED] wrote: Belldandy preserve us. You think you know everything you need to know, have every insight you require, to put a sentient being together? That the rest is just an implementation detail? That, moreover, *any* modern computer scientist knows it? Belldandy preserve us indeed! I thought you were at least a little more in tune with reality than that. Please reread the last hundred times I explained why creating a sentient being ab initio is completely infeasible for you, me or anyone else. I read your theory as, I can't do it myself because the implementation is too large, but I know everything required for a team of 10^N people to do it; moreover, any computer scientist knows it. You haven't noticed that there's anything about cognition that, how shall I put this, *confuses* you? Belldandy preserve us with formaldehyde! -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=231415user_secret=e9e40a7e
Re: [agi] Beyond AI chapters up on Kurzweil
J Storrs Hall, PhD wrote: The Age of Virtuous Machines http://www.kurzweilai.net/meme/frame.html?main=/articles/art0708.html I am referred to therein as Eliezer Yudkowsk. Hope this doesn't appear in the book too. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=231415user_secret=e9e40a7e
Re: [agi] poll: what do you look for when joining an AGI group?
Clues. Plural. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=231415user_secret=e9e40a7e
Re: [agi] poll: what do you look for when joining an AGI group?
Mark Waser wrote: P.S. You missed the time where Eliezer said at Ben's AGI conference that he would sneak out the door before warning others that the room was on fire:-) This absolutely never happened. I absolutely do not say such things, even as a joke, because I understand the logic of the multiplayer iterated prisoner's dilemma - as soon as anyone defects, everyone gets hurt. Some people who did not understand the IPD, and hence could not conceive of my understanding the IPD, made jokes about that because they could not conceive of behaving otherwise in my place. But I never, ever said that, even as a joke, and was saddened but not surprised to hear it. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=231415user_secret=e9e40a7e
Re: [agi] poll: what do you look for when joining an AGI group?
Hm. Memory may be tricking me. I did a deeper scan of my mind, and found that the only memory I actually have is that someone at the conference said that they saw I wasn't in the room that morning, and then looked around to see if there was a bomb. I have no memory of the fire thing one way or the other, but it sounds like a plausible distortion of the first event after a few repetitions. Or maybe the intended meaning is that, if I saw a fire in a room, I would leave the room first to make sure of my own safety, and then shout Fire! to warn everyone else? If so, I still don't remember saying that, but it doesn't have the same quality of being the first to defect in an iterated prisoner's dilemma - which is the main thing I feel I need to emphasize heavily that I will not do; no, not even as a joke, because talking about defection encourages people to defect, and I won't be the first to talk about it, either. So I guess the moral is that I shouldn't toss around the word absolutely - even when the point needs some heavy moral emphasis - about events so far in the past. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=231415user_secret=e9e40a7e
Re: [agi] AGI introduction
Pei Wang wrote: Hi, I put a brief introduction to AGI at http://nars.wang.googlepages.com/AGI-Intro.htm , including an AGI Overview followed by Representative AGI Projects. This looks pretty good to me. My compliments. (And now the inevitable however...) However, the distinction you intended between capability and principle did not become clear to me until I looked at the very last table, which classified AI architectures. I was initially quite surprised to see AIXI listed as principle and Cyc listed as capability. I had read capability - to solve hard problems as meaning the power to optimize a utility function, like the sort of thing AIXI does to its reward button, which when combined with the unified column would designate an AI approach that derived every element by backward chaining from the desired environmental impact. But it looks like you meant capability in the sense that the designers had a particular hard AI subproblem in mind, like natural language. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=231415user_secret=e9e40a7e
Re: [agi] What's wrong with being biased?
Stefan Pernar wrote: Now why is that? Cognitive biases could be... a) ...less fit characteristics of human cognition that did not pose too big a problem for humanity to make it to the current day (like an infection prone appendix of the mind - bad but not too bad). b) ...fitness increasing characteristics of human cognition that proved beneficial for our ancestors in the course of evolution but that have lost their value in modern times (like our craving for burgers and fries that back in the days prevented our ancestors from dying should the next harvest not go so well but are causing all kind of obesity related issues in a post caloric scarcity society) c) ...fitness increasing characteristics of human cognition that are just as valid today as they were over the course of evolution. It sounds like you're confusing cognitive bias with inductive bias. Cognitive biases are nonadaptive side effects of the adaptive heuristics that give rise to them. Some of these may be ineradicable even for superintelligences, such as the conjunction fallacy effect of using an approximation to Solomonoff induction, wherein an effect plus a compact reason may be deemed more probable than the effect alone with no compact reason found as yet. You may find it helpful to read some the following posts, which disentangle the widely different meanings of statistical bias, inductive bias, and cognitive bias. Statistical bias: http://www.overcomingbias.com/2007/03/statistical_bia.html http://www.overcomingbias.com/2007/04/useful_statisti.html Inductive bias: http://www.overcomingbias.com/2007/04/inductive_bias.html http://www.overcomingbias.com/2007/04/priors_as_mathe.html Cognitive bias: http://www.overcomingbias.com/2006/11/whats_a_bias_ag.html -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=231415user_secret=e9e40a7e
Re: [agi] What is the complexity of RSI?
Mark Waser wrote: So the real question is what is the minimal amount of intelligence needed for a system to self-engineer improvments to itself? Some folks might argue that humans are just below that threshold. Humans are only below the threshold because our internal systems are so convoluted and difficult to change. And because we lack the cultural knowledge of a theory of intelligence. But are probably quite capable of comprehending one. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244id_secret=48595571-b0508a
Re: Economic libertarianism [was Re: The first-to-market effect [WAS Re: [agi] Religion-free technical content]
J. Andrew Rogers wrote: Generally though, the point that you fail to see is that an AGI can just as easily subvert *any* power structure, whether the environment is a libertarian free market or an autocratic communist state. The problem has nothing to do with the governance of the economy but the fact that the AGI is the single most intelligent actor in the economy however you may arrange it. You can rearrange and change the rules as you wish, but any economy where transactions are something other than completely random is an economy that can be completely dominated by AGI in short order. The game is exactly the same either way, and more rigid economies have much simpler patterns that make them easier to manipulate. Regulating economies to prevent super-intelligent actors from doing bad things is rearranging the deck chairs on the Titanic. Succinctly put. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence - This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/?member_id=8660244id_secret=51662113-7b9e18
