Re: [agi] Re: Superrationality
John Oh wrote: And did I hear you correctly that you also believe Susan should cooperate in a standard one-shot Prisoner's Dilemma (assuming she believes there is a high enough probability that the opposing player is sufficiently similar to her)? Correct. For example, if Susan is facing her synchronized Copy, she should definitely cooperate. Against her desynchronized Copy she will be unable to prove anything if she is a human-level intelligence, but she should still intuitively cooperate. A superintelligence can prove similarity of decision systems that are not copies but which implement an identical algorithm at a deep level, *or* prove similarity of a decision system that contains an explicit dependency on an accurate model of the superintelligence. In other words, the superintelligence SI-A can decide to cooperate with SI-B because SI-B decided to cooperate if it modeled SI-A as cooperating and SI-A knows this. This gives SI-B a motive to decide to cooperate if it models SI-A as cooperating. That, roughly speaking, is how two dissimilar SIs would cooperate on the oneshot PD if they can obtain knowably reliable information about each other - through random sampling of computing elements, historical modeling, or even a sufficiently strong prior probability. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Re: Superrationality
Not the baby-halving threat, actually. http://www.geocities.com/eganamit/NoCDT.pdf Here Solomon's Problem is referred to as The Smoking Lesion, but the formulation is equivalent. Thanks for the reference. The paper is entertaining, in that both the theories presented (evidential decision theory and causal decision theory) are patently very stupid right from the outset ;-) EDT's foolishness is more mathematical in nature (via setting up the problem mathematically in a way that ignores relevant information) whereas CDT's foolishness is more philosophical in nature (essentially, via introducing the folk-psychology notion of causality which has no role in rigorous formal analyses of events). I really think this stuff is not that complicated; but people seem to be misled in thinking about it via commonplace illusions related to free will ... more later ben --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: Re: [agi] Re: Superrationality causality
I am trying to understand this issue... Isn't Causality Decision Theory the basis of legal law Decision? Someone was the Cause of the accident therefore they were the Cause of the accident... even some courts award based upon the mediating circumstances of those causes... Who gets what part of the blame? The factors within any given Knowledge base are causal for the expert opinions and the level of any AI system using that knowledge base. The classical example was given in medical knowledge... When the patent is bleeding, attach a tourniquet between the blood loss wound and the heart... If the subject is bleeding from the leg or foot, apply tourniquet to upper part of leg. If the subject is bleeding from the arm, apply the tourniquet above the wound. If the subject is bleeding from the head apply the tourniquet around the neck... woops that logic has dire causal consequences... but it does stop blood loss I can't breath! Dan Goe From : Ben Goertzel [EMAIL PROTECTED] To : agi@v2.listbox.com Subject : Re: [agi] Re: Superrationality Date : Fri, 26 May 2006 08:05:16 -0400 Not the baby-halving threat, actually. http://www.geocities.com/eganamit/NoCDT.pdf Here Solomon's Problem is referred to as The Smoking Lesion, but the formulation is equivalent. Thanks for the reference. The paper is entertaining, in that both the theories presented (evidential decision theory and causal decision theory) are patently very stupid right from the outset ;-) EDT's foolishness is more mathematical in nature (via setting up the problem mathematically in a way that ignores relevant information) whereas CDT's foolishness is more philosophical in nature (essentially, via introducing the folk-psychology notion of causality which has no role in rigorous formal analyses of events). I really think this stuff is not that complicated; but people seem to be misled in thinking about it via commonplace illusions related to free will ... more later ben --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED] --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Re: Superrationality
On 5/25/06, Eliezer S. Yudkowsky [EMAIL PROTECTED] wrote: Ben Goertzel wrote: I wonder if anyone knows of any mathematical analysis of superrationality. I worked out an analysis based on correlated computational processes - you treat your own decision system as a special case of computation and decide as if your decision determines the output of all computations that are similar to the decision. Or to put it another way, you don't choose as if you believe that multiple instantiations of an identical abstract computation can have different outputs. This can be formalized by extending Judea Pearl's causal graphs to include uncertainty about abstract computations, and reworking Pearl's surgical formalism for acts accordingly, which in turn is justified by considerations that these margins are too small to include. I haven't published this, but I believe I mentioned it on AGI during a discussion of AIXI. Can you restate with some math or some examples? --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Re: Superrationality
Tell me if this is also a superrationality-type issue: I commented to Eliezer that, during the last panel of the conference, I looked around for Eliezer didn't find him, and wondered if there was a bomb in the room. He replied something to the effect that he has a strong committment to ethics. This, of course, is exactly what concerned me. A person who is either not very rational, or not very ethical, can be relied on to operate within certain parameters. A person who is committed to doing whatever his computations direct, however, may be busy caring for orphaned puppies one day, and then, because he changed his estimation of some prior from 0.5 to 0.6, go out and blow up an AI conference the next day. (Perhaps this is part of why humans seem to have an evolved distrust of overly smart people.) It seems to me that there are societal inefficiencies in this approach. AFAIK, the Bayesian formalism doesn't consider things such as how irreversible the effects of an action are if it turns out to be wrong, or the advantages from cooperation if everyone biases their actions to be more like those of others (and hence stops blowing up everyone else's conferences). I think that if you posited a society of Bayesian reasoners, they would have higher total utility if they agreed on some rules, guidelines, or values. Perhaps the problem with violence in the Middle East is that the combatants are overly rational. A Bayesian reasoner might reason out, given the idea, that it is logical to construct a society with mores and laws. Is that an answer to the PD superrationality problem - that the Bayesian reasoner reasons that his utility will be maximized if everyone passes a law that cooperation is mandatory? --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
[agi] Re: Superrationality
Ben Goertzel wrote: Thanks for the reference. The paper is entertaining, in that both the theories presented (evidential decision theory and causal decision theory) are patently very stupid right from the outset ;-) EDT and CDT have been the two dominant decision theories, with CDT having the upper hand, for decades. I agree that both are wrong, but it is an audacious assertion. I haven't written up my own mathematical analysis because it would require on the order of a book to put forth an alternative theory in academia. I just did the analysis for myself because I needed to know if I had to do any special work in setting up the initial conditions of an FAI. EDT's foolishness is more mathematical in nature (via setting up the problem mathematically in a way that ignores relevant information) whereas CDT's foolishness is more philosophical in nature EDT and CDT are precisely symmetrical except in how they compute counterfactual probabilities. (essentially, via introducing the folk-psychology notion of causality which has no role in rigorous formal analyses of events). Causality a folk-psychology notion? Judea Pearl begs to disagree with you, and I beg to agree with Judea Pearl. My own theory is causal in nature - that is, it uses Pearl's graphs. I really think this stuff is not that complicated; but people seem to be misled in thinking about it via commonplace illusions related to free will ... The answer itself is simple. Justifying it is not. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
Re: [agi] Re: Superrationality
Pearl's book on causality is a nice one, but I don't think that his idea of causality fully encompasses the (very useful) folk-psychology notion of causation. Basically, the crux of Pearl's definition of causality rests on his distinction between P(Y|X) and what he calls P(Y | do(X) ) This is certainly important, and any AGI system had better be able to reason separately and effectively about these two sorts of conditional probabilities. But it's not all there is to the human notion of causation... And, more to the point, I don't think Pearl's (or any other) model of causality is *necessary* for understanding the various logical puzzles and paradoxes we've been discussing in this thread, though perhaps it may provide a useful perspective. More later, Ben On 5/26/06, Eliezer S. Yudkowsky [EMAIL PROTECTED] wrote: Ben Goertzel wrote: Thanks for the reference. The paper is entertaining, in that both the theories presented (evidential decision theory and causal decision theory) are patently very stupid right from the outset ;-) EDT and CDT have been the two dominant decision theories, with CDT having the upper hand, for decades. I agree that both are wrong, but it is an audacious assertion. I haven't written up my own mathematical analysis because it would require on the order of a book to put forth an alternative theory in academia. I just did the analysis for myself because I needed to know if I had to do any special work in setting up the initial conditions of an FAI. EDT's foolishness is more mathematical in nature (via setting up the problem mathematically in a way that ignores relevant information) whereas CDT's foolishness is more philosophical in nature EDT and CDT are precisely symmetrical except in how they compute counterfactual probabilities. (essentially, via introducing the folk-psychology notion of causality which has no role in rigorous formal analyses of events). Causality a folk-psychology notion? Judea Pearl begs to disagree with you, and I beg to agree with Judea Pearl. My own theory is causal in nature - that is, it uses Pearl's graphs. I really think this stuff is not that complicated; but people seem to be misled in thinking about it via commonplace illusions related to free will ... The answer itself is simple. Justifying it is not. -- Eliezer S. Yudkowsky http://singinst.org/ Research Fellow, Singularity Institute for Artificial Intelligence --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED] --- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
