Hi Pei, > *. page 6 of ITSSIM, rule R: within it, "P is the probability that R > holds for me" --- isn't it a self-reference? What do you mean by "me" > here? Is it the system, the rule, or the current application of the rule?
The system > *. Also, you said "it defines a quantitative measure of safety S", but I > don't see the "safety" part, or you just mean that "if the future is > sufficiently similar to the past, then it is safe". However, that is a > very limited sense of "safety" --- it is more like "conservation". Do > you really want your AGI to act to make the future more similar > to the past? By "safety" in the ITSSIM context, I just mean "having each step taken based on extremely careful deliberation." > *. Furthermore, you interpreted R as "I will only take an action if I > can prove that it is expected not to decrease safety." Here "I can > prove" is way too strong a phrase. Why not just use "I belief" or > "according to my current opinion"? To call this approach > "theorem-proving" gives people the impression that the conclusions are > derived from a constant set of axioms, which is not really the case for > safety judgments, as shown by the examples you discussed. I actually mean literal theorem-proving, in the sense of Schmidhuber's Godel Machine. These judgments are based on a constant set of axioms, together with a variable environment, as in the Godel machine. The constant axioms, in order to guide reasoning based on the variable environment, must include rules for dealing with the zero-frequency problem and the "problem of induction." For instance, they could include an axiom instructing to use the Maximum Entropy Principle to assign unknown variable-values. This should be clarified in the text, I suppose. > *. page 8 of ITSSIM: "the (tricky) goal for AGI design should be to > create an AGI system that is superhumanly smart at this particular type > of theorem-proving, but not capable of modifying itself to do anything > besides do this type of theorem-proving and discuss its > results and its progress with humans." --- to me, this is both > theoretically unnecessary and practically impossible. If the AGI is > really "superhumanly smart", then it may see long-term results that we > cannot even comprehensive, again as you suggested in the previous > discussions. Yes, but superhumanly smart doesn't mean absolutely free. It may see long-term results we can't comprehend but if it is a piece of software architected according to the ITSSIM rule then it will have to keep obeying the ITSSIM rule forever, even so. > *. The GoalGeneralization paper: I don't think the evolution of goals in > an AGI can be limited to "generalization" only. In many cases, it is > actually specialization, or derivation in general. I think the paper would have been better called "Supergoal Generalization." If an AI has an overarching supergoal, then of course it will specialize this supergoal to form various subgoals, and it will then generalize some of these subgoals. The issue of that paper was: if an AI has an overarching supergoal, and then modifies itself to as to change its supergoal, what restrictions can we make as to how the new supergoal relates to the prior supergoal. My suggestion was that one could (and perhaps should) intentionally restrict this relationship to one of "generalization", in a certain sense. -- Ben ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
