Ben,
Again, I'll omit the positive comments. ;-)
*. page 6 of ITSSIM, rule R: within it, "P is the probability that R holds for me" --- isn't it a self-reference? What do you mean by "me" here? Is it the system, the rule, or the current application of the rule?
*. Also, you said "it defines a quantitative measure of safety S", but I don't see the "safety" part, or you just mean that "if the future is sufficiently similar to the past, then it is safe". However, that is a very limited sense of "safety" --- it is more like "conservation". Do you really want your AGI to act to make the future more similar to the past?
*. Furthermore, you interpreted R as "I will only take an action if I can prove that it is expected not to decrease safety." Here "I can prove" is way too strong a phrase. Why not just use "I belief" or "according to my current opinion"? To call this approach "theorem-proving" gives people the impression that the conclusions are derived from a constant set of axioms, which is not really the case for safety judgments, as shown by the examples you discussed.
*. page 8 of ITSSIM: "the (tricky) goal for AGI design should be to create an AGI system that is superhumanly smart at this particular type of theorem-proving, but not capable of modifying itself to do anything besides do this type of theorem-proving and discuss its
results and its progress with humans." --- to me, this is both theoretically unnecessary and practically impossible. If the AGI is really "superhumanly smart", then it may see long-term results that we cannot even comprehensive, again as you suggested in the previous discussions.
*. The GoalGeneralization paper: I don't think the evolution of goals in an AGI can be limited to "generalization" only. In many cases, it is actually specialization, or derivation in general.
Pei
Ben Goertzel wrote:
Hi,
I have written two essays on the nature of future AGI's that are smart enough to use theorem-proving to guide their every action:
http://www.realai.net/ITSSIM.pdf
http://www.realai.net/GoalGeneralization.pdf
Yes, we are a long way from there now! But it's always fun to speculate.... The two essays are a series; by the end of the second one things get fairly interesting...
-- Ben G
-------
To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
-------
To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
