Hi, Having absorbed a bunch of commentary on my ITSSIM and GoalGeneralization draft papers, here are my current thoughts. Part of this is summary, but there are some new ideas at the end.
1) I continue to think ITSSIM is a valid method for providing a reasonable degree of assurance that an AI will continue to be careful as it self-modifies and self-improves. Of course, there are no guarantees with such things, as proved by the possibility that when an AI gets smarter than some particular value, superintelligent aliens will contact it and modify it. An interesting question to explore is how stable ITSSIM is if hardware errors or the like occur; I think the answer will be "quite stable." 2) I also think that it makes sense to impose a criterion of "Careful Supergoal Generalization" (which I now see is a better name than Goal Generalization). The issue here is, if an AI has an overarching supergoal, and then modifies itself to as to change its supergoal, what restrictions can we make as to how the new supergoal relates to the prior supergoal? My suggestion is that one could (and perhaps should) intentionally restrict this relationship to one of "careful generalization", in a certain sense. My definition of generalization in this context seems sound but may end up needing some refinement. 3) These two ideas can be combined naturally, if one assumes that the initial supergoal includes ITSSIM. I didn't quite say this in either of the papers but I now think this is a good way to phrase things. 4) I now agree with several commentators that the inclusion of explicit "emergency measures", as I suggested in the Goal Generalization essay, is a bad hack. Rather, I think it's better to use either a) pure ITSSIM, or b) careful supergoal generalization, with an initial supergoal that includes ITSSIM, and also includes emergency measures covering all known existential risks In case b, we are counting on the AI to generalize ITSSIM as seems appropriate, and also to carefully generalize its initial emergency measures as seems appropriate. 5) After reading Pei Wang's commentary, I see that the use of "theorem proving" in the context of proving theorems about empirical reality needs more elaboration. Specifically, it needs to be emphasized that the axioms of one's formal system must include an assumptive solution to the Humean "problem of induction". This is implicit in Schmidhuber's Godel Machine work but he doesn't elaborate on the point at all, so far as I can recall. I will product a new version of the paper (with the best parts of the goal generalization paper fused into the ITSSIM paper), when I get a chance, probably sometime late next week. Further constructive criticism is appreciated. I have also put a little thought into how these ideas could be explored using current AI technologies, prior to the advent of amazingly powerful theorem-provers. It seems to me that one could implement preliminary versions of both ITSSIM and careful supergoal generalization, by substituting "guessing" for "proving" in both algorithms. Of course this results in rather different algorithms -- but the qualitative nature of the dynamics of the evolving AI system might not be so different. At very least it would be an interesting sort of experiment. This kind of experiment could be done using an only moderately improved version of the current Novamente AI codebase. But of course, the Novamente project is in a pretty severe financial situation at this moment, so we have no resources to play with this sort of thing. Maybe next year ;-p -- Ben G ------- To unsubscribe, change your address, or temporarily deactivate your subscription, please go to http://v2.listbox.com/member/[EMAIL PROTECTED]
