hello. One algorithm I think that might be a good option and would address the concerns that I've seen on this topic is to kill the process with the latest start time when looking for resources to free. Obviously this would fail in the case where a long-running process suddenly starts consuming memory, but in general, it seems that badly behaved processes would typically begin behaving badly from the beginning of their inception. In any case, it should address the issue of X or syslogd getting killed, since they would often have older start times than processes that cause the trouble.
-Brian