For the record ... on vcs, in /etc/xinetd.d, I've just changed svn and cvspserver and git-cvspserver to invoke their daemons with nice+timeout, as we have done already with bzr and git. There were tons of stale svnserve processes (I killed them), and I noticed the psserver processes were often near the top of the list.
It looks like gitweb is the biggest frequent resource consumer now. Will see about nicing that. However, I have a feeling that since vcs supports so many projects, and vc is an intrinsically resource-intensive operation, we're not going to be able to get rid of all the high loads. Unless perhaps the disk I/O can be somehow be configured better at a low level; I don't know about that. The virtualization setup is not something I am familiar with. I suspect the timeouts people are seeing on frontend are actually due to high loads on vcs. k
