Savannah Hackers, Ruben saw that the problem was a web crawler that is crawling fast but not closing connections effectively becoming a SYN attack. It also hit www.gnu.org yesterday. Therefore he blocked it site wide since it would just move on to yet another web server after this. I have been communicating with Ruben on IRC.
I am kicking myself that I should have been able to detect this problem by looking at the network state stats. The high number of connections in the SYN_RECV state were a dead giveaway to the problem. Plus the high rate of log entries. However the apache log configuration on vcs is abysmal and definitely not good. That is cleaned up on vcs0. It would have been easier to see this problem in the web logs there. With this everything seems to have returned to normal operation. Bob
