I've been trying to track down a issue on the clamav mirror we run here. Originally we ran the clamav mirror on FreeBSD 4.11 with Apache 2.2; we're now on a FreeBSD 7.0 server. Over time we notice that tons of connections end up "stuck" in FIN_WAIT_1 (TCP half-closed).
What we end up seeing is something like the below. The connection goes to close, but the clients fail to respond properly. The server sends ACK's to the client, which the client responds with a 0 sized window. This connection just sticks around forever, until manually cleared or the server is restarted. When this happens, it holds up socket space to the server, when thousands collect, the server runs out of socket space. I'm sorry I could not catch one of these completely from start to finish. (server is very busy) These connections are very random, but happen quite often. Talking with the FreeBSD team and devs, the server is responding correctly and normal methods to close the socket fail because it's still seen as an active TCP connection. When looking at several of these clients in the log, it appears they're all freshclam (clamav) clients, but it's hard to tell. I thought maybe using p0f to OS fingerprint these would be useful, but again, it's just so much traffic and so completely random, it would be hard to track that down. The only thing I can think of is that these are freshclam processes running in deamon mode? Is it possible the socket is not closing properly/completely on the client? FreeBSD devs pointed out that it's probably because the client has run out of socket space? I'm wondering if this is causing problems on any other mirror servers out there? 1.1.1.1 = server 2.2.2.2 = client 0:13:07.640426 IP 1.1.1.1.80 > 2.2.2.2.33379: . 4208136508:4208136509(1) ack 1471446041 win 520 <nop,nop,timestamp 3019088951 5004131> 20:13:07.736505 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5022148 3019088951> 20:14:07.702647 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019148951 5022148> 20:15:07.764920 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019208951 5022148> 20:15:07.860988 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5058183 3019208951> 20:16:07.827262 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019268951 5058183> 20:16:07.923341 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5076200 3019268951> 20:17:07.889690 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019328951 5076200> 20:17:07.984770 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5094217 3019328951> 20:18:07.952167 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019388951 5094217> 20:18:08.048249 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5112234 3019388951> 20:19:08.014715 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019448951 5112234> 20:19:08.110803 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5130252 3019448951> 20:20:08.077321 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019508951 5130252> 20:21:08.139995 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019568951 5130252> 20:21:08.236063 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5166286 3019568951> 20:22:08.202435 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019628951 5166286> 20:22:08.297499 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5184303 3019628951> 20:23:08.264631 IP 1.1.1.1.80 > 2.2.2.2.33379: . 0:1(1) ack 1 win 520 <nop,nop,timestamp 3019688951 5184303> 20:23:08.360700 IP 2.2.2.2.33379 > 1.1.1.1.80: . ack 0 win 0 <nop,nop,timestamp 5202321 3019688951> -- Robert Blayzor, BOFH INOC, LLC [EMAIL PROTECTED] http://www.inoc.net/~rblayzor/ _______________________________________________ http://lurker.clamav.net/list/clamav-devel.html Please submit your patches to our Bugzilla: http://bugs.clamav.net
