On Wed, Jan 16, 2002 at 05:25:52PM -0800, Terry Lambert wrote: > Chad David wrote: > > The direct cause is a bug in my client. I call close(2) out side of the > > main loop (one line off :( ), so none of the client side sockets were > > getting closed. When I fixed this all of the connections went to > > TIME_WAIT right away. > > > > I'm still not convinced that all is well though, as on Solaris 5.9 and > > 4.4-STABLE I do not see the problem with the bad client. > > So it's the resource track close of the sockets.
Maybe, but it looks like TCP is ignoring the client FIN when sent from the local machine (see the below tcpdump). > > If the client and the server were the same program, you could > be seeing this as a timing thing on order of operation. I'm > guessing they aren't, though... No, they are seperate programs. I've written a sample client and server that demonstrate the issue (I can send them to you if you care). The problem occurs when the client does not close any of its sockets, and just runs until there are no more to be had. At that point it exits. The server accepts the connections, does one read(2), one write(2), and then closes the connection (in a seperate thread). The client was able to connect 3976 times before it fails, at which point it exits and I killed the server. The (cleaned up) output of tcpdump -S -i lo0 is: 15:17:08.742699 localhost.2233 > localhost.9999: S 1241089741:1241089741(0) win 65535 <mss 16344,nop,wscale 1,nop,nop,timestamp 11844203 0> (DF) 15:17:08.742741 localhost.9999 > localhost.2233: S 1530034189:1530034189(0) ack 1241089742 win 65535 <mss 16344,nop,wscale 1,nop,nop,timestamp 11844203 11844203> 15:17:08.742756 localhost.2233 > localhost.9999: . ack 1530034190 win 32767 <nop,nop,timestamp 11844203 11844203> (DF) 15:17:08.742987 localhost.2233 > localhost.9999: P 1241089742:1241089767(25) ack 1530034190 win 32767 <nop,nop,timestamp 11844203 11844203> (DF) 15:17:08.743233 localhost.9999 > localhost.2233: P 1530034190:1530034446(256) ack 1241089767 win 32755 <nop,nop,timestamp 11844203 11844203> (DF) 15:17:08.743340 localhost.9999 > localhost.2233: F 1530034446:1530034446(0) ack 1241089767 win 32755 <nop,nop,timestamp 11844203 11844203> (DF) 15:17:08.743374 localhost.2233 > localhost.9999: . ack 1530034447 win 32639 <nop,nop,timestamp 11844203 11844203> (DF) 15:20:19.793831 localhost.2233 > localhost.9999: F 1241089767:1241089767(0) ack 1530034447 win 32639 <nop,nop,timestamp 11863308 11844203> (DF) 15:21:23.793759 localhost.2233 > localhost.9999: F 1241089767:1241089767(0) ack 1530034447 win 32639 <nop,nop,timestamp 11869708 11844203> (DF) 15:22:27.792756 localhost.2233 > localhost.9999: F 1241089767:1241089767(0) ack 1530034447 win 32639 <nop,nop,timestamp 11876108 11844203> (DF) 15:23:31.791632 localhost.2233 > localhost.9999: F 1241089767:1241089767(0) ack 1530034447 win 32639 <nop,nop,timestamp 11882508 11844203> (DF) 15:25:39.791074 localhost.2233 > localhost.9999: R 1241089768:1241089768(0) ack 1530034447 win 32639 (DF) I don't know the code that well (which I'm working on), so I have to ask, what can be preventing the server socket (which has no process), from sending an ack before the client socket (which also has no process) gives up and sends a reset? > > > > I'll address your points below, but if you don't feel like chasing this > > anymore that is fine with me... I'll add it to my list of things to > > try and understand on my next vacation :). > > Unless there's something that jumps out at you, this is probably > a good plan. 8-). Something jumped out.. > > > > > You should probably call shutdown(2), if you want your code > > > to be mostly correct. > > > > Call shutdown(2) instead of close(2)? > > Nope. Before close. Depending on the argument, perhaps not > before the last read or write, then the close. I read Stevens, and now I understand... this whole adventure was worth it if for no other reason then I now understand this point! > > > > This indicates a 2MSL draining. The resource track close could > > > also be slow. You could probably get an incredible speedup by > > > doing explicit closes in the client program, starting with the > > > highest used fd, and working down, instead of going the other > > > way (it's probably a good idea to modify the FreeBSD resource > > > track close to so the same thing). > > > > If I had been doing any explicit closes :(. > > Yes, but your ordering is reverse optimal, actually, so you are > going to be rate limited at the client. > > Did the client actually exit? If it didn't, that would explain > everything. As you already know, the client does exit, but it looks to me like it is the server sockets problem, not the clients? > > > > I actually implemented something for this type of problem over Christmas > > with one of the Solaris engineers. It was inspired by Jeff Bonwick's > > vmem stuff (Usenix 2001), but was bit mask based, so the actual storage > > overhead was a lot less, with what appeared to be very good allocate and > > free times (O(n) as the worst case with O(1) typically). > > This would be nice for FreeBSD, assuming we could pry it out > of you. 8-). Assuming I can ever get it into a presentable state... between work family and FreeBSD I've decided to sleep when I'm dead. > > > [ ... timer code, Rice U. Opportunistic Timers ... ] > > > I think I have that paper around here somewhere... is it older, > > like from around 1990? > > No, you are probably thinking of the WRL paper by Jeff Mogul. > The paper I'm referring to is late mid-90's. I'll go find it. > > > > > > Nope. Stock -current, none of my patches applied. > > > > > > Heh... "not useful information without a date of cvsup, > > > and then possibly not even then". Moving target problems... > > > > The original email has the uname and a dmesg, but: > > FreeBSD colnta 5.0-CURRENT FreeBSD 5.0-CURRENT #17: Sun Jan 13 03:51:32 MST 2002 > > I would need to check it out, and build my own copy, and see > if I could repeat it (I'd need your broken client and your > server code. It would be much better to see if other well > known and tested versions of FreeBSD exhibited the same symptoms, > and, if not, track it down with a bsearch of the CVS tree by date. If need be I will do this tonight. I have a local cvs repo, and a nice fast devel. machine so it shouldn't take too long. > > > > Give that each connection is in its own thread this is very doable... > > This would at least isolate it to the client vs. server code > and order of operation. If it's the server close, then the > issue is perhaps a re-ACK after FIN ACK from the close making > the SYN cache think it has a new connection, then a re-FIN, > with the close, to get it into the strange state... ?? > > > > > If nothing else I'm learning... I just wish I could read as fast > > as you can type :). > > Heh. My max rate is 135 WPM, which is 6*135 CPM or 13.5 CPS, or > 135 BAUD, which is slow as molasses compared to most people's > read rates (by about 2 orders of magnitude). Just an indirect comment about your tendancy to reply in more detail that a lot other others, nothing else :). Again, thank you. -- Chad David [EMAIL PROTECTED] www.FreeBSD.org [EMAIL PROTECTED] ACNS Inc. Calgary, Alberta Canada Fourthly, The constant breeders, beside the gain of eight shillings sterling per annum by the sale of their children, will be rid of the charge of maintaining them after the first year. - Johnathan Swift To Unsubscribe: send mail to [EMAIL PROTECTED] with "unsubscribe freebsd-current" in the body of the message
