On Thu, Jan 17, 2002 at 10:20:00AM -0500, Greg Ames wrote: [snip] > As a matter of fact, I typically listen on two ports while testing to disable > S_L_U_A, so I can easily figure out which process will get the next connection > in case I want to gdb it. While trying out ktrace on my test config, I saw that > the fcntl() accept mutex has got a thundering herd problem on daedalus. After > releasing the fcntl mutex, you see the kernel context switching to all of the > idle httpd processes. The first process that wakes up gets the mutex, the rest > of the context switches simply burn a little CPU, then block again. Moral: the > default cure looks as bad as the disease.
Is there anywhere else that we've started using cross process locks since 2.0.28? If fcntl() is known to cause this behaviour, why is it enabled at all on this version of FreeBSD? Based on your ktrace output from a couple days ago, I have a working theory that I have yet to reproduce: I noticed that there is a very high occurance of sendfile returning with errno 35 (Resource temporarily unavailable). The sendfile(2) man page reveals that this means sendfile was called in non-blocking mode, and the kernel buffers have filled up. A quick glance at sendfile_it_all in server/core.c looks to me like it would loop repeatedly on that error condition, as we are seeing it in the ktrace output. This means that on low-bandwidth connections like modems, we'll burn cycles testing if the kernel buffer is ready for more data while the modem still hasn't pulled the next packet down. I haven't been able to reproduce this, since I am not exactly sure what conditions are required to trigger sendfile. Can someone give me a URL that will consistently trigger the use of sendfile? Unfortunately, I don't think this will account for the short bursts of run queue growth we're talking about here, but it is something to look into. -aaron
