Thanks all for your replies to my question. Because of the nature of our application, we can't really load everything at start up, but I did some digging and there are clearly some inefficiencies here and the situation would indeed be improved by cleaning these up. However, it does look like it may be the interaction between these inefficiencies and freebsd in particular that is causing the error.
So, here's what I did: on a linux development box I wrote a script using Linux::Inotify2 to watch the directories containing the code for our application that we (think we) have to load at run time. A typical request generated an average of 225 IN_OPEN inotify events. (FWIW, it also generated around 850 IN_ACCESS events. I don't understand the difference, but IN_OPEN events seem more relevant to what I'm getting to.) Now, we have MaxClients set super high, but with just 150 concurrent requests that's 33,750 open files and that's not taking into account files opened for other processes on the machine, such as the kernel or the web server, etc. But, here's where freebsd comes in. The freebsd kernel has a bunch of tunables, one of which is kern.maxfiles. The freebsd manual says "This variable indicates the maximum number of file descriptors on your system... Each open file, socket, or fifo uses one file descriptor. A large-scale production server may easily require many thousands of file descriptors, depending on the kind and number of services running concurrently" Well, we had kern.maxfiles = 12328. Also possibly relevant is kern.ipc.somaxconn which "limits the size of the listen queue for accepting new TCP connections". We had that at the default of 128. I also don't understand the interaction between this setting and apache's MaxClients since when we had kern.ipc.somaxconn = 128 and MaxClients 150 we quickly maxed out max clients. So, we bumped up kern.maxfiles to 50000 and kern.ipc.somaxconn to 8192 and things appeared to be better last night under heavy load. But, as I said, there are clearly some in efficiencies too in the way our application loads code at run time, and cleaning these up would certainly help. If you're interested, here is a discussion from a free-bsd mailing list yesterday about low kernel limits that a colleague forwarded to me: http://goo.gl/Dpzre Thanks again for all of your replies, Jason On Wed, Oct 3, 2012 at 12:48 PM, Perrin Harkins <per...@elem.com> wrote: > On Wed, Oct 3, 2012 at 9:45 AM, Jason Aubrey <aubre...@gmail.com> wrote: > > Hi All, > > > > We have modperl application running on 64bit (2 cores), freebsd 8.2, > perl > > 1.14.1, modperl 2.0.5, apache 2.2.21, 24GB of ram and about 5.5k users. > > Additional system details are below. Under heavy load we are seeing > errors > > of the form > > > >> Failed to evaluate module Parser::Legacy: Attempt to reload > >> Parser/Legacy.pm aborted.\nCompilation failed in require at (eval 15635) > >> line 1.... > > I remember seeing this kind of thing when I was trying to use > ActiveState's mod_perl-ish thing with IIS years ago. It would fail > to find things like strict.pm when it was under load. I guessed the > same thing: some sort of concurrency issue. I don't have a solution > to offer though, since I gave up on that one and used CGI. > > - Perrin >
