On Thu, Oct 4, 2012 at 9:53 AM, demerphq <demer...@gmail.com> wrote: > On 4 October 2012 16:36, Jason Aubrey <aubre...@gmail.com> wrote: > > Thanks all for your replies to my question. > > > > Because of the nature of our application, we can't really load > everything at > > start up, but I did some digging and there are clearly some > inefficiencies > > here and the situation would indeed be improved by cleaning these up. > > However, it does look like it may be the interaction between these > > inefficiencies and freebsd in particular that is causing the error. > > > > So, here's what I did: on a linux development box I wrote a script using > > Linux::Inotify2 to watch the directories containing the code for our > > application that we (think we) have to load at run time. A typical > request > > generated an average of 225 IN_OPEN inotify events. (FWIW, it also > > generated around 850 IN_ACCESS events. I don't understand the > difference, > > but IN_OPEN events seem more relevant to what I'm getting to.) > > > > Now, we have MaxClients set super high, but with just 150 concurrent > > requests that's 33,750 open files and that's not taking into account > files > > opened for other processes on the machine, such as the kernel or the web > > server, etc. > > > > But, here's where freebsd comes in. The freebsd kernel has a bunch of > > tunables, one of which is kern.maxfiles. The freebsd manual says "This > > variable indicates the maximum number of file descriptors on your > system... > > Each open file, socket, or fifo uses one file descriptor. A large-scale > > production server may easily require many thousands of file descriptors, > > depending on the kind and number of services running concurrently" > > > > Well, we had kern.maxfiles = 12328. Also possibly relevant is > > kern.ipc.somaxconn which "limits the size of the listen queue for > accepting > > new TCP connections". We had that at the default of 128. I also don't > > understand the interaction between this setting and apache's MaxClients > > since when we had kern.ipc.somaxconn = 128 and MaxClients 150 we quickly > > maxed out max clients. > > > > So, we bumped up kern.maxfiles to 50000 and kern.ipc.somaxconn to 8192 > and > > things appeared to be better last night under heavy load. But, as I said, > > there are clearly some in efficiencies too in the way our application > loads > > code at run time, and cleaning these up would certainly help. > > You would also see reduced memory pressure and performance > improvements. Perl is actually pretty slow at compiling code (in > comparison to executing code), and code that is compiled prefork is > going to be shared, whereas code compiled post-fork is going to be > duplicated per process. > > By preloading you will both win performance improvements and reduce > memory pressure on your application. > > I strongly recommend you approach this subject aggressively. > > yves >
Hi all, Here's another update which you will be glad to hear because it proves you are awesome (or at least that you were right). The freebsd tuning I mentioned below helped but didn't fix the problem. We still got a bunch of such errors after this tuning (fewer, but still a lot). Well, I added just one file to the list of code compiled in at start up and since doing so we have had exactly zero such errors. Admittedly, that file pulls in a large package, but this single 8-character change to one line of code seems to have had a tremendously positive impact on the performance of our application. Thanks again for all of of your help, Jason
