---------------------------------------------------------------- BEFORE YOU POST, search the faq at <http://java.apache.org/faq/> WHEN YOU POST, include all relevant version numbers, log files, and configuration files. Don't make us guess your problem!!! ---------------------------------------------------------------- The problem we found was that between our application, the web process gang, the JVM, JServ and the OS... we were running out of file descriptors and their associated resources. The solution was to recomplie the Linux kernel and here are the step-by-step instructions for what we did: Change the following C header files. In /usr/include/linux/tasks.h, change: NR_TASKS 512 -> 4090 MAX_TASKS_PER_USER (NR_TASKS/2) -> NR_TASKS In /usr/include/linux/limits.h, change: NR_OPEN 1024 -> 4096 OPEN_MAX 256 -> 4096 In /usr/include/linux/posix_types.h, change: __FD_SETSIZE 1024 -> 4096 In /usr/include/bits/types.h, change: __FD_SETSIZE 1024 -> 4096 To allow users to increase their file descriptor limits, change the following configuration files: In /etc/security/limits.conf, add the lines: * soft nofile 1024 * hard nofile 4096 In /etc/pam.d/login, add: session required /lib/security/pam_limits.so To increase the system-wide file descriptor limit, add the following three lines to the /etc/rc.d/rc.local startup script: # Increase system-wide file descriptor limit. echo 8192 > /proc/sys/fs/file-max echo 24576 > /proc/sys/fs/inode-max Change to the directory /usr/src/linux and set up the kernel sources with: make mrproper Make any other kernel configuration changes, such as setting the processor family type, by entering the command below. Save your changes and exit the configuration panel when you are done. make xconfig Prepare for the kernel build by entering: make dep make clean Rebuild your kernel with the commands: make bzImage make modules Backup your old modules directory by entering (for example): mv /lib/modules/2.2.5-15 /lib/modules/2.2.5-15.old Create a new set of kernel modules with the command: make modules_install Backup your old kernel by entering (for example): mv /boot/vmlinuz /boot/vmlinuz.old Copy over your new kernel image with: cp /usr/src/linux/arch/i386/boot/bzImage /boot/vmlinuz Edit the Linux loader configuration in /etc/lilo.conf so that your new kernel is the default and your old kernel is still available from the boot prompt, as in the example shown below: boot=/dev/hda map=/boot/map install=/boot/boot.b prompt timeout=50 image=/boot/vmlinuz label=linux root=/dev/hda1 read-only image=/boot/vmlinuz.old label=linux.old root=/dev/hda1 read-only Generate the new Linux loader configuration by running: /sbin/lilo Shutdown and reboot with your new kernel by entering: shutdown -r now After rebooting, verify that the changes were effective. --- Nick Bauman <[EMAIL PROTECTED]> wrote: > ---------------------------------------------------------------- > BEFORE YOU POST, search the faq at > <http://java.apache.org/faq/> > WHEN YOU POST, include all relevant version numbers, > log files, > and configuration files. Don't make us guess your > problem!!! > ---------------------------------------------------------------- > > We have a very high traffic website that is > increasing > in it's traffic. Yesterday we had 41,000 unique > visitors for over 2.9 million "hits". That number is > expceted to increase over the coming weeks. > > The problem is we are seeing as many as 500-750 of > WAIT status in netstat -a, so naturally we wanted to > up the max children to 500-750 to accommodate this > observation. When we do this, we see a marked > increase > in the number of concurrent requests we can handle > and > much better performance of the app, but as we break > the 200 concurrent connections mark, JServ just > gives > up with the java process gang just disappearing from > the process table, leaving no core and nothing in > the > logs. Of course we are running using Native threads. > We are limping along right now under the load trying > to load balance servers right now with a Local > Director. We have tried: > > 1) increasing the max stack and max heap size to > 2/256MB > 2) switched VMs and OSs > 3) turned off JIT > > ENVIRONMENT: > > Software: Apache 1.3.9 with JServ 2.0 as a DSO > > Operating System: tried on Solaris 2.7 and Linux > RH6.0 > > JVM: Sun's JDK 1.1.8 and IBM's JDK 1.1.8 for Linux > > Hardware: Sun E350 with 512k RAM and Compaq Proliant > PIII dual 500 with 512k RAM > > > > ===== > # Nick Bauman > # Technical Programmer > # http://webhelp.com > # real people, real answers, real time > __________________________________________________ > Do You Yahoo!? > Thousands of Stores. Millions of Products. All in > one place. > Yahoo! Shopping: http://shopping.yahoo.com > > > -- > -------------------------------------------------------------- > Please read the FAQ! <http://java.apache.org/faq/> > To subscribe: > [EMAIL PROTECTED] > To unsubscribe: > [EMAIL PROTECTED] > Archives and Other: > <http://java.apache.org/main/mail.html> > Problems?: [EMAIL PROTECTED] > > ===== # Nick Bauman # Technical Programmer # http://webhelp.com # real people, real answers, real time __________________________________________________ Do You Yahoo!? Thousands of Stores. Millions of Products. All in one place. Yahoo! Shopping: http://shopping.yahoo.com -- -------------------------------------------------------------- Please read the FAQ! <http://java.apache.org/faq/> To subscribe: [EMAIL PROTECTED] To unsubscribe: [EMAIL PROTECTED] Archives and Other: <http://java.apache.org/main/mail.html> Problems?: [EMAIL PROTECTED]
