OutOfMemoryError under high load
Hello, We're running into the following problem when our Tomcat instances are under heavy load. SEVERE: Caught exception executing [EMAIL PROTECTED], terminating thread java.lang.OutOfMemoryError: unable to create new native thread The exception is always on thread creation, and correlates a condition where the process count on the machine suddenly starts to ramp up from a few hundred to a little above 1000 processes. (Load is fairly constant during this time, and the condition always expresses itself at right around the same level of traffic, about 6Mbps of throughput per server.) There appears to be plenty of heap memory available when this happens, so that coupled with the thread creation message leads us to suspect that we're running into an OS tuning problem, instead of specifically a problem with our app. (A radical rewrite of how our domain objects are cached, and tuning of various caches to various sizes, resulted in near identical performance, aside from large differences in CPU utilization. This seems to further support this conclusion.) Tuning maxProcessors and a variety of other parameters in our app seems to have no discernible effect, neither beneficial nor detrimental, beyond changes in CPU utilization, which always falls in a band between 10 and 60%, depending on the tuning choices. We're currently using Tomcat 4.1.13 under Linux 2.4.17 with Sun JDK 1.4.1_01. We've tuned all the obvious parameters, (ulimit -n and /proc/sys/kernel/threads-max, for example,) or at least all the ones we could think of. I think we're missing something obvious, and the OS is not letting us spawn more threads. Anyone know what we're doing wrong? Thanks in advance, -Ian McFarland - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError under high load
Howdy, How much have you played around with the JVM tuning parameters? Yoav Shapira Millennium ChemInformatics -Original Message- From: Ian McFarland [mailto:[EMAIL PROTECTED] Sent: Wednesday, June 11, 2003 8:05 AM To: Tomcat List Cc: Ian McFarland Subject: OutOfMemoryError under high load Hello, We're running into the following problem when our Tomcat instances are under heavy load. SEVERE: Caught exception executing [EMAIL PROTECTED], terminating thread java.lang.OutOfMemoryError: unable to create new native thread The exception is always on thread creation, and correlates a condition where the process count on the machine suddenly starts to ramp up from a few hundred to a little above 1000 processes. (Load is fairly constant during this time, and the condition always expresses itself at right around the same level of traffic, about 6Mbps of throughput per server.) There appears to be plenty of heap memory available when this happens, so that coupled with the thread creation message leads us to suspect that we're running into an OS tuning problem, instead of specifically a problem with our app. (A radical rewrite of how our domain objects are cached, and tuning of various caches to various sizes, resulted in near identical performance, aside from large differences in CPU utilization. This seems to further support this conclusion.) Tuning maxProcessors and a variety of other parameters in our app seems to have no discernible effect, neither beneficial nor detrimental, beyond changes in CPU utilization, which always falls in a band between 10 and 60%, depending on the tuning choices. We're currently using Tomcat 4.1.13 under Linux 2.4.17 with Sun JDK 1.4.1_01. We've tuned all the obvious parameters, (ulimit -n and /proc/sys/kernel/threads-max, for example,) or at least all the ones we could think of. I think we're missing something obvious, and the OS is not letting us spawn more threads. Anyone know what we're doing wrong? Thanks in advance, -Ian McFarland - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError under high load
Linux is the problem here. As you increase the mx JVM flag to be, the number of threads the java process can create decreases. This not a tomcat thing but a java on linux thing. For example, setting mx to be 1028 MBytes results in a thread ceiling of something like 427, which is really low for any loaded tomcat server. Unfortunately I don't have any solutions. You could switch to Unix, but that's hardly practical solution. I guess you could run two tomcat instances on the same server and load-balance somehow, but that sounds tough too. I never solved this one myself. -Original Message- From: Ian McFarland [mailto:[EMAIL PROTECTED] Sent: Wednesday, June 11, 2003 5:05 AM To: Tomcat List Cc: Ian McFarland Subject: OutOfMemoryError under high load Hello, We're running into the following problem when our Tomcat instances are under heavy load. SEVERE: Caught exception executing [EMAIL PROTECTED], terminating thread java.lang.OutOfMemoryError: unable to create new native thread The exception is always on thread creation, and correlates a condition where the process count on the machine suddenly starts to ramp up from a few hundred to a little above 1000 processes. (Load is fairly constant during this time, and the condition always expresses itself at right around the same level of traffic, about 6Mbps of throughput per server.) There appears to be plenty of heap memory available when this happens, so that coupled with the thread creation message leads us to suspect that we're running into an OS tuning problem, instead of specifically a problem with our app. (A radical rewrite of how our domain objects are cached, and tuning of various caches to various sizes, resulted in near identical performance, aside from large differences in CPU utilization. This seems to further support this conclusion.) Tuning maxProcessors and a variety of other parameters in our app seems to have no discernible effect, neither beneficial nor detrimental, beyond changes in CPU utilization, which always falls in a band between 10 and 60%, depending on the tuning choices. We're currently using Tomcat 4.1.13 under Linux 2.4.17 with Sun JDK 1.4.1_01. We've tuned all the obvious parameters, (ulimit -n and /proc/sys/kernel/threads-max, for example,) or at least all the ones we could think of. I think we're missing something obvious, and the OS is not letting us spawn more threads. Anyone know what we're doing wrong? Thanks in advance, -Ian McFarland - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError under high load
Do you have any native code components? I had a serious problem with OutOfMemoryError: unable to create new native thread errors too. They pretty much went away when I ditched the Oracle OCI driver in favor of the thin driver. BTW, the default linux kernel puts a hard limit of ~1k threads on your processes. A simple java program that does nothing but spawn threads typically tops out around 970-980 threads on linux. From what our ops guys have said, this parameter cannot be changed without recompiling the kernel. Why on earth would you want more than 1000 threads? Surely you don't need that many Ajp/Coyote/whatever processors. Jeff Schnitzer [EMAIL PROTECTED] The Sims Online -Original Message- From: Ian McFarland [mailto:[EMAIL PROTECTED] Sent: Wednesday, June 11, 2003 5:05 AM To: Tomcat List Cc: Ian McFarland Subject: OutOfMemoryError under high load Hello, We're running into the following problem when our Tomcat instances are under heavy load. SEVERE: Caught exception executing [EMAIL PROTECTED], terminating thread java.lang.OutOfMemoryError: unable to create new native thread The exception is always on thread creation, and correlates a condition where the process count on the machine suddenly starts to ramp up from a few hundred to a little above 1000 processes. (Load is fairly constant during this time, and the condition always expresses itself at right around the same level of traffic, about 6Mbps of throughput per server.) There appears to be plenty of heap memory available when this happens, so that coupled with the thread creation message leads us to suspect that we're running into an OS tuning problem, instead of specifically a problem with our app. (A radical rewrite of how our domain objects are cached, and tuning of various caches to various sizes, resulted in near identical performance, aside from large differences in CPU utilization. This seems to further support this conclusion.) Tuning maxProcessors and a variety of other parameters in our app seems to have no discernible effect, neither beneficial nor detrimental, beyond changes in CPU utilization, which always falls in a band between 10 and 60%, depending on the tuning choices. We're currently using Tomcat 4.1.13 under Linux 2.4.17 with Sun JDK 1.4.1_01. We've tuned all the obvious parameters, (ulimit -n and /proc/sys/kernel/threads-max, for example,) or at least all the ones we could think of. I think we're missing something obvious, and the OS is not letting us spawn more threads. Anyone know what we're doing wrong? Thanks in advance, -Ian McFarland - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
