Re: Any synchronization issues with SMP?
TC clustering uses GZIP(In|Out)putStream in replication. I don't know, if there is buffering on top. Filip? Rainer Martin Schulz wrote: OT: The following is advice to servlet developers rather than container developers. For what it's worth, just be careful when you use GZIP(In|Out)putStream, since the native code uses a mutex (for no good reason, imho), to prevent garbage collection from interfering with the array being used. Bottleneck where I didn't expect one. In particular, never place a Object(In|Out)putStream directly on top of the GZIP streams, always use a buffer stream between the two. Reason is that the object stream reads/writes in very small chunks, causing four system calls per chunk. Or else interesting things are going to happen on larger SMP systems. Martin - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Any synchronization issues with SMP?
Clustering is using GZIP byte only with byte array, so it doesnt link to an actual stream, only a byte array stream. Don't think we have a problem Houston! :) Filip - Original Message - From: Rainer Jung [EMAIL PROTECTED] To: Tomcat Developers List [EMAIL PROTECTED] Sent: Friday, July 02, 2004 3:59 AM Subject: Re: Any synchronization issues with SMP? TC clustering uses GZIP(In|Out)putStream in replication. I don't know, if there is buffering on top. Filip? Rainer Martin Schulz wrote: OT: The following is advice to servlet developers rather than container developers. For what it's worth, just be careful when you use GZIP(In|Out)putStream, since the native code uses a mutex (for no good reason, imho), to prevent garbage collection from interfering with the array being used. Bottleneck where I didn't expect one. In particular, never place a Object(In|Out)putStream directly on top of the GZIP streams, always use a buffer stream between the two. Reason is that the object stream reads/writes in very small chunks, causing four system calls per chunk. Or else interesting things are going to happen on larger SMP systems. Martin - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Any synchronization issues with SMP?
OT: The following is advice to servlet developers rather than container developers. For what it's worth, just be careful when you use GZIP(In|Out)putStream, since the native code uses a mutex (for no good reason, imho), to prevent garbage collection from interfering with the array being used. Bottleneck where I didn't expect one. In particular, never place a Object(In|Out)putStream directly on top of the GZIP streams, always use a buffer stream between the two. Reason is that the object stream reads/writes in very small chunks, causing four system calls per chunk. Or else interesting things are going to happen on larger SMP systems. Martin Martin Schulz wrote: it appears that the JVM slows everything down to a crawl, including the code path which should lead to another accept being called., for up to 8 minutes!!! Furthermore, the mpstat has the nice porperty that CPU usage adds up to exactly 100%, i.e. a single CPU is used... no more, no less. This corresponds to 12% or 13% CPU utilization shown in prstat based on 8 CPUs. My interpretation is that the JVM is effectively preventing parallel execution (which otherwise appears to work fine). Nearly all threads either wait, read from a Socket, or zip/unzip data. I'm not sure what all that means, but Tomcat appears to be a victim of it. I'll experiment some more. Main difference with the systems Rainer mentioned is the JVM (1.4.2_04) and the CPU (Sparc III 1.2GHz). If any of this rings a bell, drop me a note. I'll be happy to share data as appropriate. I'll repost to the list only if I learn anything which impacts Tomcat directly (other than that the code path to hand of the socket accept responsibility is not suitable for _very_ high hit rates, which does not worry me too much at this point). Cheers! Martin Martin Schulz wrote: Rainer, Thanks for the tips. I am about to take timing stats internally in the ThreadPool and the Tcp workers. Also, the described symptoms do not disappear, but seem to be of much shorter duration when only 4 CPUs are used for the application. I'll summarize what I find. Martin Rainer Jung wrote: Hi, we know one application running on 9 systems with 4 US II CPUs each under Solaris 9. Peak request rates at 20 requests/second per system. Tomcat is 4.1.29, Java is 1.3.1_09. No symptoms like yours! You should send a signal QUIT to the jvm process during the unresponsiveness time. This is a general JVM mechanism (at least for sun JVMs). The signal writes a stack trace for each thread on STDOUT (so you should also start tomcat with redirection of STDOUT the output to some file). Beware: older JVM in rare cases stopped working after getting this signal (not expected with 1.3.1_09). In this stack dump you should be able to figure out, in which methods most of your threads stay and what the status is. Is there native code included (via JNI)? Any synchronization done in the application itself? Are you using Tomcat clustering? Which JVM? Sincerely Rainer Jung - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Any synchronization issues with SMP?
it appears that the JVM slows everything down to a crawl, including the code path which should lead to another accept being called., for up to 8 minutes!!! Furthermore, the mpstat has the nice porperty that CPU usage adds up to exactly 100%, i.e. a single CPU is used... no more, no less. This corresponds to 12% or 13% CPU utilization shown in prstat based on 8 CPUs. My interpretation is that the JVM is effectively preventing parallel execution (which otherwise appears to work fine). Nearly all threads either wait, read from a Socket, or zip/unzip data. I'm not sure what all that means, but Tomcat appears to be a victim of it. I'll experiment some more. Main difference with the systems Rainer mentioned is the JVM (1.4.2_04) and the CPU (Sparc III 1.2GHz). If any of this rings a bell, drop me a note. I'll be happy to share data as appropriate. I'll repost to the list only if I learn anything which impacts Tomcat directly (other than that the code path to hand of the socket accept responsibility is not suitable for _very_ high hit rates, which does not worry me too much at this point). Cheers! Martin Martin Schulz wrote: Rainer, Thanks for the tips. I am about to take timing stats internally in the ThreadPool and the Tcp workers. Also, the described symptoms do not disappear, but seem to be of much shorter duration when only 4 CPUs are used for the application. I'll summarize what I find. Martin Rainer Jung wrote: Hi, we know one application running on 9 systems with 4 US II CPUs each under Solaris 9. Peak request rates at 20 requests/second per system. Tomcat is 4.1.29, Java is 1.3.1_09. No symptoms like yours! You should send a signal QUIT to the jvm process during the unresponsiveness time. This is a general JVM mechanism (at least for sun JVMs). The signal writes a stack trace for each thread on STDOUT (so you should also start tomcat with redirection of STDOUT the output to some file). Beware: older JVM in rare cases stopped working after getting this signal (not expected with 1.3.1_09). In this stack dump you should be able to figure out, in which methods most of your threads stay and what the status is. Is there native code included (via JNI)? Any synchronization done in the application itself? Are you using Tomcat clustering? Which JVM? Sincerely Rainer Jung - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Any synchronization issues with SMP?
The only time I have seen something like this happen is when we had a memory leak, an interesting one. We were running on solaris, our CPU, sometimes one, sometimes both would go up to 100% and stay there. sometimes for as much as 10 minutes and then go back to normal again. We ran thread analyzer (a neat util on solaris) to measure which thread was using a lot of cpu, and found out it was a VM thread. In our scenario, we had a thread being an inner class of an object, and the thread (inner class) referenced the parent and the parent referenced the thread. Even though nothing referenced any of the two objects, it would not be garbage collected, I suspect it had to do with the fact that the inner class was a thread object. Another guess we had was that the VM thread was freakin out cause it tried to resolve garbage collection references and kept looking our weird dependency. Once we resolved the memory leak (removed any references from the parent object to the thread and vice versa) our CPU usage never behaved weirdly again. hope this helps Filip -Original Message- From: Martin Schulz [mailto:[EMAIL PROTECTED] Sent: Friday, June 25, 2004 8:24 PM To: Tomcat Developers List Subject: Re: Any synchronization issues with SMP? it appears that the JVM slows everything down to a crawl, including the code path which should lead to another accept being called., for up to 8 minutes!!! Furthermore, the mpstat has the nice porperty that CPU usage adds up to exactly 100%, i.e. a single CPU is used... no more, no less. This corresponds to 12% or 13% CPU utilization shown in prstat based on 8 CPUs. My interpretation is that the JVM is effectively preventing parallel execution (which otherwise appears to work fine). Nearly all threads either wait, read from a Socket, or zip/unzip data. I'm not sure what all that means, but Tomcat appears to be a victim of it. I'll experiment some more. Main difference with the systems Rainer mentioned is the JVM (1.4.2_04) and the CPU (Sparc III 1.2GHz). If any of this rings a bell, drop me a note. I'll be happy to share data as appropriate. I'll repost to the list only if I learn anything which impacts Tomcat directly (other than that the code path to hand of the socket accept responsibility is not suitable for _very_ high hit rates, which does not worry me too much at this point). Cheers! Martin Martin Schulz wrote: Rainer, Thanks for the tips. I am about to take timing stats internally in the ThreadPool and the Tcp workers. Also, the described symptoms do not disappear, but seem to be of much shorter duration when only 4 CPUs are used for the application. I'll summarize what I find. Martin Rainer Jung wrote: Hi, we know one application running on 9 systems with 4 US II CPUs each under Solaris 9. Peak request rates at 20 requests/second per system. Tomcat is 4.1.29, Java is 1.3.1_09. No symptoms like yours! You should send a signal QUIT to the jvm process during the unresponsiveness time. This is a general JVM mechanism (at least for sun JVMs). The signal writes a stack trace for each thread on STDOUT (so you should also start tomcat with redirection of STDOUT the output to some file). Beware: older JVM in rare cases stopped working after getting this signal (not expected with 1.3.1_09). In this stack dump you should be able to figure out, in which methods most of your threads stay and what the status is. Is there native code included (via JNI)? Any synchronization done in the application itself? Are you using Tomcat clustering? Which JVM? Sincerely Rainer Jung - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] --- Incoming mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.706 / Virus Database: 462 - Release Date: 6/14/2004 --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.706 / Virus Database: 462 - Release Date: 6/14/2004 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Any synchronization issues with SMP?
Hi, we know one application running on 9 systems with 4 US II CPUs each under Solaris 9. Peak request rates at 20 requests/second per system. Tomcat is 4.1.29, Java is 1.3.1_09. No symptoms like yours! You should send a signal QUIT to the jvm process during the unresponsiveness time. This is a general JVM mechanism (at least for sun JVMs). The signal writes a stack trace for each thread on STDOUT (so you should also start tomcat with redirection of STDOUT the output to some file). Beware: older JVM in rare cases stopped working after getting this signal (not expected with 1.3.1_09). In this stack dump you should be able to figure out, in which methods most of your threads stay and what the status is. Is there native code included (via JNI)? Any synchronization done in the application itself? Are you using Tomcat clustering? Which JVM? Sincerely Rainer Jung Martin Schulz wrote: Can someone confirm that Tomcat works well on busy SMP systems (e.g. Sun V1280), or whether there are problems in Tomcat. Here's what we have at our end: We are currently performance testing our application (Tomcat 4.1.30) on Solaris 9, on a V1280 system w. 8 CPUs. (SDK 1.4.2_04). Transaction rates are moderate, around 30/s. The application, after about 30-40 minutes, gets unresponsive for a little while (1-10 minutes), gets back to work (for a varying period of time, but definitely under 30 min), and then the cycle repeats. At half the transaction rate, the smptoms are no longer easily observed,. The above symptoms disappear when we use a single CPU, or a single board of 4 CPUs. That scenario seems to imply synchronization problems soemwhere in the Java code. The problem could not be observed in development configurations either (Wintel, 1-CPU Sun boxes.) The behavior is such that connections get accepted, but no response is sent (established connections remain at a fixed level). The number of connections in this state varies (20-70). From the timers we keep we learn that the service gets stuck when reading the input. Once unblocked the responses get sent out rapidly again. We have tuned the system for efficient, high-volume TCP-IP traffic. We tried the coyote connector and the HttpConnector, both with the same effect. Please respond if you can confirm or dismiss threading issues in Tomcat. We would also be ineterested in testing approaches for such a scenario. .I have kept system statistics for both scenarios and can provide these on request. Thanks! Martin Schulz - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Any synchronization issues with SMP?
Rainer, Thanks for the tips. I am about to take timing stats internally in the ThreadPool and the Tcp workers. Also, the described symptoms do not disappear, but seem to be of much shorter duration when only 4 CPUs are used for the application. I'll summarize what I find. Martin Rainer Jung wrote: Hi, we know one application running on 9 systems with 4 US II CPUs each under Solaris 9. Peak request rates at 20 requests/second per system. Tomcat is 4.1.29, Java is 1.3.1_09. No symptoms like yours! You should send a signal QUIT to the jvm process during the unresponsiveness time. This is a general JVM mechanism (at least for sun JVMs). The signal writes a stack trace for each thread on STDOUT (so you should also start tomcat with redirection of STDOUT the output to some file). Beware: older JVM in rare cases stopped working after getting this signal (not expected with 1.3.1_09). In this stack dump you should be able to figure out, in which methods most of your threads stay and what the status is. Is there native code included (via JNI)? Any synchronization done in the application itself? Are you using Tomcat clustering? Which JVM? Sincerely Rainer Jung Martin Schulz wrote: Can someone confirm that Tomcat works well on busy SMP systems (e.g. Sun V1280), or whether there are problems in Tomcat. Here's what we have at our end: We are currently performance testing our application (Tomcat 4.1.30) on Solaris 9, on a V1280 system w. 8 CPUs. (SDK 1.4.2_04). Transaction rates are moderate, around 30/s. The application, after about 30-40 minutes, gets unresponsive for a little while (1-10 minutes), gets back to work (for a varying period of time, but definitely under 30 min), and then the cycle repeats. At half the transaction rate, the smptoms are no longer easily observed,. The above symptoms disappear when we use a single CPU, or a single board of 4 CPUs. That scenario seems to imply synchronization problems soemwhere in the Java code. The problem could not be observed in development configurations either (Wintel, 1-CPU Sun boxes.) The behavior is such that connections get accepted, but no response is sent (established connections remain at a fixed level). The number of connections in this state varies (20-70). From the timers we keep we learn that the service gets stuck when reading the input. Once unblocked the responses get sent out rapidly again. We have tuned the system for efficient, high-volume TCP-IP traffic. We tried the coyote connector and the HttpConnector, both with the same effect. Please respond if you can confirm or dismiss threading issues in Tomcat. We would also be ineterested in testing approaches for such a scenario. .I have kept system statistics for both scenarios and can provide these on request. Thanks! Martin Schulz - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
AW: Any synchronization issues with SMP?
Hi, I'm not sure whether this could be of any help, we had a similar problem and created a javacore and found a real basic locking in java output stream. The reason were concurrent System.out.println() in different threads for different requests which blocked the real stdout (some jsp's with some console output for checks). It was not a Tomcat application, but a wep app in bea (aix, 2 Processors), but it was really a problem in java, so could be the same. After removal of all System.out/err calls the error disappeared - obvious because reason was identified in core file. Sascha -Ursprüngliche Nachricht- Von: Martin Schulz [mailto:[EMAIL PROTECTED] Gesendet: Donnerstag, 24. Juni 2004 00:37 An: Tomcat Developers List Betreff: Re: Any synchronization issues with SMP? Rainer, Thanks for the tips. I am about to take timing stats internally in the ThreadPool and the Tcp workers. Also, the described symptoms do not disappear, but seem to be of much shorter duration when only 4 CPUs are used for the application. I'll summarize what I find. Martin Rainer Jung wrote: Hi, we know one application running on 9 systems with 4 US II CPUs each under Solaris 9. Peak request rates at 20 requests/second per system. Tomcat is 4.1.29, Java is 1.3.1_09. No symptoms like yours! You should send a signal QUIT to the jvm process during the unresponsiveness time. This is a general JVM mechanism (at least for sun JVMs). The signal writes a stack trace for each thread on STDOUT (so you should also start tomcat with redirection of STDOUT the output to some file). Beware: older JVM in rare cases stopped working after getting this signal (not expected with 1.3.1_09). In this stack dump you should be able to figure out, in which methods most of your threads stay and what the status is. Is there native code included (via JNI)? Any synchronization done in the application itself? Are you using Tomcat clustering? Which JVM? Sincerely Rainer Jung Martin Schulz wrote: Can someone confirm that Tomcat works well on busy SMP systems (e.g. Sun V1280), or whether there are problems in Tomcat. Here's what we have at our end: We are currently performance testing our application (Tomcat 4.1.30) on Solaris 9, on a V1280 system w. 8 CPUs. (SDK 1.4.2_04). Transaction rates are moderate, around 30/s. The application, after about 30-40 minutes, gets unresponsive for a little while (1-10 minutes), gets back to work (for a varying period of time, but definitely under 30 min), and then the cycle repeats. At half the transaction rate, the smptoms are no longer easily observed,. The above symptoms disappear when we use a single CPU, or a single board of 4 CPUs. That scenario seems to imply synchronization problems soemwhere in the Java code. The problem could not be observed in development configurations either (Wintel, 1-CPU Sun boxes.) The behavior is such that connections get accepted, but no response is sent (established connections remain at a fixed level). The number of connections in this state varies (20-70). From the timers we keep we learn that the service gets stuck when reading the input. Once unblocked the responses get sent out rapidly again. We have tuned the system for efficient, high-volume TCP-IP traffic. We tried the coyote connector and the HttpConnector, both with the same effect. Please respond if you can confirm or dismiss threading issues in Tomcat. We would also be ineterested in testing approaches for such a scenario. .I have kept system statistics for both scenarios and can provide these on request. Thanks! Martin Schulz - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Any synchronization issues with SMP?
Can someone confirm that Tomcat works well on busy SMP systems (e.g. Sun V1280), or whether there are problems in Tomcat. Here's what we have at our end: We are currently performance testing our application (Tomcat 4.1.30) on Solaris 9, on a V1280 system w. 8 CPUs. (SDK 1.4.2_04). Transaction rates are moderate, around 30/s. The application, after about 30-40 minutes, gets unresponsive for a little while (1-10 minutes), gets back to work (for a varying period of time, but definitely under 30 min), and then the cycle repeats. At half the transaction rate, the smptoms are no longer easily observed,. The above symptoms disappear when we use a single CPU, or a single board of 4 CPUs. That scenario seems to imply synchronization problems soemwhere in the Java code. The problem could not be observed in development configurations either (Wintel, 1-CPU Sun boxes.) The behavior is such that connections get accepted, but no response is sent (established connections remain at a fixed level). The number of connections in this state varies (20-70). From the timers we keep we learn that the service gets stuck when reading the input. Once unblocked the responses get sent out rapidly again. We have tuned the system for efficient, high-volume TCP-IP traffic. We tried the coyote connector and the HttpConnector, both with the same effect. Please respond if you can confirm or dismiss threading issues in Tomcat. We would also be ineterested in testing approaches for such a scenario. .I have kept system statistics for both scenarios and can provide these on request. Thanks! Martin Schulz - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
