Memory leak- yeah I know
Ok, I know that these emails are usually dismissed with the words:
Gets us some proof from a profiler, and I will...gimme some time, in the
meanwhile, if anyone else wants to play with it.
well the story is that during heavy load tests of the session replication my
system my VMs always run out of memory.
What am I doing different than others, well, during my load tests I am not
using keep alive connections to achieve true round robin. I'm doing this by
setting method.setHttp11(false) in the Http client.
Running the test (attached, instructions below) I am hitting
http://localhost:8080/index.jsp (home page) and watching the memory grow and
grow and grow.
I am using a profiler and having somewhat of a hard time reading the output.
I am gonna download a trial of jprobe, see how that works out for me... I
will be back with results from that.
in the meantime, for those who want to see their memory grow here are the
instructions
My env:
Machine: PIII 1.6Ghz, 512MB Ram
OS: Redhat 8
VM: Sun JDK 1.4.2_02 (build 1.4.2_02-b03)
Instructions:
download the attached client.jar
execute:
java -cp testclient.jar;commons-httpclient.jar;commons-logging.jar
org.apache.catalina.cluster.test.client.MemTestClient http
://192.168.0.103:7080/index.jsp 100 10 1000 false (exchange ; for : on
unix)
Watching the memory grow using Top, although it shouldn't increase at all.
Filip
package org.apache.catalina.cluster.test.client;
import org.apache.commons.httpclient.*;
import org.apache.commons.httpclient.methods.*;
public class MemTestClient {
public MemTestClient() {
}
public static void main(String[] args) throws Exception {
System.out.println(Usage org.apache.catalina.cluster.test.client.MemTestClient url nrofthreads nrofrequests delay useHttp11);
String url = args[0];
int nrofthreads = Integer.parseInt(args[1]);
int nrofrequests = Integer.parseInt(args[2]);
long delay = Long.parseLong(args[3]);
boolean useHttp11 = Boolean.getBoolean(args[4]);
for ( int i=0; inrofthreads; i++ ) {
TestThread thread = new TestThread(useHttp11,url,delay,nrofrequests);
thread.setDaemon(false);
thread.start();
Thread.currentThread().sleep(delay/3);
}//for
}
static int threadCount = 0;
static synchronized void incThread() {
threadCount++;
printStats();
}
static synchronized void decThread() {
threadCount--;
printStats();
}
static int[] codes = new int[1000];
static int nrofrequests = 0;
static long totalTime = 0;
static long maxTime = Long.MIN_VALUE;
static long minTime = Long.MAX_VALUE;
public static void printStats() {
StringBuffer stats = new StringBuffer(Memory test stats);
stats.append(\n\tThread count=).append(threadCount);
stats.append(\n\tNr Of Requests=).append(nrofrequests);
stats.append(\n\tTotal time=).append(totalTime).append( ms.);
stats.append(\n\tAverage request time=).append(totalTime /
(nrofrequests==0?1:nrofrequests)).append( ms.);
stats.append(\n\tMax request time=).append(maxTime).append( ms.);
stats.append(\n\tMin request time=).append(minTime).append( ms.);
for (int i = 0; i codes.length; i++) {
if (codes[i] 0) {
stats.append(\n\tNr of ).append(i).append( codes=).append(
codes[i]);
}
} //for
System.out.println();
System.out.println(stats);
}
public static void addStats(int code, long time) {
codes[code] = codes[code]+1;
nrofrequests++;
maxTime = Math.max(maxTime,time);
minTime = Math.min(minTime,time);
totalTime += time;
if ( (nrofrequests % 100 ) == 0 ) {
printStats();
}//if
}//addStats
public static class TestThread extends Thread {
boolean http11 = true;
String url = null;
long delay = 1000;
int nrofreps = 100;
public TestThread(boolean http11, String url, long delay, int nrofreps) {
this.http11 = http11;
this.url = url;
this.delay = delay;
this.nrofreps = nrofreps;
}
public void run() {
incThread();
int cnt = 0;
HttpClient client = new HttpClient();
GetMethod method = new GetMethod(url);
method.setHttp11(http11);
while ( cnt nrofreps ) {
try {
long start = System.currentTimeMillis();
int status = client.executeMethod(method);
method.getResponseHeaders();
method.getResponseBody();
long stop = System.currentTimeMillis();
addStats(status,stop-start);
Thread.currentThread().sleep(delay);
} catch (
Re: Memory leak- yeah I know
How heavy of a load does this generate, in terms of # of req/s? [EMAIL PROTECTED] 1/14/04 3:49:00 PM Ok, I know that these emails are usually dismissed with the words: Gets us some proof from a profiler, and I will...gimme some time, in the meanwhile, if anyone else wants to play with it. well the story is that during heavy load tests of the session replication my system my VMs always run out of memory. What am I doing different than others, well, during my load tests I am not using keep alive connections to achieve true round robin. I'm doing this by setting method.setHttp11(false) in the Http client. Running the test (attached, instructions below) I am hitting http://localhost:8080/index.jsp (home page) and watching the memory grow and grow and grow. I am using a profiler and having somewhat of a hard time reading the output. I am gonna download a trial of jprobe, see how that works out for me... I will be back with results from that. in the meantime, for those who want to see their memory grow here are the instructions My env: Machine: PIII 1.6Ghz, 512MB Ram OS: Redhat 8 VM: Sun JDK 1.4.2_02 (build 1.4.2_02-b03) Instructions: download the attached client.jar execute: java -cp testclient.jar;commons-httpclient.jar;commons-logging.jar org.apache.catalina.cluster.test.client.MemTestClient http ://192.168.0.103:7080/index.jsp 100 10 1000 false (exchange ; for : on unix) Watching the memory grow using Top, although it shouldn't increase at all. Filip Jeff Tulley ([EMAIL PROTECTED]) (801)861-5322 Novell, Inc., The Leading Provider of Net Business Solutions http://www.novell.com - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory leak- yeah I know
Filip Hanik wrote: Ok, I know that these emails are usually dismissed with the words: Gets us some proof from a profiler, and I will...gimme some time, in the meanwhile, if anyone else wants to play with it. well the story is that during heavy load tests of the session replication my system my VMs always run out of memory. What am I doing different than others, well, during my load tests I am not using keep alive connections to achieve true round robin. I'm doing this by setting method.setHttp11(false) in the Http client. Running the test (attached, instructions below) I am hitting http://localhost:8080/index.jsp (home page) and watching the memory grow and grow and grow. I am using a profiler and having somewhat of a hard time reading the output. I am gonna download a trial of jprobe, see how that works out for me... I will be back with results from that. in the meantime, for those who want to see their memory grow here are the instructions My env: Machine: PIII 1.6Ghz, 512MB Ram OS: Redhat 8 VM: Sun JDK 1.4.2_02 (build 1.4.2_02-b03) Instructions: download the attached client.jar execute: java -cp testclient.jar;commons-httpclient.jar;commons-logging.jar org.apache.catalina.cluster.test.client.MemTestClient http ://192.168.0.103:7080/index.jsp 100 10 1000 false (exchange ; for : on unix) Watching the memory grow using Top, although it shouldn't increase at all. There's ab for that kind of stuff, you know ... No need to waste your time writing some new stuff. Your tone is a bit sarcastic, I think. I'm eagerly awaiting your proofs ;-) Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Memory leak- yeah I know
How heavy of a load does this generate, in terms of # of req/s? execute: java -cp testclient.jar;commons-httpclient.jar;commons-logging.jar org.apache.catalina.cluster.test.client.MemTestClient http ://192.168.0.103:7080/index.jsp 100 10 1000 false java -cp testclient.jar;commons-httpclient.jar;commons-logging.jar org.apache.catalina.cluster.test.client.MemTestClient get-url nrofthreads nrofrequests sleep-between-request usehttp11 - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Memory leak- yeah I know
Remy, absolutely no intentions of being sarcastic :), if there isn't a leak, that is great, if there is, will fix it. I'm trying to figure it out, I didn't ask anyone else to spend time on it, or even read my email. but if anyone is doing work on this lets work on it together. just got jprobe up and running, once I have ran a few tests, I will experiment with the garbage collection, the memory is slowly growing, its been running for an hour. Filip -Original Message- From: Remy Maucherat [mailto:[EMAIL PROTECTED] Sent: Wednesday, January 14, 2004 3:08 PM To: Tomcat Developers List Subject: Re: Memory leak- yeah I know Filip Hanik wrote: Ok, I know that these emails are usually dismissed with the words: Gets us some proof from a profiler, and I will...gimme some time, in the meanwhile, if anyone else wants to play with it. well the story is that during heavy load tests of the session replication my system my VMs always run out of memory. What am I doing different than others, well, during my load tests I am not using keep alive connections to achieve true round robin. I'm doing this by setting method.setHttp11(false) in the Http client. Running the test (attached, instructions below) I am hitting http://localhost:8080/index.jsp (home page) and watching the memory grow and grow and grow. I am using a profiler and having somewhat of a hard time reading the output. I am gonna download a trial of jprobe, see how that works out for me... I will be back with results from that. in the meantime, for those who want to see their memory grow here are the instructions My env: Machine: PIII 1.6Ghz, 512MB Ram OS: Redhat 8 VM: Sun JDK 1.4.2_02 (build 1.4.2_02-b03) Instructions: download the attached client.jar execute: java -cp testclient.jar;commons-httpclient.jar;commons-logging.jar org.apache.catalina.cluster.test.client.MemTestClient http ://192.168.0.103:7080/index.jsp 100 10 1000 false (exchange ; for : on unix) Watching the memory grow using Top, although it shouldn't increase at all. There's ab for that kind of stuff, you know ... No need to waste your time writing some new stuff. Your tone is a bit sarcastic, I think. I'm eagerly awaiting your proofs ;-) Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Memory leak- yeah I know
Filip Hanik wrote: How heavy of a load does this generate, in terms of # of req/s? execute: java -cp testclient.jar;commons-httpclient.jar;commons-logging.jar org.apache.catalina.cluster.test.client.MemTestClient http ://192.168.0.103:7080/index.jsp 100 10 1000 false java -cp testclient.jar;commons-httpclient.jar;commons-logging.jar org.apache.catalina.cluster.test.client.MemTestClient get-url nrofthreads nrofrequests sleep-between-request usehttp11 Maybe this isn't useless after all: I've finally found a benchmark where 5.0.17 wipes the floor with 4.1.29 (I did increase the maxProcessors to 150; the bench with those settings obviously won't work with only 75 processors). For a static request (2 KB; /tomcat.gif) the average response time is 4 ms (compared to 10 ms), less memory (about 20 MB less; of course, this is more heap due to more GC, not actually used memory), and less CPU time (0-3% compared to 12-14%). Somehow, ab doesn't show that kind of difference (too bad, I would have been happy). Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Memory leak- yeah I know
Title: RE: Memory leak- yeah I know Filip, I spent some time a while back trying to get some reproducibility into memory use in Tomcat (4.1.24) I found that even sitting idle tomcats memory use would increase. Here is my original mail to the tomcat-dev mailing list 4.1.24 secure connector throws a LOT of garbage After I did some more investigations and figured out a fix the following bug was raised http://nagoya.apache.org/bugzilla/show_bug.cgi?id=19264 To reduce the amount of garbage thrown by tomcat I would recommend two things, doing these might help you track down other memory issues. 1 - Use -XmxsomeValuem -XmssomeValuem. The above sizes the heap to whatever value you put in there (note the value is megabytes). I found (at least on the Solaris JVM) that sizing the heap and making the min and max heap sizes the same not only sped up start up but also made the garbage collector behave much better. 2- Set the serverSoTimeout value to 0 in the connector configuration in your server configuration file. This will prevent the sever socket timing out. Which DOES throw garbage, and if it is a secure connector you are using it throws SO much garbage that the garbage collector can not keep up too well. Using just number tip 1 above (as the fix to the bug above wasn't in Tomcat 4.1.x before we needed to go into production. We managed to run at something like 100 TPS for 48 hours with no falling over and no net increase in memory usage. If you need any more information on the above let me know. Michael Yates Software Engineer, Location Center Nortel Networks Wollongong, Australia. -Original Message- From: Filip Hanik [mailto:[EMAIL PROTECTED]] Sent: Thursday, 15 January 2004 11:09 AM To: Tomcat Developers List Subject: RE: Memory leak- yeah I know Remy, absolutely no intentions of being sarcastic :), if there isn't a leak, that is great, if there is, will fix it. I'm trying to figure it out, I didn't ask anyone else to spend time on it, or even read my email. but if anyone is doing work on this lets work on it together. just got jprobe up and running, once I have ran a few tests, I will experiment with the garbage collection, the memory is slowly growing, its been running for an hour. Filip -Original Message- From: Remy Maucherat [mailto:[EMAIL PROTECTED]] Sent: Wednesday, January 14, 2004 3:08 PM To: Tomcat Developers List Subject: Re: Memory leak- yeah I know Filip Hanik wrote: Ok, I know that these emails are usually dismissed with the words: Gets us some proof from a profiler, and I will...gimme some time, in the meanwhile, if anyone else wants to play with it. well the story is that during heavy load tests of the session replication my system my VMs always run out of memory. What am I doing different than others, well, during my load tests I am not using keep alive connections to achieve true round robin. I'm doing this by setting method.setHttp11(false) in the Http client. Running the test (attached, instructions below) I am hitting http://localhost:8080/index.jsp (home page) and watching the memory grow and grow and grow. I am using a profiler and having somewhat of a hard time reading the output. I am gonna download a trial of jprobe, see how that works out for me... I will be back with results from that. in the meantime, for those who want to see their memory grow here are the instructions My env: Machine: PIII 1.6Ghz, 512MB Ram OS: Redhat 8 VM: Sun JDK 1.4.2_02 (build 1.4.2_02-b03) Instructions: download the attached client.jar execute: java -cp testclient.jar;commons-httpclient.jar;commons-logging.jar org.apache.catalina.cluster.test.client.MemTestClient http ://192.168.0.103:7080/index.jsp 100 10 1000 false (exchange ; for : on unix) Watching the memory grow using Top, although it shouldn't increase at all. There's ab for that kind of stuff, you know ... No need to waste your time writing some new stuff. Your tone is a bit sarcastic, I think. I'm eagerly awaiting your proofs ;-) Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] ---BeginMessage--- Title: 4.1.24 secure connector throws a LOT of garbage Hi All, How do I turn on low level Tomcat debugging? Specifically I want to log the output of the PoolTcpEndpoint class within the org.apache.tomcat.util.net package. Tomcat seems to be throwing a LOT of garbage when the secure connector is enabled (in the order of 50K a second). The garbage collector doesn't keep up very well and the heap size grows and grows. It is not a memory leak as such because when I force garbage collection the amount of used memory drops right back to the original base line. I
RE: Memory leak- yeah I know
RE: Memory leak- yeah I knowhi Michael, thanks for your info. I will look into that as well and set those settings The thing I am investigating right now is if the fact that we are registering a lot of things with JMX causes some references to never be released. more info to follow. Filip -Original Message- From: Michael Yates [mailto:[EMAIL PROTECTED] Sent: Wednesday, January 14, 2004 4:43 PM To: Tomcat Developers List Subject: RE: Memory leak- yeah I know Filip, I spent some time a while back trying to get some reproducibility into memory use in Tomcat (4.1.24) I found that even sitting idle tomcats memory use would increase. Here is my original mail to the tomcat-dev mailing list 4.1.24 secure connector throws a LOT of garbage After I did some more investigations and figured out a fix the following bug was raised http://nagoya.apache.org/bugzilla/show_bug.cgi?id=19264 To reduce the amount of garbage thrown by tomcat I would recommend two things, doing these might help you track down other memory issues. 1 - Use -XmxsomeValuem -XmssomeValuem. The above sizes the heap to whatever value you put in there (note the value is megabytes). I found (at least on the Solaris JVM) that sizing the heap and making the min and max heap sizes the same not only sped up start up but also made the garbage collector behave much better. 2- Set the serverSoTimeout value to 0 in the connector configuration in your server configuration file. This will prevent the sever socket timing out. Which DOES throw garbage, and if it is a secure connector you are using it throws SO much garbage that the garbage collector can not keep up too well. Using just number tip 1 above (as the fix to the bug above wasn't in Tomcat 4.1.x before we needed to go into production. We managed to run at something like 100 TPS for 48 hours with no falling over and no net increase in memory usage. If you need any more information on the above let me know. Michael Yates Software Engineer, Location Center Nortel Networks Wollongong, Australia. -Original Message- From: Filip Hanik [mailto:[EMAIL PROTECTED] Sent: Thursday, 15 January 2004 11:09 AM To: Tomcat Developers List Subject: RE: Memory leak- yeah I know Remy, absolutely no intentions of being sarcastic :), if there isn't a leak, that is great, if there is, will fix it. I'm trying to figure it out, I didn't ask anyone else to spend time on it, or even read my email. but if anyone is doing work on this lets work on it together. just got jprobe up and running, once I have ran a few tests, I will experiment with the garbage collection, the memory is slowly growing, its been running for an hour. Filip -Original Message- From: Remy Maucherat [mailto:[EMAIL PROTECTED] Sent: Wednesday, January 14, 2004 3:08 PM To: Tomcat Developers List Subject: Re: Memory leak- yeah I know Filip Hanik wrote: Ok, I know that these emails are usually dismissed with the words: Gets us some proof from a profiler, and I will...gimme some time, in the meanwhile, if anyone else wants to play with it. well the story is that during heavy load tests of the session replication my system my VMs always run out of memory. What am I doing different than others, well, during my load tests I am not using keep alive connections to achieve true round robin. I'm doing this by setting method.setHttp11(false) in the Http client. Running the test (attached, instructions below) I am hitting http://localhost:8080/index.jsp (home page) and watching the memory grow and grow and grow. I am using a profiler and having somewhat of a hard time reading the output. I am gonna download a trial of jprobe, see how that works out for me... I will be back with results from that. in the meantime, for those who want to see their memory grow here are the instructions My env: Machine: PIII 1.6Ghz, 512MB Ram OS: Redhat 8 VM: Sun JDK 1.4.2_02 (build 1.4.2_02-b03) Instructions: download the attached client.jar execute: java -cp testclient.jar;commons-httpclient.jar;commons-logging.jar org.apache.catalina.cluster.test.client.MemTestClient http ://192.168.0.103:7080/index.jsp 100 10 1000 false (exchange ; for : on unix) Watching the memory grow using Top, although it shouldn't increase at all. There's ab for that kind of stuff, you know ... No need to waste your time writing some new stuff. Your tone is a bit sarcastic, I think. I'm eagerly awaiting your proofs ;-) Rémy - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
