Re: Tomcat threads, II
Hi. To implement a recommendation from someone on this list, I took a few snapshots using jmap -heap pid while Tomcat was starting up and also starting up a webapp. I just ran the jmap command several times at about 5 second intervals, and redirected the output to a file. Below if the file content. What I would like to know now, is how I interpret these numbers. Is there somewhere a tutorial giving some pointers as to what I am looking at ? Or, does anyone feel like making some comments on the data below ? As a slightly narrower question, how do some of the numbers below relate to JVM startup parameters such as -Xms, -Xmx, and/or GC-related parameters ? And, is there somewhere I could find a description of terms used below, such as Eden, New Generation, Perm Generation etc.. ? Thanks in advance. Output of jmap : using thread-local object allocation. Mark Sweep Compact GC Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 268435456 (256.0MB) NewSize = 655360 (0.625MB) MaxNewSize = 4294901760 (4095.9375MB) OldSize = 1441792 (1.375MB) NewRatio = 8 SurvivorRatio= 8 PermSize = 16777216 (16.0MB) MaxPermSize = 67108864 (64.0MB) Heap Usage: New Generation (Eden + 1 Survivor Space): capacity = 16187392 (15.4375MB) used = 14485952 (13.81488037109375MB) free = 1701440 (1.62261962890625MB) 89.48910361842105% used Eden Space: capacity = 14417920 (13.75MB) used = 12716480 (12.12738037109375MB) free = 1701440 (1.62261962890625MB) 88.1991299715909% used From Space: capacity = 1769472 (1.6875MB) used = 1769472 (1.6875MB) free = 0 (0.0MB) 100.0% used To Space: capacity = 1769472 (1.6875MB) used = 0 (0.0MB) free = 1769472 (1.6875MB) 0.0% used tenured generation: capacity = 143376384 (136.734375MB) used = 109058784 (104.00656127929688MB) free = 34317600 (32.727813720703125MB) 76.06467742972232% used Perm Generation: capacity = 16777216 (16.0MB) used = 9495520 (9.055633544921875MB) free = 7281696 (6.944366455078125MB) 56.59770965576172% used using thread-local object allocation. Mark Sweep Compact GC Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 268435456 (256.0MB) NewSize = 655360 (0.625MB) MaxNewSize = 4294901760 (4095.9375MB) OldSize = 1441792 (1.375MB) NewRatio = 8 SurvivorRatio= 8 PermSize = 16777216 (16.0MB) MaxPermSize = 67108864 (64.0MB) Heap Usage: New Generation (Eden + 1 Survivor Space): capacity = 16187392 (15.4375MB) used = 7146784 (6.815704345703125MB) free = 9040608 (8.621795654296875MB) 44.15031155111336% used Eden Space: capacity = 14417920 (13.75MB) used = 5377320 (5.128211975097656MB) free = 9040600 (8.621788024902344MB) 37.29608709161932% used From Space: capacity = 1769472 (1.6875MB) used = 1769464 (1.6874923706054688MB) free = 8 (7.62939453125E-6MB) 99.99954788773148% used To Space: capacity = 1769472 (1.6875MB) used = 0 (0.0MB) free = 1769472 (1.6875MB) 0.0% used tenured generation: capacity = 143376384 (136.734375MB) used = 98216248 (93.66631317138672MB) free = 45160136 (43.06806182861328MB) 68.50238878949548% used Perm Generation: capacity = 16777216 (16.0MB) used = 9495520 (9.055633544921875MB) free = 7281696 (6.944366455078125MB) 56.59770965576172% used using thread-local object allocation. Mark Sweep Compact GC Heap Configuration: MinHeapFreeRatio = 40 MaxHeapFreeRatio = 70 MaxHeapSize = 268435456 (256.0MB) NewSize = 655360 (0.625MB) MaxNewSize = 4294901760 (4095.9375MB) OldSize = 1441792 (1.375MB) NewRatio = 8 SurvivorRatio= 8 PermSize = 16777216 (16.0MB) MaxPermSize = 67108864 (64.0MB) Heap Usage: New Generation (Eden + 1 Survivor Space): capacity = 16187392 (15.4375MB) used = 15972760 (15.232810974121094MB) free = 214632 (0.20468902587890625MB) 98.67407918459008% used Eden Space: capacity = 14417920 (13.75MB) used = 14203296 (13.545318603515625MB) free = 214624 (0.204681396484375MB) 98.51140802556819% used From Space: capacity = 1769472 (1.6875MB) used = 1769464 (1.6874923706054688MB) free = 8 (7.62939453125E-6MB) 99.99954788773148% used To Space: capacity = 1769472 (1.6875MB) used = 0 (0.0MB) free = 1769472 (1.6875MB) 0.0% used tenured generation: capacity = 143376384 (136.734375MB) used = 138744856 (132.3174057006836MB) free = 4631528 (4.416969299316406MB) 96.76967163574163% used Perm Generation: capacity = 16777216 (16.0MB) used = 9495616 (9.05572509765625MB) free = 7281600 (6.94427490234375MB)
Re: Tomcat threads, II
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, André Warnier wrote: When I started in this business, 64 Kb was a nice quantity of memory to program in, and quite expensive too. I created and ran a payroll application for a 1,000 people company in there. This Java app looks a lot cuter than the payroll did, but 500 Megabyte of memory for one single Tomcat app, mmm. Some reflexes remain for a lifetime. If you're concerned about the amount of memory used by your process, feel free to trim the heap back. Our application served thousands of requests per day on a 64MB heap for a long time before we had to bump it up. If you give Java room, it'll take it. If you constrain your heap, GCs will simply happen more frequently. If it takes a lot of time and memory to start up, but the steady-state behavior doesn't require too much memory, then a smaller heap might actually be a good idea. That will certainly leave more memory for other processes and make your 'top' output look less scary. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkkcW+4ACgkQ9CaO5/Lv0PDYVQCgwHTkd9AsrGT/wNP4y2ODhUwJ 89YAn0BldLa1hWtQ2BXwRLeL5m8hI71J =oPP+ -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat threads, II
David kerber wrote: Pid wrote: ... Having said that, parsing XML can be a memory intensive operation as well, so your old box could be experiencing problems there - check the garbage collection log. p I try to avoid XML for this exact reason. We found that our application could handle between 6 and 10 times more concurrent requests on the same hardware when we went away from XML data transmission, and switched to a simple text-based HTTP POST request. D Thank you all very much. But, as mentioned before, I have no choice about the XML. That's how the application works, and we don't have the code. We have already tried - because that is one thing we can do - to split the XML data into smaller chunks, but that does not seem to have any significant impact. We do really like the application though. So before I decide to junk it, we will make further tests on a better server. But my question now was, apart from the things I cannot change, what could I try changing on the current test server, to see if it makes any difference ? Should I give more/less meory to Tomcat or Java, and if yes how ? Can I tell Tomcat/Java to start this application, but use only some maximum percentage of the memory/cpu time while doing so ? I also do not know where to look for the garbage collection log. Hints anyone ? And what should I look for in there ? Is there any way, from the outside of the application, that I can check what exactly the application is busy doing, when it is starting up ? - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat threads, II
From: André Warnier [mailto:[EMAIL PROTECTED] Subject: Re: Tomcat threads, II We have already tried - because that is one thing we can do - to split the XML data into smaller chunks, but that does not seem to have any significant impact. Nor would I expect it to. The total amount of memory consumed is going to be about the same, regardless of whether or not you split it up. From the numbers you posted before, it doesn't look like memory is the issue - you're maxing out the CPU. Parsing XML is very CPU intensive, and you simply don't have enough cycles to get it done in a reasonable amount of time. If you can run on a multi-CPU system and figure out how to process pieces of the XML file in parallel, that would help. Should I give more/less meory to Tomcat or Java, and if yes how ? Use JConsole to see what's going on while the webapp is initializing. Can I tell Tomcat/Java to start this application, but use only some maximum percentage of the memory/cpu time while doing so ? No. You can only limit memory by the -Xmx setting. I also do not know where to look for the garbage collection log. Likely because you didn't ask for one. Turn on -verbose:gc, and set -Xloggc:file to specify the GC logging location. However, I'd use JConsole first - it's easier. Is there any way, from the outside of the application, that I can check what exactly the application is busy doing, when it is starting up ? Again, use JConsole, or turn on JVM profiling (I'd recommend the former). - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat threads, II
From: André Warnier [mailto:[EMAIL PROTECTED] Subject: Tomcat threads, II -classpath :/usr/lib/jvm/java-1.5.0-sun/jre//lib/jcert.jar: /usr/lib/jvm/java-1.5.0-sun/jre//lib/jnet.jar: /usr/lib/jvm/java-1.5.0-sun/jre//lib/jsse.jar: /usr/share/tomcat5.5/bin/bootstrap.jar: /usr/share/tomcat5.5/bin/commons-logging-api.jar The classpath above is very wrong: no jars from jre/lib should *ever* be on the classpath. Those jars are already on the JVM's bootclasspath and will at best be ignored anywhere else. Their presence will at a minimum slow down Tomcat initialization. Also, the commons-logging-api.jar should not be on the classpath; Tomcat finds that all by itself, based on entries in the bootstrap.jar manifest. If you need commons-logging for a webapp, it should be in the webapp's WEB-INF/lib directory, not on classpath. Finally, the leading colon on the classpath shouldn't be there, but it appears to do little harm. -Djava.security.manager Enabling the security manager slows things down; do you really need it? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat threads, II
Caldarale, Charles R wrote: From: André Warnier [mailto:[EMAIL PROTECTED] Subject: Tomcat threads, II -Djava.security.manager Enabling the security manager slows things down; do you really need it? Not as far as I know. It was only the default setting on that system, from the (non-official) Tomcat5.5 Debian package. I have turned it off now, and restarted Tomcat, but it did not have any noticeable impact on the startup time. -classpath :/usr/lib/jvm/java-1.5.0-sun/jre//lib/jcert.jar: /usr/lib/jvm/java-1.5.0-sun/jre//lib/jnet.jar: /usr/lib/jvm/java-1.5.0-sun/jre//lib/jsse.jar: /usr/share/tomcat5.5/bin/bootstrap.jar: /usr/share/tomcat5.5/bin/commons-logging-api.jar The classpath above is very wrong: no jars from jre/lib should *ever* be on the classpath. Those jars are already on the JVM's bootclasspath and will at best be ignored anywhere else. Their presence will at a minimum slow down Tomcat initialization. Also, the commons-logging-api.jar should not be on the classpath; Tomcat finds that all by itself, based on entries in the bootstrap.jar manifest. If you need commons-logging for a webapp, it should be in the webapp's WEB-INF/lib directory, not on classpath. Finally, the leading colon on the classpath shouldn't be there, but it appears to do little harm. Now that is another story. As far as I can tell, these settings come from the catalina.sh script. I have no idea if this script is modified, compared to the Official version. It looks like it covers several platforms though, so I doubt that it is specifically from the Debian packager. I attach it for inspection. Now I'll try what you indicate above, gradually removing selected portions, tell you later what it does to my system. ... Well, it seems to have survived (apart from bootstrap.jar, which I had to leave), and now runs as tomcat55 8106 1 97 119903 196100 0 17:54 pts/200:04:56 /usr/lib/jvm/java-1.5.0-sun/bin/java -Djava.awt.headless=true -Xmx256M -server -Djava.endorsed.dirs=/usr/share/tomcat5.5/common/endorsed -classpath :/usr/share/tomcat5.5/bin/bootstrap.jar -Dcatalina.base=/var/lib/tomcat5.5 -Dcatalina.home=/usr/share/tomcat5.5 -Djava.io.tmpdir=/var/lib/tomcat5.5/temp org.apache.catalina.startup.Bootstrap start But that did not apparently reduce its startup time much. If anything, it seems now to occupy more memory than before : PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 8106 tomcat55 25 0 485m 202m 12m S 0.0 40.1 6:22.38 java Against before : PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 2864 tomcat55 25 0 482m 188m 12m S 0.0 37.4 6:46.74 java Is there an explanation for that ? As far as I know, the same applications are loaded as last time. I also don't seem to be able to run JConsole. I am connecting to that system via a terminal-like SSH connection only. [EMAIL PROTECTED]:/usr/share/tomcat5.5/bin# jconsole Exception in thread AWT-EventQueue-0 java.awt.HeadlessException: No X11 DISPLAY variable was set, but this program performed an operation which requires it. at java.awt.GraphicsEnvironment.checkHeadless(GraphicsEnvironment.java:159) etc... Is there a way around this ? #!/bin/sh # - # Start/Stop Script for the CATALINA Server # # Environment Variable Prequisites # # CATALINA_HOME May point at your Catalina build directory. # # CATALINA_BASE (Optional) Base directory for resolving dynamic portions # of a Catalina installation. If not present, resolves to # the same directory that CATALINA_HOME points to. # # CATALINA_OPTS (Optional) Java runtime options used when the start, # stop, or run command is executed. # # CATALINA_TMPDIR (Optional) Directory path location of temporary directory # the JVM should use (java.io.tmpdir). Defaults to # $CATALINA_BASE/temp. # # JAVA_HOME Must point at your Java Development Kit installation. # Required to run the with the debug or javac argument. # # JRE_HOMEMust point at your Java Development Kit installation. # Defaults to JAVA_HOME if empty. # # JAVA_OPTS (Optional) Java runtime options used when the start, # stop, or run command is executed. # # JPDA_TRANSPORT (Optional) JPDA transport used when the jpda start # command is executed. The default is dt_socket. # # JPDA_ADDRESS(Optional) Java runtime options used when the jpda start # command is executed. The default is 8000. # # JSSE_HOME (Optional) May point at your Java Secure Sockets Extension # (JSSE) installation, whose JAR files will be added to the # system class path used to start Tomcat. # # CATALINA_PID(Optional) Path of the file which should
Re: Tomcat threads, II
André Warnier wrote: Hi again. Now that I have (with your help) established some basic facts, I have a practical case for analysis. Before I get into details (because this is quite long), I'd like to explain the reason why I'm asking this. We have been testing a Tomcat application from a vendor. It works very nicely when it is started on a test system, and does what we expect. But the load it imposes on our test Tomcat at startup, makes me a bit insecure about deploying it to a production server, before I really understand what is happening. Not being a Java or Tomcat expert, I'm just trying to collect some pointers, to see if I can do something about what I am observing, or if that's it and we just have to live with it. The main thing is, I don't even know where to start, so I would appreciate some help, even if very basic. There is quite a bit of data below. We have an old clunker PC (Pentium II, 512 Mb Ram, average ATA disks), which we use to test some applications (not only Java/Tomcat). Its a Linux system, and runs this Tomcat : Using CATALINA_BASE: /usr/share/tomcat5.5 Using CATALINA_HOME: /usr/share/tomcat5.5 Using CATALINA_TMPDIR: /usr/share/tomcat5.5/temp Using JRE_HOME: /usr/lib/jvm/java-1.5.0-sun Server version: Apache Tomcat/5.5 Server built: Jan 20 2008 12:32:00 Server number: 5.5.20.0 OS Name:Linux OS Version: 2.6.18-6-686 Architecture: i386 JVM Version:1.5.0_14-b03 JVM Vendor: Sun Microsystems Inc. The full command used to start Tomcat is : /usr/lib/jvm/java-1.5.0-sun/bin/java -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/var/lib/tomcat5.5/conf/logging.properties -Djava.awt.headless=true -Xmx256M -server -Djava.endorsed.dirs=/usr/share/tomcat5.5/common/endorsed -classpath :/usr/lib/jvm/java-1.5.0-sun/jre//lib/jcert.jar:/usr/lib/jvm/java-1.5.0-sun/jre//lib/jnet.jar:/usr/lib/jvm/java-1.5.0-sun/jre//lib/jsse.jar:/usr/share/tomcat5.5/bin/bootstrap.jar:/usr/share/tomcat5.5/bin/commons-logging-api.jar -Djava.security.manager -Djava.security.policy==/var/lib/tomcat5.5/conf/catalina.policy -Dcatalina.base=/var/lib/tomcat5.5 -Dcatalina.home=/usr/share/tomcat5.5 -Djava.io.tmpdir=/var/lib/tomcat5.5/temp org.apache.catalina.startup.Bootstrap start Under Tomcat there, there is an application which we get from outside and of which we do not have the source. This application is loaded at Tomcat startup. When Tomcat has been running for a while, but the system and Tomcat are basically idle, a top with the option n (sort by memory usage), shows this (first few lines only) : top - 10:15:54 up 162 days, 16:25, 2 users, load average: 0.19, 0.09, 0.03 Tasks: 82 total, 1 running, 81 sleeping, 0 stopped, 0 zombie Cpu(s): 2.0%us, 0.3%sy, 0.0%ni, 97.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem:516920k total, 509716k used, 7204k free,71992k buffers Swap: 1297564k total, 131340k used, 1166224k free,95148k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 6226 tomcat55 25 0 506m 191m 10m S 0.0 37.9 140:26.15 java 5937 root 22 0 267m 26m 8056 S 0.0 5.2 25:26.88 java 24608 root 23 0 260m 15m 9332 S 0.0 3.1 0:02.08 java 24638 root 15 0 29564 15m 5984 S 0.0 3.0 0:02.16 apache2 24627 efs 15 0 16964 12m 1968 S 0.0 2.6 0:06.68 MiraLoader.pl 24662 www-data 15 0 29564 10m 1632 S 0.0 2.2 0:02.91 apache2 24668 www-data 16 0 29564 10m 1624 S 0.0 2.2 0:02.50 apache2 24663 www-data 15 0 29564 10m 1620 S 0.0 2.2 0:02.78 apache2 etc.. (the other 2 java processes are separate java daemons, nothing to do with Tomcat). At this point Tomcat and this external application are very responsive, at least in single-user mode. We have not really done yet a multi-user test. Now, if I stop and restart Tomcat, for something like 6 minutes after that, the same top display shows this kind of thing : (There are 4 consecutive snapshots here, taken at more or less regular intervals during the 6 minutes) I) top - 12:44:56 up 162 days, 18:54, 3 users, load average: 0.99, 0.27, 0.08 Tasks: 85 total, 3 running, 82 sleeping, 0 stopped, 0 zombie Cpu(s): 98.0%us, 2.0%sy, 0.0%ni, 0.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st Mem:516920k total, 351160k used, 165760k free,54548k buffers Swap: 1297564k total,7k used, 1227564k free, 128152k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 2864 tomcat55 25 0 463m 40m 11m R 97.9 7.9 0:25.62 java 5937 root 16 0 267m 24m 8056 S 0.0 4.9 25:38.84 java 24608 root 23 0 260m 15m 9332 S 0.0 3.1 0:02.17 java 24638 root 15 0 29564 15m 5984 S 0.0 3.0 0:02.16 apache2 24627 efs 15 0 16964 12m 1968 S 0.0 2.6 0:06.74 MiraLoader.pl 24664 www-data 15 0 29692 12m 3428 S
RE: Tomcat threads, II
From: André Warnier [mailto:[EMAIL PROTECTED] Subject: Re: Tomcat threads, II I have turned it off now, and restarted Tomcat, but it did not have any noticeable impact on the startup time. Wouldn't expect it to, since you're spending nearly all the time parsing that XML file. It may have an effect when you actually start processing requests, but the impact may be small. As far as I can tell, these settings come from the catalina.sh script. I have no idea if this script is modified, compared to the Official version. Yes, the Debians screwed it up, by trying to patch it to work with gcj (a futile and pointless task). The real script only adds the extra jars if you specify an alternate secure sockets library with the JSSE_HOME environment variable. it seems to have survived (apart from bootstrap.jar, which I had to leave) Yes, bootrstrap.jar is the only thing that should be on the classpath. But that did not apparently reduce its startup time much. If your startup were not completely dominated by the XML parsing, you should see some difference. If anything, it seems now to occupy more memory than before : Is there an explanation for that ? No, the exact amount probably varies considerably over time and really isn't anything to worry about. I also don't seem to be able to run JConsole. You'll need to attach remotely. Assuming your target system is protected from potential threats by some external means, you can set jmxremote handling without authentication by adding these parameters for starting Tomcat: -Dcom.sun.management.jmxremote.port=jmxport -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false (all on one line, of course). Use CATALINA_OPTS or JAVA_OPTS to set them. Then run JConsole from your desktop, and give it the IP address and JMX port of the Tomcat machine in the Remote Process box. Full reference is here: http://java.sun.com/j2se/1.5.0/docs/tooldocs/share/jconsole.html http://java.sun.com/j2se/1.5.0/docs/guide/management/agent.html - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat threads, II
Pid wrote: ... Having said that, parsing XML can be a memory intensive operation as well, so your old box could be experiencing problems there - check the garbage collection log. p I try to avoid XML for this exact reason. We found that our application could handle between 6 and 10 times more concurrent requests on the same hardware when we went away from XML data transmission, and switched to a simple text-based HTTP POST request. D - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat threads, II
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, Christopher Schultz wrote: A few parting thoughts: I knew I was forgetting something: 3. During your 5-minute startup freeze, try taking a few thread dumps (send a QUIT signal to your JVM) and seeing what the app is doing. Maybe it's just taking forever to parse that big old XML document, but it could be something else entirely. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkkbUQwACgkQ9CaO5/Lv0PAL+QCdFG5HdLVeEnRd2kFUoE6BBmRE 0iQAnRr3BJxC/cMN5tmEgKB+bEIR1VSA =zqaM -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat threads, II
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
André,
At the risk of repeating things said by others...
André Warnier wrote:
We have an old clunker PC (Pentium II, 512 Mb Ram, average ATA disks),
which we use to test some applications (not only Java/Tomcat).
Based upon this hardware configuration, things are going to suck no
matter what else you do. Just bear that in mind going forward.
-Xmx256M
[snip]
6226 tomcat55 25 0 506m 191m 10m S 0.0 37.9 140:26.15 java
Interesting that the JVM grows to nearly twice the size of the maximum
heap that you gave to it. If it performs okay, then I wouldn't worry too
much about that, but I'm surprised that the total virtual size is so big
given your (relatively) small maximum heap.
Now, if I stop and restart Tomcat, for something like 6 minutes after
that, the same top display shows this kind of thing :
(There are 4 consecutive snapshots here, taken at more or less regular
intervals during the 6 minutes)
[snipped top output showing large memory use]
In other words, while this application is being loaded, our Tomcat and
the whole system are totally unresponsive for about 5 minutes.
When you say the whole system, do you mean that you can't get other
services to respond? What about something simple like 'ls' on a
command-line? Is the whole machine thrashing or is it just that Tomcat
appears dead for 5 minutes or so?
I agree with your assessment that your application is at fault; neither
Java nor Tomcat are the problem, here.
Now my questions are of the kind :
Our production servers are ASP servers, where several customers would be
using this same application, each customer with its own 25 Mb XML file
of data. They are more powerful servers, but if I have 5 customers and
these things compete with one another for memory or resources, I could
have a server that is unresponsive for a hour maybe at start.
You are absolutely right to be considering the scalability of your
application, and I would be worried, too. The good news is that web
applications spend most of their time running and very little of it
initializing themselves. ;)
So basically, I am asking if there are any parameters I can vary for the
application startup or Tomcat in general, to evaluate the behaviour
under different circumstances
Nope. If the JVM and Tomcat startup quickly (15 seconds is just fine on
that hardware) then the only thing you could do to help your application
is to optimize the initialization.
Actually, before I say never, you might want to check on two things:
1. DNS settings. Especially when parsing XML (which often requires
looking up a DTD or XML Schema file on a remote server), you might be
hitting a problem where your DNS resolver is taking a long time, but
finally succeeding (which is why you don't see any errors). This was
happening to us when our hosting provider took a DNS server out of
service without telling us: our lookups against our primary DNS server
were timing out (after tens of seconds), but the secondary server worked
right away. So, we had to wait for the primary to time out before
getting an instant response. The result was that most DNS lookups took
forever and it was a simple DNS configuration fix.
2. Cryptographic entropy. If your application ultimately uses
/dev/random for random numbers, you may be running out of entropy and
your application may be blocking during startup waiting for more. You
can modify your JVM configuration to use /dev/urandom which will not
block in these situations. Most recent JVMs are rigged to use
/dev/urandom on Linux and I don't suspect that this is your problem.
[Should I] put more RAM in the server[?]
Always! ;)
or if I should lower or raise the amount of memory available to
Tomcat (and how), etc.
This is very application-specific. If you aren't running out of memory,
there's no reason to increase your heap size. If you set your heap too
large, you'll end up with lots of swapping which will kill your performance.
I am also a bit surprised about the amounts I see in terms of memory
usage, but not being an expert, I have not really anything to compare
them with. It's just that these java applications seem very hungry in
memory, if what is shown is really what they are using. For comparison,
we have other applications running on that server, of comparable
complexity, and they use 10-15 Mb apiece.
Just remember a few things about your application:
1. Java is a memory pig
2. XML is a memory pig (at least during parsing)
The faster you can throw String objects out of memory, the better.
(A complete digression here:
I hope this has been fixed, but old DOM parsers used to do this when you
called element.getText() (or whatever the equivalent is):
public String getText()
{
return new String(wholeDocument.substring(start, end));
}
This returns a piece of the whole XML document. That doesn't sound
like a bad idea until you consider how Java's String.substring works.
The String class
Re: Tomcat threads, II
André Warnier wrote: Now that I have (with your help) established some basic facts, I have a practical case for analysis. We have an old clunker PC (Pentium II, 512 Mb Ram, average ATA disks), which we use to test some applications (not only Java/Tomcat). ... Now, if I stop and restart Tomcat, for something like 6 minutes after that, the same top display shows this kind of thing : (There are 4 consecutive snapshots here, taken at more or less regular intervals during the 6 minutes) Comments on your situation and top snapshots: - you're low on memory, but not paging heavily yet; you're likely taking some kind of hit from not having enough memory for a decent buffer cache (disk cache), though - your swap usage remains constant to 7k (on which I base my claim that you're not paging heavily) - CPU usage, at least based on these snapshots, is almost pure user-mode CPU use, so there's little else happening than some computation internal to the application/Tomcat - looks like you could do with slightly smaller Java heap size, at least at this point of execution -- but it also could be that a smaller heap would get you an OutOfMemoryError later on - based on the fact that 'top' shows only 188M memory resident of the total 482M address space, the application is not even close to heap size (thus GC limits), but you could verily the GC issue by enabling verbose garbage collection for your JVM (with -verbose:gc flag to Java startup parameters) - you could also see which of the threads are taking time, with ps -fLp pid-of-tomcat-java (and getting the thread information f.ex. by getting a thread dump with kill -QUIT pid-of-tomcat-java to catalina.out In other words, while this application is being loaded, our Tomcat and the whole system are totally unresponsive for about 5 minutes. Whole system - as in ls from command line taking ages to complete when the startup is running? If so, then this could indicate a bottleneck in disk I/O (and as such, would be, partially at least, in line with my comment on not having enough memory for a buffer cache). However, this could also contradict my assumption that paging is not the issue. If it's a memory issue, you could help it a bit by lowering the heap size limit for the Tomcat JVM (but therein lies the risk of getting an OOME). However, as you noted, the machine is weak in terms of memory capacity and CPU power -- and I would suspect in hard drive speed just as well. And what it is you're doing: loading a largish file from the drive (while having minimal amount of memory available for buffer cache), and doing some memory- and CPU-intensive processing (XML parsing) on it. So, I would encourage you to have a look at the start times of the application on one of your production servers, preferably at a time of low traffic. In case of hangs, it might be good to have a command-line ready with kill -9 pid-of-tomcat-java to forcibly end the Tomcat process (however I strongly suspect you're not going to need it). -- ..Juha - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat threads, II
Christopher Schultz schrieb am 12.11.2008 um 16:42:06 (-0500): André Warnier wrote: In other words, while this application is being loaded, our Tomcat and the whole system are totally unresponsive for about 5 minutes. 1. DNS settings. Especially when parsing XML (which often requires looking up a DTD or XML Schema file on a remote server), you might be hitting a problem where your DNS resolver is taking a long time, but finally succeeding (which is why you don't see any errors). If this is happening here (and it may be), consider using XML catalogs, which are simply local filesystem caches of XML/SGML resources. (On Debian, IIRC, the package is called xml-base.) Waiting on the net, however, wouldn't account for your server's unresponsiveness. Processing 25 MB of XML in memory (as opposed to by stream), as in DOM or XSL, may easily consume 200 MB of RAM. If it's XSL and not done properly, five minutes of CPU hogging are easily attainable :-) I'd ask the vendor to explain what's going on. Michael Ludwig - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat threads, II
Chris, Chuck and others, many thanks for taking the time to educate me (on both Tomcat threads threads). I got lots of information and tips, which will be useful now or later. I'll now go sift through them again. At least now I have an idea where to start. About the fact that my hardware sucks, for Java : I know. On the other hand, that machine is a good filter against programs' and programmer's hubris, particularly Java ones. If it runs there, it will run anywhere kind of thing, and I don't need 50 fake clients to stress it out (obviously). On the other hand again, on the same machine I have a text search and retrieval application that can sift through a full-text index of 100,000 documents (1 Gb of text) and retrieve the ones I want in couple of seconds. It has a 10 Mb memory footprint. That's why the 500 Mb footprint of Tomcat (with the app) and the 5 minute delay in starting the app over 25 Mb of XML so struck me. I also have learned (separately, and confirmed here several times) that XML parsing is a hog, and that is not only in Java. Particularly the DOM-style of parsing exhibits exponential time behaviour in relation to document size. Large text fields are absolute killers, and making them CDATA only partly alleviates that. One can always throw more hardware at things, and sometimes it may be cheaper than trying to over-optimise. But some applications out there will kill any hardware. It is sometimes surprisingly easy to gain a factor 2 with little investment though, and if that means halving the number of servers and their attendant care and paraphernalia, it's still worth it for us. Even when they are virtual. Our main business is processing documents, text-intensive, that's why I am interested. Gaining 5 seconds in processing a document counts, if you're processing thousands per day. For a user with his finger on the mouse button too, there is a lot of difference between 1 and 3 seconds. A note here of one of you regarding substrings in Java has particularly caught my interest. And I'll go check if the XML parser in that application could be replaced by a newer version maybe. One alternative to XML in feeding that application with data is CSV files (the text version of spreadsheet). I had discarded it until now as old-fashioned, passé, limited etc.. XML is so much more in. But I am having second thoughts now, and I will give it a try. When I started in this business, 64 Kb was a nice quantity of memory to program in, and quite expensive too. I created and ran a payroll application for a 1,000 people company in there. This Java app looks a lot cuter than the payroll did, but 500 Megabyte of memory for one single Tomcat app, mmm. Some reflexes remain for a lifetime. Thanks. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat threads, II
André Warnier schrieb am 13.11.2008 um 00:08:05 (+0100): [...] on the same machine I have a text search and retrieval application that can sift through a full-text index of 100,000 documents (1 Gb of text) and retrieve the ones I want in couple of seconds. It has a 10 Mb memory footprint. That's why the 500 Mb footprint of Tomcat (with the app) and the 5 minute delay in starting the app over 25 Mb of XML so struck me. The app vendor should (a) seperate XML data analysis (which is what it seems to be doing) from application server startup for both time and memory reasons, (b) make the result of the analysis available to the server in a compact and easily digestible form, and (c) make data analysis run in 10 seconds instead of five minutes, which may well be feasible. I also have learned (separately, and confirmed here several times) that XML parsing is a hog, and that is not only in Java. Particularly the DOM-style of parsing exhibits exponential time behaviour in relation to document size. Large text fields are absolute killers, and making them CDATA only partly alleviates that. I don't believe it's the parsing that takes so much time here. There must be some subsequent inefficient processing going on. One alternative to XML in feeding that application with data is CSV files (the text version of spreadsheet). I had discarded it until now as old-fashioned, passé, limited etc.. XML is so much more in. But I am having second thoughts now, and I will give it a try. If your data is indeed tabular, CSV is probably fine. Some reflexes remain for a lifetime. Fortunately! Michael Ludwig - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
