Re: Tomcat not using multiple cores
My guesses: Your application has bad sync locking between some threads, which is causing one thread to block the others. Your application has bad database access code, which allows one thread to block others while waiting on the DB. Your application is disk bound, and the single long process is hashing the disk drives so much that is has dragged down the rest of the OS, making it very slow to respond to the IO request of the other, small tasks. You haven't configured the GC well - however, I would expect this to cause sporadic problems, rather than continuous ones. There are numerous JVM tools out there to help you with all of these issues - from the simple garbage collection monitor that comes with java (JConsole http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html) to full blown tools like JProfiler (which last I checked, had a free trial) - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat not using multiple cores
On Oct 16, 2008, at 9:05 PM, Matthew Laird wrote: Caldarale, Charles R wrote: "The only time I began to see the other cores actually start being used is when I enabled multi-threaded GC. But that doesn't give much improvement since the threads responding the web requests are still all on the same core." The most likely cause is internal synchronization in the webapp or the database it references. I've heard similar from someone else but I'm not sure how that's possible. The app is not at all threaded, is 100% read-only from the database (aside from creation of temp tables which have no interaction between client requests) and the MySQL server is on another machine humming along without any bottlenecks that I can see. But of course I'm not a Java or JVM expert and have no idea what kind of interlinks can exist between different client connections. But to my knowledge they're all pretty straight forward handlers, "get data from database, format, return to user" I'm leaning more towards GC issues. I setup Tomcat on a 64-bit machine and tried a few configurations: -Xms4096M -Xmx4096M -server -XX:+DisableExplicitGC -XX: +UseConcMarkSweepGC -Xms4096M -Xmx4096M -server -Xms2048M -Xmx2048M -server As I went through each of those the app became more and more sluggish and a single core finally in the last configuration did this 100% CPU usage again. Now, two issues I see. First, it's not being aggressive enough at spreading the load among different cores. Second, GC does seem to be an issue. Unless I'm missing something, which I might be. Thanks. Make sure IP6 is not enabled if You are an IP4 network. We had problems like You describe because of it. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat not using multiple cores
On Fri, Oct 17, 2008 at 3:05 AM, Matthew Laird <[EMAIL PROTECTED]> wrote: > > Caldarale, Charles R wrote: > >> "The only time I began to see the other cores actually start being used is >> when I enabled multi-threaded GC. But that doesn't give much improvement >> since the threads responding the web requests are still all on the same >> core." >> >> The most likely cause is internal synchronization in the webapp or the >> database it references. > > I've heard similar from someone else but I'm not sure how that's possible. > The app is not at all threaded, is 100% read-only from the database (aside > from creation of temp tables which have no interaction between client > requests) and the MySQL server is on another machine humming along without > any bottlenecks that I can see. And how many connections to the database do you use and which kind of connections? Leon - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat not using multiple cores
Matt, Chuck is right. You want to get real information. Try the verbose GC output flag on the JVM and see what that says. If you /don't/ see a bunch of major collections in the output while that task is running, then GC is not your problem. B. Caldarale, Charles R wrote: From: Matthew Laird [mailto:[EMAIL PROTECTED] Subject: Re: Tomcat not using multiple cores But of course I'm not a Java or JVM expert and have no idea what kind of interlinks can exist between different client connections. Again - get a thread dump when at 100% on one core and 0% on the others so you will have real information rather than speculation. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat not using multiple cores
> From: Matthew Laird [mailto:[EMAIL PROTECTED] > Subject: Re: Tomcat not using multiple cores > > But of course I'm not a Java or JVM expert and have no idea > what kind of interlinks can exist between different client > connections. Again - get a thread dump when at 100% on one core and 0% on the others so you will have real information rather than speculation. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat not using multiple cores
Caldarale, Charles R wrote: "The only time I began to see the other cores actually start being used is when I enabled multi-threaded GC. But that doesn't give much improvement since the threads responding the web requests are still all on the same core." The most likely cause is internal synchronization in the webapp or the database it references. I've heard similar from someone else but I'm not sure how that's possible. The app is not at all threaded, is 100% read-only from the database (aside from creation of temp tables which have no interaction between client requests) and the MySQL server is on another machine humming along without any bottlenecks that I can see. But of course I'm not a Java or JVM expert and have no idea what kind of interlinks can exist between different client connections. But to my knowledge they're all pretty straight forward handlers, "get data from database, format, return to user" I'm leaning more towards GC issues. I setup Tomcat on a 64-bit machine and tried a few configurations: -Xms4096M -Xmx4096M -server -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -Xms4096M -Xmx4096M -server -Xms2048M -Xmx2048M -server As I went through each of those the app became more and more sluggish and a single core finally in the last configuration did this 100% CPU usage again. Now, two issues I see. First, it's not being aggressive enough at spreading the load among different cores. Second, GC does seem to be an issue. Unless I'm missing something, which I might be. Thanks. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat not using multiple cores
This is the exact thing that makes me think that that's the problem. I saw the exact same symptoms: No matter how many CPU's I threw at it and no matter how I threaded my app, I was only consuming one CPU in total. I know it sounds counter-intuitive, and I could easily be wrong (I'm no JVM memory expert), but this is almost verbatim the exact same problem I had, and it was solved with GC tuning. I'm no gambler, but if I were I'd be willing bet that when he enables GC verbose output that he'll see more and more time going to major collections. Brantley Caldarale, Charles R wrote: From: Brantley Hobbs [mailto:[EMAIL PROTECTED] Subject: Re: Tomcat not using multiple cores I think that this is a garbage collection issue. Extremely unlikely, since the OP has already stated: "The only time I began to see the other cores actually start being used is when I enabled multi-threaded GC. But that doesn't give much improvement since the threads responding the web requests are still all on the same core." The most likely cause is internal synchronization in the webapp or the database it references. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat not using multiple cores
> From: Brantley Hobbs [mailto:[EMAIL PROTECTED] > Subject: Re: Tomcat not using multiple cores > > I think that this is a garbage collection issue. Extremely unlikely, since the OP has already stated: "The only time I began to see the other cores actually start being used is when I enabled multi-threaded GC. But that doesn't give much improvement since the threads responding the web requests are still all on the same core." The most likely cause is internal synchronization in the webapp or the database it references. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat not using multiple cores
This will be a helpful document: http://java.sun.com/docs/hotspot/gc5.0/ergo5.html Brantley Matthew Laird wrote: We're pulling our hair out with a Tomcat issue. We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6. The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of RAM. We're running OpenSuSE 10.2, 32-bit. Java memory size set to 2GB, multi-threaded GC enabled. What occurs is when a user clicks a certain kind of analysis on the website, data is retrieved from a database and then a lot of formatting is done before returning it to the user. This typically causes 100% CPU usage for this thread for a few minutes (bioinformatics application, that part isn't going to change). Unfortunately what then occurs is all other threads suddenly become unusably slow. The entire web application grinds to a halt until this thread that's running hot completes. Looking at top, it appears that these threads aren't spreading among all the cores. I see one core go to 100% usage, and the others stay at 100% idle. So we're running multi-thread, but because everything is staying on the same core, we're still getting thread contention that's bringing the entire application to its knees. The only time I began to see the other cores actually start being used is when I enabled multi-threaded GC. But that doesn't give much improvement since the threads responding the web requests are still all on the same core. I'm not sure how to convince the Tomcat/Java container to spread its threads among the cores. Thanks. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat not using multiple cores
I think that this is a garbage collection issue. Enable the garbage collection output to see if that's the case (the "-verbose:gc" flag) . I struggled with this myself just a month ago and ended up learning more about Java GC than I ever wanted to know. The gist of it is that when the JVM runs out of heap memory and does a major collection, it will pause all threads while it's collecting. It doesn't matter what the priority of the thread is or whether or not you are using a parallel GC. As you approach heap memory limits, the GC is working its tail off trying to recover memory wherever it can, pausing all JVM activity as necessary. I have a feeling that you'll see more and more GC calls as the app grinds to a halt. A couple of points: Disabling GC hints can sometimes dramatically increase performance if you're not actually running out of memory. Don't assume that just because *your* code isn't requesting a collection that a third-party library isn't. As someone pointed out elsewhere, you're going to want to switch to a 64 bit OS and JRE at some point, if you want to take full advantage of all that RAM. In case you're curious, these are the JRE flags that we ended up running with: -XX:+UseParallelGC -XX:+AggressiveOpts -XX:+DisableExplicitGC -XX:+UseAdaptiveSizePolicy -XX:MaxPermSize=512m (Obviously the MaxPermSize flag is dependent on how your applications allocate memory.) A profiler will be helpful too. I've had some success using the one built-in to Netbeans. Good Luck! Brantley Matthew Laird wrote: We're pulling our hair out with a Tomcat issue. We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6. The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of RAM. We're running OpenSuSE 10.2, 32-bit. Java memory size set to 2GB, multi-threaded GC enabled. What occurs is when a user clicks a certain kind of analysis on the website, data is retrieved from a database and then a lot of formatting is done before returning it to the user. This typically causes 100% CPU usage for this thread for a few minutes (bioinformatics application, that part isn't going to change). Unfortunately what then occurs is all other threads suddenly become unusably slow. The entire web application grinds to a halt until this thread that's running hot completes. Looking at top, it appears that these threads aren't spreading among all the cores. I see one core go to 100% usage, and the others stay at 100% idle. So we're running multi-thread, but because everything is staying on the same core, we're still getting thread contention that's bringing the entire application to its knees. The only time I began to see the other cores actually start being used is when I enabled multi-threaded GC. But that doesn't give much improvement since the threads responding the web requests are still all on the same core. I'm not sure how to convince the Tomcat/Java container to spread its threads among the cores. Thanks. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat not using multiple cores
> From: Matthew Laird [mailto:[EMAIL PROTECTED] > Subject: Re: Tomcat not using multiple cores > > From Tomcat, as far as I understand you can only do 2GB per Tomcat > instance. Please correct me if I'm wrong. Not true. The limitation is due to the OS, not the JVM and certainly not Tomcat. A 32-bit OS gives each process only 2 or 3 GB of virtual space to work with, and this space must hold the Java heap, code, non-Java data space, and myriad resource buffers of varying types. A 64-bit OS provides a much larger virtual space for each process, so you can configure huge heap spaces for Java programs, including Tomcat. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat not using multiple cores
From the OS, no. From Tomcat, as far as I understand you can only do 2GB per Tomcat instance. Please correct me if I'm wrong. Jim Cox wrote: On Thu, Oct 16, 2008 at 10:30 PM, Matthew Laird <[EMAIL PROTECTED]> wrote: [...lines snipped...] We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6. The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of RAM. We're running OpenSuSE 10.2, 32-bit. Java memory size set to 2GB, multi-threaded GC enabled. [...rest of post snipped...] Apologizing in advance for straying a off-topic, but have you had any issues seeing the full 16GB with a 32-bit Linux install? - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat not using multiple cores
On Thu, Oct 16, 2008 at 10:30 PM, Matthew Laird <[EMAIL PROTECTED]> wrote: [...lines snipped...] > We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6. > The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of > RAM. We're running OpenSuSE 10.2, 32-bit. Java memory size set to 2GB, > multi-threaded GC enabled. [...rest of post snipped...] Apologizing in advance for straying a off-topic, but have you had any issues seeing the full 16GB with a 32-bit Linux install?
Re: Tomcat not using multiple cores
Hello Matthew, Without wanting to advertise own product, I think moskito can help you a lot here. It was specially designed to detect cpu eaters among call trees. Feel free to contact me of list if you want to give it a try ;-) http://moskito.anotheria.net/moskitodemo/mui/mskShowAllProducers regards Leon On Thu, Oct 16, 2008 at 10:30 PM, Matthew Laird <[EMAIL PROTECTED]> wrote: > We're pulling our hair out with a Tomcat issue. > > We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6. > The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of > RAM. We're running OpenSuSE 10.2, 32-bit. Java memory size set to 2GB, > multi-threaded GC enabled. > > What occurs is when a user clicks a certain kind of analysis on the > website, data is retrieved from a database and then a lot of formatting > is done before returning it to the user. This typically causes 100% CPU > usage for this thread for a few minutes (bioinformatics application, > that part isn't going to change). > > Unfortunately what then occurs is all other threads suddenly become > unusably slow. The entire web application grinds to a halt until this > thread that's running hot completes. > > Looking at top, it appears that these threads aren't spreading among all > the cores. I see one core go to 100% usage, and the others stay at 100% > idle. So we're running multi-thread, but because everything is staying > on the same core, we're still getting thread contention that's bringing > the entire application to its knees. > > The only time I began to see the other cores actually start being used > is when I enabled multi-threaded GC. But that doesn't give much > improvement since the threads responding the web requests are still all > on the same core. > > I'm not sure how to convince the Tomcat/Java container to spread its > threads among the cores. > > Thanks. > > -- > Matthew Laird > Lead Software Developer, Bioinformatics > Brinkman Laboratory, MBB Dept. > Simon Fraser University > > > - > To start a new topic, e-mail: users@tomcat.apache.org > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat not using multiple cores
> From: Matthew Laird [mailto:[EMAIL PROTECTED] > Subject: Tomcat not using multiple cores > > Unfortunately what then occurs is all other threads suddenly become > unusably slow. The entire web application grinds to a halt until this > thread that's running hot completes. We've run Tomcat on 32 cores and can keep them all busy - the problem is almost certainly in your webapp. Take a thread dump (with locking information enabled) and see what everyone is waiting for. The jstack utility is good for this. > I'm not sure how to convince the Tomcat/Java container to spread its > threads among the cores. Convince your application; it's not Tomcat or the JVM. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]