Re: Tomcat not using multiple cores

2008-10-17 Thread János Löbb


On Oct 16, 2008, at 9:05 PM, Matthew Laird wrote:



Caldarale, Charles R wrote:

The only time I began to see the other cores actually start being  
used is when I enabled multi-threaded GC.  But that doesn't give  
much improvement since the threads responding the web requests are  
still all on the same core.
The most likely cause is internal synchronization in the webapp or  
the database it references.


I've heard similar from someone else but I'm not sure how that's  
possible.  The app is not at all threaded, is 100% read-only from  
the database (aside from creation of temp tables which have no  
interaction between client requests) and the MySQL server is on  
another machine humming along without any bottlenecks that I can see.


But of course I'm not a Java or JVM expert and have no idea what  
kind of interlinks can exist between different client connections.   
But to my knowledge they're all pretty straight forward handlers,  
get data from database, format, return to user


I'm leaning more towards GC issues.  I setup Tomcat on a 64-bit  
machine and tried a few configurations:


-Xms4096M -Xmx4096M -server -XX:+DisableExplicitGC -XX: 
+UseConcMarkSweepGC


-Xms4096M -Xmx4096M -server

-Xms2048M -Xmx2048M -server

As I went through each of those the app became more and more  
sluggish and a single core finally in the last configuration did  
this 100% CPU usage again.


Now, two issues I see.  First, it's not being aggressive enough at  
spreading the load among different cores.  Second, GC does seem to  
be an issue.


Unless I'm missing something, which I might be.

Thanks.



Make sure IP6 is not enabled if You are an IP4 network.  We had  
problems like You describe because of it.



-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Tomcat not using multiple cores

2008-10-17 Thread Dan Armbrust
My guesses:

Your application has bad sync locking between some threads, which is
causing one thread to block the others.

Your application has bad database access code, which allows one thread
to block others while waiting on the DB.

Your application is disk bound, and the single long process is hashing
the disk drives so much that is has dragged down the rest of the OS,
making it very slow to respond to the IO request of the other, small
tasks.

You haven't configured the GC  well - however, I would expect this to
cause sporadic  problems, rather than continuous ones.

There are numerous JVM tools out there to help you with all of these
issues - from the simple garbage collection monitor that comes with
java (JConsole 
http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html)
 to full blown tools like JProfiler (which last I checked, had a free
trial)

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Tomcat not using multiple cores

2008-10-16 Thread Caldarale, Charles R
 From: Matthew Laird [mailto:[EMAIL PROTECTED]
 Subject: Tomcat not using multiple cores

 Unfortunately what then occurs is all other threads suddenly become
 unusably slow.  The entire web application grinds to a halt until this
 thread that's running hot completes.

We've run Tomcat on 32 cores and can keep them all busy - the problem is almost 
certainly in your webapp.  Take a thread dump (with locking information 
enabled) and see what everyone is waiting for.  The jstack utility is good for 
this.

 I'm not sure how to convince the Tomcat/Java container to spread its
 threads among the cores.

Convince your application; it's not Tomcat or the JVM.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Tomcat not using multiple cores

2008-10-16 Thread Leon Rosenberg
Hello Matthew,

Without wanting to advertise own product, I think moskito can help you
a lot here. It was specially designed to detect
cpu eaters among call trees. Feel free to contact me of list if you
want to give it a try ;-)

http://moskito.anotheria.net/moskitodemo/mui/mskShowAllProducers

regards
Leon

On Thu, Oct 16, 2008 at 10:30 PM, Matthew Laird [EMAIL PROTECTED] wrote:
 We're pulling our hair out with a Tomcat issue.

 We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6.
  The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of
 RAM.  We're running OpenSuSE 10.2, 32-bit.  Java memory size set to 2GB,
 multi-threaded GC enabled.

 What occurs is when a user clicks a certain kind of analysis on the
 website, data is retrieved from a database and then a lot of formatting
 is done before returning it to the user.  This typically causes 100% CPU
 usage for this thread for a few minutes (bioinformatics application,
 that part isn't going to change).

 Unfortunately what then occurs is all other threads suddenly become
 unusably slow.  The entire web application grinds to a halt until this
 thread that's running hot completes.

 Looking at top, it appears that these threads aren't spreading among all
 the cores.  I see one core go to 100% usage, and the others stay at 100%
 idle.  So we're running multi-thread, but because everything is staying
 on the same core, we're still getting thread contention that's bringing
 the entire application to its knees.

 The only time I began to see the other cores actually start being used
 is when I enabled multi-threaded GC.  But that doesn't give much
 improvement since the threads responding the web requests are still all
 on the same core.

 I'm not sure how to convince the Tomcat/Java container to spread its
 threads among the cores.

 Thanks.

 --
 Matthew Laird
 Lead Software Developer, Bioinformatics
 Brinkman Laboratory, MBB Dept.
 Simon Fraser University


 -
 To start a new topic, e-mail: users@tomcat.apache.org
 To unsubscribe, e-mail: [EMAIL PROTECTED]
 For additional commands, e-mail: [EMAIL PROTECTED]



-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Tomcat not using multiple cores

2008-10-16 Thread Matthew Laird

From the OS, no.

From Tomcat, as far as I understand you can only do 2GB per Tomcat 
instance.  Please correct me if I'm wrong.


Jim Cox wrote:

On Thu, Oct 16, 2008 at 10:30 PM, Matthew Laird [EMAIL PROTECTED] wrote:
[...lines snipped...]

We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6.
 The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of
RAM.  We're running OpenSuSE 10.2, 32-bit.  Java memory size set to 2GB,
multi-threaded GC enabled.

[...rest of post snipped...]

Apologizing in advance for straying a off-topic, but have you had any issues
seeing the full 16GB with a 32-bit Linux install?


-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Tomcat not using multiple cores

2008-10-16 Thread Caldarale, Charles R
 From: Matthew Laird [mailto:[EMAIL PROTECTED]
 Subject: Re: Tomcat not using multiple cores

 From Tomcat, as far as I understand you can only do 2GB per Tomcat
 instance.  Please correct me if I'm wrong.

Not true.  The limitation is due to the OS, not the JVM and certainly not 
Tomcat.  A 32-bit OS gives each process only 2 or 3 GB of virtual space to work 
with, and this space must hold the Java heap, code, non-Java data space, and 
myriad resource buffers of varying types.

A 64-bit OS provides a much larger virtual space for each process, so you can 
configure huge heap spaces for Java programs, including Tomcat.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Tomcat not using multiple cores

2008-10-16 Thread Brantley Hobbs

I think that this is a garbage collection issue.

Enable the garbage collection output to see if that's the case (the 
-verbose:gc flag) .  I struggled with this myself just a month ago and 
ended up learning more about Java GC than I ever wanted to know.  The 
gist of it is that when the JVM runs out of heap memory and does a major 
collection, it will pause all threads while it's collecting.  It doesn't 
matter what the priority of the thread is or whether or not you are 
using a parallel GC.  As you approach heap memory limits, the GC is 
working its tail off trying to recover memory wherever it can, pausing 
all JVM activity as necessary.  I have a feeling that you'll see more 
and more GC calls as the app grinds to a halt.


A couple of points:
Disabling GC hints can sometimes dramatically increase performance if 
you're not actually running out of memory.  Don't assume that just 
because *your* code isn't requesting a collection that a third-party 
library isn't. 

As someone pointed out elsewhere, you're going to want to switch to a 64 
bit OS and JRE at some point, if you want to take full advantage of all 
that RAM.


In case you're curious, these are the JRE flags that we ended up running 
with:

-XX:+UseParallelGC
-XX:+AggressiveOpts
-XX:+DisableExplicitGC
-XX:+UseAdaptiveSizePolicy
-XX:MaxPermSize=512m

(Obviously the MaxPermSize flag is dependent on how your applications 
allocate memory.)


A profiler will be helpful too.  I've had some success using the one 
built-in to Netbeans.


Good Luck!
Brantley



Matthew Laird wrote:

We're pulling our hair out with a Tomcat issue.

We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6.
 The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of
RAM.  We're running OpenSuSE 10.2, 32-bit.  Java memory size set to 2GB,
multi-threaded GC enabled.

What occurs is when a user clicks a certain kind of analysis on the
website, data is retrieved from a database and then a lot of formatting
is done before returning it to the user.  This typically causes 100% CPU
usage for this thread for a few minutes (bioinformatics application,
that part isn't going to change).

Unfortunately what then occurs is all other threads suddenly become
unusably slow.  The entire web application grinds to a halt until this
thread that's running hot completes.

Looking at top, it appears that these threads aren't spreading among all
the cores.  I see one core go to 100% usage, and the others stay at 100%
idle.  So we're running multi-thread, but because everything is staying
on the same core, we're still getting thread contention that's bringing
the entire application to its knees.

The only time I began to see the other cores actually start being used
is when I enabled multi-threaded GC.  But that doesn't give much
improvement since the threads responding the web requests are still all
on the same core.

I'm not sure how to convince the Tomcat/Java container to spread its
threads among the cores.

Thanks.




-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Tomcat not using multiple cores

2008-10-16 Thread Brantley Hobbs

This will be a helpful document:

http://java.sun.com/docs/hotspot/gc5.0/ergo5.html

Brantley

Matthew Laird wrote:

We're pulling our hair out with a Tomcat issue.

We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6.
 The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of
RAM.  We're running OpenSuSE 10.2, 32-bit.  Java memory size set to 2GB,
multi-threaded GC enabled.

What occurs is when a user clicks a certain kind of analysis on the
website, data is retrieved from a database and then a lot of formatting
is done before returning it to the user.  This typically causes 100% CPU
usage for this thread for a few minutes (bioinformatics application,
that part isn't going to change).

Unfortunately what then occurs is all other threads suddenly become
unusably slow.  The entire web application grinds to a halt until this
thread that's running hot completes.

Looking at top, it appears that these threads aren't spreading among all
the cores.  I see one core go to 100% usage, and the others stay at 100%
idle.  So we're running multi-thread, but because everything is staying
on the same core, we're still getting thread contention that's bringing
the entire application to its knees.

The only time I began to see the other cores actually start being used
is when I enabled multi-threaded GC.  But that doesn't give much
improvement since the threads responding the web requests are still all
on the same core.

I'm not sure how to convince the Tomcat/Java container to spread its
threads among the cores.

Thanks.




-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Tomcat not using multiple cores

2008-10-16 Thread Caldarale, Charles R
 From: Brantley Hobbs [mailto:[EMAIL PROTECTED]
 Subject: Re: Tomcat not using multiple cores

 I think that this is a garbage collection issue.

Extremely unlikely, since the OP has already stated:

The only time I began to see the other cores actually start being used is when 
I enabled multi-threaded GC.  But that doesn't give much improvement since the 
threads responding the web requests are still all on the same core.

The most likely cause is internal synchronization in the webapp or the database 
it references.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Tomcat not using multiple cores

2008-10-16 Thread Brantley Hobbs
This is the exact thing that makes me think that that's the problem.  I 
saw the exact same symptoms:  No matter how many CPU's I threw at it and 
no matter how I threaded my app, I was only consuming one CPU in total.


I know it sounds counter-intuitive, and I could easily be wrong (I'm no 
JVM memory expert), but this is almost verbatim the exact same problem I 
had, and it was solved with GC tuning.


I'm no gambler, but if I were I'd be willing bet that when he enables GC 
verbose output that he'll see more and more time going to major collections.


Brantley

Caldarale, Charles R wrote:

From: Brantley Hobbs [mailto:[EMAIL PROTECTED]
Subject: Re: Tomcat not using multiple cores

I think that this is a garbage collection issue.



Extremely unlikely, since the OP has already stated:

The only time I began to see the other cores actually start being used is when I 
enabled multi-threaded GC.  But that doesn't give much improvement since the threads 
responding the web requests are still all on the same core.

The most likely cause is internal synchronization in the webapp or the database 
it references.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

  



-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Tomcat not using multiple cores

2008-10-16 Thread Matthew Laird


Caldarale, Charles R wrote:


The only time I began to see the other cores actually start being used is when I 
enabled multi-threaded GC.  But that doesn't give much improvement since the threads 
responding the web requests are still all on the same core.

The most likely cause is internal synchronization in the webapp or the database 
it references.


I've heard similar from someone else but I'm not sure how that's 
possible.  The app is not at all threaded, is 100% read-only from the 
database (aside from creation of temp tables which have no interaction 
between client requests) and the MySQL server is on another machine 
humming along without any bottlenecks that I can see.


But of course I'm not a Java or JVM expert and have no idea what kind of 
interlinks can exist between different client connections.  But to my 
knowledge they're all pretty straight forward handlers, get data from 
database, format, return to user


I'm leaning more towards GC issues.  I setup Tomcat on a 64-bit machine 
and tried a few configurations:


-Xms4096M -Xmx4096M -server -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC

-Xms4096M -Xmx4096M -server

-Xms2048M -Xmx2048M -server

As I went through each of those the app became more and more sluggish 
and a single core finally in the last configuration did this 100% CPU 
usage again.


Now, two issues I see.  First, it's not being aggressive enough at 
spreading the load among different cores.  Second, GC does seem to be an 
issue.


Unless I'm missing something, which I might be.

Thanks.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



RE: Tomcat not using multiple cores

2008-10-16 Thread Caldarale, Charles R
 From: Matthew Laird [mailto:[EMAIL PROTECTED]
 Subject: Re: Tomcat not using multiple cores

 But of course I'm not a Java or JVM expert and have no idea
 what kind of interlinks can exist between different client
 connections.

Again - get a thread dump when at 100% on one core and 0% on the others so you 
will have real information rather than speculation.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Tomcat not using multiple cores

2008-10-16 Thread Brantley Hobbs

Matt,

Chuck is right.  You want to get real information.  Try the verbose GC 
output flag on the JVM and see what that says.  If you /don't/ see a 
bunch of major collections in the output while that task is running, 
then GC is not your problem.


B.

Caldarale, Charles R wrote:

From: Matthew Laird [mailto:[EMAIL PROTECTED]
Subject: Re: Tomcat not using multiple cores

But of course I'm not a Java or JVM expert and have no idea
what kind of interlinks can exist between different client
connections.



Again - get a thread dump when at 100% on one core and 0% on the others so you 
will have real information rather than speculation.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]

  



-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]



Re: Tomcat not using multiple cores

2008-10-16 Thread Leon Rosenberg
On Fri, Oct 17, 2008 at 3:05 AM, Matthew Laird [EMAIL PROTECTED] wrote:

 Caldarale, Charles R wrote:

 The only time I began to see the other cores actually start being used is
 when I enabled multi-threaded GC.  But that doesn't give much improvement
 since the threads responding the web requests are still all on the same
 core.

 The most likely cause is internal synchronization in the webapp or the
 database it references.

 I've heard similar from someone else but I'm not sure how that's possible.
  The app is not at all threaded, is 100% read-only from the database (aside
 from creation of temp tables which have no interaction between client
 requests) and the MySQL server is on another machine humming along without
 any bottlenecks that I can see.

And how many connections to the database do you use and which kind of
connections?

Leon

-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]