Tomcat not using multiple cores
We're pulling our hair out with a Tomcat issue. We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6. The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of RAM. We're running OpenSuSE 10.2, 32-bit. Java memory size set to 2GB, multi-threaded GC enabled. What occurs is when a user clicks a certain kind of analysis on the website, data is retrieved from a database and then a lot of formatting is done before returning it to the user. This typically causes 100% CPU usage for this thread for a few minutes (bioinformatics application, that part isn't going to change). Unfortunately what then occurs is all other threads suddenly become unusably slow. The entire web application grinds to a halt until this thread that's running hot completes. Looking at top, it appears that these threads aren't spreading among all the cores. I see one core go to 100% usage, and the others stay at 100% idle. So we're running multi-thread, but because everything is staying on the same core, we're still getting thread contention that's bringing the entire application to its knees. The only time I began to see the other cores actually start being used is when I enabled multi-threaded GC. But that doesn't give much improvement since the threads responding the web requests are still all on the same core. I'm not sure how to convince the Tomcat/Java container to spread its threads among the cores. Thanks. -- Matthew Laird Lead Software Developer, Bioinformatics Brinkman Laboratory, MBB Dept. Simon Fraser University - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat not using multiple cores
From the OS, no. From Tomcat, as far as I understand you can only do 2GB per Tomcat instance. Please correct me if I'm wrong. Jim Cox wrote: On Thu, Oct 16, 2008 at 10:30 PM, Matthew Laird [EMAIL PROTECTED] wrote: [...lines snipped...] We have an in-house application running on Tomcat 5.5 with Sun JDK 1.6. The machine is an x86 dual-CPU, quad core (8 cores total) with 16GB of RAM. We're running OpenSuSE 10.2, 32-bit. Java memory size set to 2GB, multi-threaded GC enabled. [...rest of post snipped...] Apologizing in advance for straying a off-topic, but have you had any issues seeing the full 16GB with a 32-bit Linux install? - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat not using multiple cores
Caldarale, Charles R wrote: The only time I began to see the other cores actually start being used is when I enabled multi-threaded GC. But that doesn't give much improvement since the threads responding the web requests are still all on the same core. The most likely cause is internal synchronization in the webapp or the database it references. I've heard similar from someone else but I'm not sure how that's possible. The app is not at all threaded, is 100% read-only from the database (aside from creation of temp tables which have no interaction between client requests) and the MySQL server is on another machine humming along without any bottlenecks that I can see. But of course I'm not a Java or JVM expert and have no idea what kind of interlinks can exist between different client connections. But to my knowledge they're all pretty straight forward handlers, get data from database, format, return to user I'm leaning more towards GC issues. I setup Tomcat on a 64-bit machine and tried a few configurations: -Xms4096M -Xmx4096M -server -XX:+DisableExplicitGC -XX:+UseConcMarkSweepGC -Xms4096M -Xmx4096M -server -Xms2048M -Xmx2048M -server As I went through each of those the app became more and more sluggish and a single core finally in the last configuration did this 100% CPU usage again. Now, two issues I see. First, it's not being aggressive enough at spreading the load among different cores. Second, GC does seem to be an issue. Unless I'm missing something, which I might be. Thanks. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
jk Status not showing errors
Good afternoon, I've been trying to get the jkstatus component of mod_jk running, and I'm not quite sure what I'm doing wrong in trying to have it report dead Tomcat instances. I have two tomcat instances setup in a load balancer, as a test I've taken down one of them. However the jkstatus screen still shows both of them as OK. I'm not sure what I'm missing from my workers.properties file to make it test the Tomcat and report a failed instance, so I can set Nagios to monitor this page and report problems. My workers.properties is: worker.list=production,development,old,jkstatus worker.production.type=lb worker.production.balance_workers=production1,production2 worker.production.sticky_session=True worker.production.method=S worker.lbbasic.type=ajp13 worker.lbbasic.connect_timeout=1 worker.lbbasic.recovery_options=7 worker.lbbasic.socket_keepalive=1 worker.lbbasic.socket_timeout=60 worker.production1.reference=worker.lbbasic worker.production1.port=8009 worker.production1.host=localhost #worker.production1.redirect=production2 worker.production2.reference=worker.lbbasic worker.production2.port=8012 worker.production2.host=localhost #worker.production2.activation=disabled worker.development.port=8010 worker.development.host=localhost worker.development.type=ajp13 worker.old.port=8011 worker.old.host=localhost worker.old.type=ajp13 worker.jkstatus.type=status Any advice on extra options to make jkstatus check and report when one of the Tomcat instances isn't responding would be appreciated. Thanks. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: jk Status not showing errors
Unfortunately I'm not seeing that. What I did was start both Tomcats in my LB pair, start Apache, then I take the second Tomcat down to see if it will detect it being failed. Unfortunately it never seems to, it just shows the second as OK/IDLE, and happily directs all requests to the first. This concerns me, because if the second were to fail, then later the first, everything would die and I'd have no advance warning. I can't seem to make it ping and detect a dead Tomcat. I am using the latest version of mod_jk, I upgraded that before I began playing with the load balancer settings. I'd appreciate any feedback on what I might be doing wrong. Thanks. workers.properties: worker.list=production,development,old,jkstatus worker.production.type=lb worker.production.balance_workers=production1,production2 worker.production.sticky_session=True worker.production.method=S worker.lbbasic.type=ajp13 worker.lbbasic.connect_timeout=1 worker.lbbasic.recovery_options=7 worker.lbbasic.socket_keepalive=1 worker.lbbasic.socket_timeout=60 worker.lbbasic.ping_mode=CI worker.production1.reference=worker.lbbasic worker.production1.port=8009 worker.production1.host=localhost worker.production2.reference=worker.lbbasic worker.production2.port=8012 worker.production2.host=localhost worker.development.port=8010 worker.development.host=localhost worker.development.type=ajp13 worker.old.port=8011 worker.old.host=localhost worker.old.type=ajp13 worker.jkstatus.type=status Lawrence Lamprecht wrote: I do not know if this is relevant or not, but I have just installed the latest version of mod_jk and the jkstatus is very much better than it used to be. I had the same issue with loadbalancers not showing when they are offline or broken. With the latest version, jksataus has the possibility to auto refresh itself. This now shouws when load balancers go down without a request being send to it. It is pretty dynamic as well. I ran several tests where I took one of the balancers down, and left jkstatus refreshing every 10 seconds and that told me that the worker was in error. It also shows you that the work is OK - IDLE when the worker is not being used but is good. As soon as it receives a request the status then changes to OK. Hope this helps. Kind regards / Met vriendelijke groet, Lawrence Lamprecht Application Content Manager QUADREM Netherlands B.V. Kabelweg 61, 1014 BA Amsterdam Post Office Box 20672, 1001 NR Amsterdam Office: +31 20 880 41 16 Mobile: +31 6 13 14 26 31 Fax: +31 20 880 41 02 Read our blog: Intelligent Supply Management - Your advantage -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Saturday, May 30, 2009 2:46 PM To: Tomcat Users List Subject: Re: jk Status not showing errors On 29.05.2009 22:50, Matthew Laird wrote: Good afternoon, I've been trying to get the jkstatus component of mod_jk running, and I'm not quite sure what I'm doing wrong in trying to have it report dead Tomcat instances. I have two tomcat instances setup in a load balancer, as a test I've taken down one of them. However the jkstatus screen still shows both of them as OK. I'm not sure what I'm missing from my workers.properties file to make it test the Tomcat and report a failed instance, so I can set Nagios to monitor this page and report problems. My workers.properties is: worker.list=production,development,old,jkstatus worker.production.type=lb worker.production.balance_workers=production1,production2 worker.production.sticky_session=True worker.production.method=S worker.lbbasic.type=ajp13 worker.lbbasic.connect_timeout=1 worker.lbbasic.recovery_options=7 worker.lbbasic.socket_keepalive=1 worker.lbbasic.socket_timeout=60 worker.production1.reference=worker.lbbasic worker.production1.port=8009 worker.production1.host=localhost #worker.production1.redirect=production2 worker.production2.reference=worker.lbbasic worker.production2.port=8012 worker.production2.host=localhost #worker.production2.activation=disabled worker.development.port=8010 worker.development.host=localhost worker.development.type=ajp13 worker.old.port=8011 worker.old.host=localhost worker.old.type=ajp13 worker.jkstatus.type=status Any advice on extra options to make jkstatus check and report when one of the Tomcat instances isn't responding would be appreciated. I assume, that the actual error detection works and you are really only asking about display in status worker. I also assume your are using a recent mod_jk. Nevertheless do yourself a favor and look at the Timeouts documentation page to improve your configuration. Until recently, only workers used via a load balancing worker had good manageability with jkstatus. Very recently also pure AJP workers without any load balancer got more useful information in their display. So let's talk about your worker production. Whenever a request comes in the lb first checks whether it already carries a session for one
Re: jk Status not showing errors
I'm not seeing anything like that. I just took both Tomcats down, I instantly get the 503 from Apache when I try to load the application. However tailing the mod_jk.log, I just see entries like this: [Tue Jun 02 12:36:23 2009] jkstatus www.innatedb.ca 0.000360 [Tue Jun 02 12:36:26 2009] jkstatus www.innatedb.ca 0.000263 [Tue Jun 02 12:36:39 2009] production www.innatedb.ca 0.498998 [Tue Jun 02 12:36:40 2009] jkstatus www.innatedb.ca 0.000282 mod_jk seems happy sending the requests to Tomcat, and doesn't seem to notice there's no actual Tomcat responding. Only after a few minutes does the JK Status screen go to ERR/REC for both. I would think this is the kind of thing mod_jk should notice instantly, when there's no Tomcat where there should be one. Or am I missing something? Thanks. Lawrence Lamprecht wrote: What you could do is tail -f mod_jk.log file. Then take down the tomcat, see if the errors appear. You should see something like the following. Good Entries to Track Attempting to map context URI '/search-engine*' ajp_unmarshal_response::jk_ajp_common.c (621): status = 302 Maintaining worker loadbalancer1 Maintaining worker prod_se1 Maintaining worker prod_se2 Maintaining worker prod_sea Maintaining worker prod_seb service::jk_lb_worker.c (612): service worker=prod_sea jvm_route=prod_sea service::jk_lb_worker.c (612): service worker=prod_seb jvm_route=prod_seb service::jk_lb_worker.c (612): service worker=prod_sea jvm_route=prod_se1 service::jk_lb_worker.c (612): service worker=prod_seb jvm_route=prod_se2 Possible Error Entries Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=prod_se1 failed Error connecting to tomcat. Tomcat is probably not started or is listening on the wrong port. worker=prod_se2 failed You should be able to trace where your config is problematic. Kind regards / Met vriendelijke groet, Lawrence Lamprecht -Original Message- From: Matthew Laird [mailto:lai...@sfu.ca] Sent: Tuesday, June 02, 2009 8:53 PM To: Tomcat Users List Subject: Re: jk Status not showing errors Unfortunately I'm not seeing that. What I did was start both Tomcats in my LB pair, start Apache, then I take the second Tomcat down to see if it will detect it being failed. Unfortunately it never seems to, it just shows the second as OK/IDLE, and happily directs all requests to the first. This concerns me, because if the second were to fail, then later the first, everything would die and I'd have no advance warning. I can't seem to make it ping and detect a dead Tomcat. I am using the latest version of mod_jk, I upgraded that before I began playing with the load balancer settings. I'd appreciate any feedback on what I might be doing wrong. Thanks. workers.properties: worker.list=production,development,old,jkstatus worker.production.type=lb worker.production.balance_workers=production1,production2 worker.production.sticky_session=True worker.production.method=S worker.lbbasic.type=ajp13 worker.lbbasic.connect_timeout=1 worker.lbbasic.recovery_options=7 worker.lbbasic.socket_keepalive=1 worker.lbbasic.socket_timeout=60 worker.lbbasic.ping_mode=CI worker.production1.reference=worker.lbbasic worker.production1.port=8009 worker.production1.host=localhost worker.production2.reference=worker.lbbasic worker.production2.port=8012 worker.production2.host=localhost worker.development.port=8010 worker.development.host=localhost worker.development.type=ajp13 worker.old.port=8011 worker.old.host=localhost worker.old.type=ajp13 worker.jkstatus.type=status Lawrence Lamprecht wrote: I do not know if this is relevant or not, but I have just installed the latest version of mod_jk and the jkstatus is very much better than it used to be. I had the same issue with loadbalancers not showing when they are offline or broken. With the latest version, jksataus has the possibility to auto refresh itself. This now shouws when load balancers go down without a request being send to it. It is pretty dynamic as well. I ran several tests where I took one of the balancers down, and left jkstatus refreshing every 10 seconds and that told me that the worker was in error. It also shows you that the work is OK - IDLE when the worker is not being used but is good. As soon as it receives a request the status then changes to OK. Hope this helps. Kind regards / Met vriendelijke groet, Lawrence Lamprecht Application Content Manager QUADREM Netherlands B.V. Kabelweg 61, 1014 BA Amsterdam Post Office Box 20672, 1001 NR Amsterdam Office: +31 20 880 41 16 Mobile: +31 6 13 14 26 31 Fax: +31 20 880 41 02 Read our blog: Intelligent Supply Management - Your advantage -Original Message- From: Rainer Jung [mailto:rainer.j...@kippdata.de] Sent: Saturday, May 30, 2009 2:46 PM To: Tomcat Users List Subject: Re: jk Status not showing errors On 29.05.2009 22:50, Matthew Laird wrote: Good afternoon, I've been trying to get
Re: jk Status not showing errors
Rainer Jung wrote: Assuming that you did refresh the jkstatus display: what is your test client? The fact that you see OK/IDLE, but all requests go to the other node indicates, that you are using requests with associated session, so the balancer is not allowed to send them to the other node and thus does not detect the down node. Check to remove the JSESSIONID cookie before sending requests, or use a client which allows cookie disabling (like curl). Is there any way to make it ping and detect a dead Tomcat without a request coming in? I thought I was doing that with the worker.lbbasic.ping_mode=CI setting. Thanks. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
