Limit user sessions in tomcat
Hello, We frequently have situations where a user has brought down a tomcat entirely by himself by running the same transaction multiple times because the response was not quick enough. Is there a way through configuation of tomcat and mod_jk to control the number of concurrent transactions/sessions a user can maintain? Since this is something that can happen to anyone out there, I am curious as to how you are handling this scenario.. Thanks, Chetan
Re: Limit user sessions in tomcat
Andre, We have a vast user population thats geographically dispersed, so implementing something thru the system would be the favourable approach.. Can you point me to links on the web that explain the add on modules and their implementation? All, Is there a 3rd party tool available to manage tomcat sessions and kill them once they go rogue? We dont have an inhouse development staff, so the best approach would be buying something off the shelf if it exists.. Thanks, Chetan From: Peter Crowther peter.crowt...@melandra.com To: Tomcat Users List users@tomcat.apache.org Sent: Tue, December 15, 2009 12:07:15 PM Subject: Re: Limit user sessions in tomcat 2009/12/15 André Warnier a...@ice-sa.com On another level, I don't quite understand yet how this squares with the fact that most browsers will not establish more than 2 connections with the same webserver at the same time. It seems a bit difficult to imagine that one single user can crash a Tomcat just by repeatedly hitting the same link. As far as the browser's concerned, clicking a link while a request is pending cancels the previous request (and generally closes the socket) and opens a new one. So it only has one connection open at any one time. As far as Tomcat's concerned - as shown by the recent emails on the topic - there's no way of detecting that closed socket and stopping its thread from trying to service it. So old requests build up, unwanted but impossible to discard until they complete or try to write something to the (closed) socket. - Peter
Re: Limit user sessions in tomcat
Thanks Peter for the clarification. My background is that of a UNIX administrator not a web administrator and its showing from my posts.. My problem is long running requests. If the requests take longer than their fancy, the users just close the browser window, open a new one and resubmit the same request. I also noticed a lot of the following in my mod_jk logs which to me means the user killed the browser? ... [Tue Dec 15 02:56:30.491 2009] [3460:93] [info] jk_ajp_common.c (1688): Writing to client aborted or client network problems [Tue Dec 15 02:56:30.492 2009] [3460:93] [info] jk_ajp_common.c (2315): (31) sending request to tomcat failed (unrecoverable), becau se of client write error (attempt=1) [Tue Dec 15 02:56:30.778 2009] [3460:93] [info] jk_lb_worker.c (1339): service failed, worker 31 is in error state [Tue Dec 15 02:56:30.778 2009] [3460:93] [info] jk_lb_worker.c (1360): unrecoverable error 200, request failed. Client failed in the middle of request, we can't recover to another instance. [Tue Dec 15 02:56:30.778 2009] [3460:93] [info] mod_jk.c (2421): Aborting connection for worker=loadbalancer From: Peter Crowther peter.crowt...@melandra.com To: Tomcat Users List users@tomcat.apache.org Sent: Tue, December 15, 2009 12:42:01 PM Subject: Re: Limit user sessions in tomcat 2009/12/15 Chetan Chheda chetan_chh...@yahoo.com Is there a 3rd party tool available to manage tomcat sessions and kill them once they go rogue? Can I just check two pieces of terminology? In Tomcat (and many other web servers), a session is the notion that a user will make multiple requests to the server from the same browser over time. It's also used to refer to any data that the application keeps between these requests from the same user. I'm not aware of a way that a session could go rogue except by getting more and more data stored in it - in which case, it's time to file a bug report with your external development house and tell them to fix the leaky code! A single request to the server is one HTTP request from the browser* to the server, and the response from server to browser. This can cause problems if it takes a long time to service. Each request uses one thread while it is being serviced, then that thread is returned to the pool once the request completes. I'm not aware of any tool that will allow you to kill threads in the middle of a request if they go rogue. Are you having problems with rogue (large) sessions, or with rogue (long-running) requests? - Peter * Yes, to the pedants out there, I know this should more generally be called a user-agent or even a client, as so many HTTP requests now come from web services.
Tomcat load balancing
Hello folks, I have 2 resource hungry processes that need to run on 2 separate tomcats. I have setup mod_jk to load balance between the 2 tomcats, but sometimes they end up on the same one. Is there a way to guarantee they run on separate tomcats? Is there an option to do load based load balancing? THanks, Chetan
Re: What to upgrade?
Thanks for all your replies. The plan I proposed is a step-by-step approach to upgrade to JDK 1.5, tomcat 5.5 , JDK 1.6 with application testing after each step. So I changed the tomcat 5.0.25 startup script to use JDK 1.5.0.03 and I am running into the following error Cannot exec large heap executable: No such file or directory Could not create the Java virtual machine. My heap sizes are -Xmx2400m -Xms2400m -Xmn800m -XX:PermSize=96m -XX:MaxPermSize=128m. I reduced it to -Xmx1024m -Xms1024m -Xmn400m and was able to startup. Is there a heap size limitation? Chetan From: David Kerber dcker...@verizon.net To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, June 10, 2009 8:58:37 PM Subject: Re: What to upgrade? Chetan Chheda wrote: We are going to regression test it in non - prod environments. I agree with going to the latest version, but our application is only supported on 1.4.2 So I am thinking 1.5 would be a smaller jump than 1.6. I doubt it. If it will run on 1.5 (which it most likely will), it will run on 1.6. The jump from 1.4 to 1.5 is much bigger than the one from 1.5 to 1.6. I moved my app from TC 5.5.15 running under jvm 1.5.0_12 directly to TC 6.0.17 or .18 (I don't recall which) under jvm 1.6.latest, and it ran absolutely unchanged except that it was much faster (could handle more concurrent requests). Exactly zero code changes required in over 300 classes. The initial jump from 1.4.something to 1.5 was quite painless as well, but I wanted some of the new code features in 1.5, so I never did extensive testing under of the 1.4 code until I had my 1.5 modifications done. D - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: What to upgrade?
32 bit application running on a 64bit HP-UX Itanium platform. Found a document to get around the heap size limitation. Solution was to use another java executable(java_q4p) for the expanded heap. From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, June 12, 2009 11:11:44 AM Subject: RE: What to upgrade? From: Chetan Chheda [mailto:chetan_chh...@yahoo.com] Subject: Re: What to upgrade? I reduced it to -Xmx1024m -Xms1024m -Xmn400m and was able to startup. Is there a heap size limitation? What platform are you running on? Is this a 32- or 64-bit JVM? In a 32-bit environment, somewhere around 1500m is often the most you can get for the heap; the exact value depends on the particular OS and JVM levels (and phase of the moon, it seems like). - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat maxThreads Issue
What does your Apache access_log say at the time when tomcat threads are exhausted ? I had a very similar issue in my environment. The culprit turned out to be JkMount options. Apache was sending all requests, whether it is for static or dynamic content to tomcat. At the same time my tomcats locked up, I was seeing a lot of requests for static content(gifs in my case) in the apache access_log Setting up mod_jk to only send *.jsp to tomcats fixed the issue for me. From: Jones, Keven keven.jo...@ncr.com To: users@tomcat.apache.org users@tomcat.apache.org Sent: Wednesday, June 10, 2009 11:11:11 AM Subject: Tomcat maxThreads Issue Hello, I have Tomcat (can't figure out what version as the version.sh script is giving all kinds of errors) running on separate Linux servers. So an overview of my config is 1 web server, 4 app servers (tomcat), and 1 db server. We were seeing an increase in the httpd processes on the web server to the point where I had to increase the MaxClients on the apache 2 server to 2048. I finally figured out that the normal number of httpd daemons on the web server (50-100) was spiking due to tomcat processes stalling or going completely down. So I started looking at why tomcat was failing and I found this error message in the ACStarError.log : SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200) or check the servlet status. So I checked the server.xml file and found the following: Connector port=81 protocol=HTTP/1.1 server=- maxThreads=600 connectionTimeout=2 redirectPort=443 / !-- Awele 20070402 For SSL Requests, processed non-SSL, due to Coyote SSL Accelerator -- !-- Connector port=443 address=10.5.3.9 protocol=HTTP/1.1 server=- secure=true maxThreads=600 connectionTimeout=2 / -- !-- Define a SSL HTTP/1.1 Connector on port 8443 This connector uses the JSSE configuration, when using APR, the connector should be using the OpenSSL style configuration described in the APR documentation -- !-- Connector port=8443 protocol=HTTP/1.1 SSLEnabled=true maxThreads=150 scheme=https secure=true clientAuth=false sslProtocol=TLS / -- So It looks like the maxThreads has already been increased to 600. The server has been rebooted and tomcat stopped/started multiple times. Again all servers are Red Hat 5.2. Is there an additional parameter that needs to be set in order for tomcat to actually increase the maxThreads ? Thank You Keven Jones
What to upgrade?
Our management has directed us to upgrade our JRE to 1.5 from 1.4.2 because they have heard that 1.5 has better memory management features. I am doing my due diligence to present a plan on what it will take to get us there. The question is how do I approach this ... We have apache(2.0.59), Tomcat(5.0.30), and a few JVM based application processes. Tomcat and the application processes use the same JRE(1.4.2). - Is tomcat 5.0.30 supported with JRE 1.5? Is it as simple as updating the startup scripts with the new JRE path? - Should I be upgrading tomcat too since our version is not even supported(as per ? - What is the impact to the code? Is it all backward compatible and will work with the new JRE/tomcat version? - Where can I find a comparison of enhancements/improvements between various tomcat versions? I saw the change log but its just too vast for me to digest... - What is the impact to Apache/Mod_jk configurations? Chetan
Re: What to upgrade?
We are going to regression test it in non - prod environments. I agree with going to the latest version, but our application is only supported on 1.4.2 So I am thinking 1.5 would be a smaller jump than 1.6. Chetan From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, June 10, 2009 4:53:27 PM Subject: Re: What to upgrade? -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, On 6/10/2009 4:33 PM, André Warnier wrote: Save your old tomcat/conf files somewhere for future reference, then install the new JVM and the new Tomcat, with the standard new conf files for the new Tomcat. Make sure it runs with some example webapp. Then /manually/ introduce the needed changes to the new configuration, one by one, on the base of your old conf files, and check after each change. +1 Also, note that Tomcat 5.5 and higher really /really/ */really/* want you to put your Context elements into yourapp/META-INF/context.xml instead of putting them into Tomcat's server.xml. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkowHUcACgkQ9CaO5/Lv0PDyEgCeITM9GNPk8rQHNFtfRsxx8vwS s3cAn3fBx+bijQIYPko/I3/UHam/EY6r =p6DD -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Application monitoring
I forgot to add we are using JDK1.4 From: Leon Rosenberg rosenberg.l...@googlemail.com To: Tomcat Users List users@tomcat.apache.org Sent: Saturday, June 6, 2009 7:52:16 AM Subject: Re: Application monitoring On Sat, Jun 6, 2009 at 4:19 AM, Chetan Chhedachetan_chh...@yahoo.com wrote: All, What are you using for production monitoring and alerting for JVM apache based applications? I am referring to proactively monitoring and alerting of apache utilization(workers, requests etc), tomcat utilization(threads, heap, etc) , tracking Java methods, JSPs , servlets and also transaction tracking. Of course moskito , here's the demo monitoring app: http://moskito.anotheria.net/moskitodemo/mui/mskShowAllProducers here's the homepage https://moskito.dev.java.net http://moskito.anotheria.net - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Application monitoring
All, What are you using for production monitoring and alerting for JVM apache based applications? I am referring to proactively monitoring and alerting of apache utilization(workers, requests etc), tomcat utilization(threads, heap, etc) , tracking Java methods, JSPs , servlets and also transaction tracking. We are currently evaluating a number of tools like CA( Wily Introscope CEM) , Opnet (Panorama ACE Live) HP (Diagnostics, SiteScope and BAC). Do you have any opinions/experiences about these tools? Is there any other tool out there that you are using which is worth investigating? If you are using these tools, are you running them continuously in production or only during project implementation? THanks, Chetan
Re: User tracking/monitoring
Thanks Tim and Chuck. Tim, Is this something a novice programmer like me can add to our Footer.jsp file? Can you add your recommendation to a code snippet , I would really appreciate it. Chuck, Can you tell me more about 2? How would the sessionId be displayed in the page with that method? Can you demonstrate that to me with a small example? 3 wont work as most of the users are business users and designers who are not that savvy. Thank you so much! Chetan From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, May 29, 2009 1:59:01 PM Subject: RE: User tracking/monitoring From: Chetan Chheda [mailto:chetan_chh...@yahoo.com] Subject: User tracking/monitoring Is there anyway we can add the tomcat sessionID to their page so that they can tell us where they are? Besides adding it to some common component of each page of the web site as Tim suggested, you could do any of the following: 1) Disable cookies and use URL rewriting, forcing the session ID into the URL (can't really recommend that). 2) Add a filter or valve to append the session ID to each response. 3) Have the users look in the cookie cache of the browser when they have a problem. Tim's suggestion or perhaps #3 would seem to be the simplest. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: User tracking/monitoring
I had to add
%= request.getSession().getId()%
for it to display the sessionId.
Tim,
Can you elaborate on what the side effects would be to display the full
sessionId as compared to just the session route?
Chetan
From: Tim Funk funk...@apache.org
To: Tomcat Users List users@tomcat.apache.org
Sent: Monday, June 1, 2009 11:54:26 AM
Subject: Re: User tracking/monitoring
The snippet I post was the snippet. (if you are using jsp) But it can have
security side effects. A safer snippet might be this:
%@ taglib uri=http://java.sun.com/jsp/jstl/functions; prefix=fn %
${(fn:split(pageContext.session.id,
'.')[fn:length(fn:split(pageContext.session.id, '.'))-1])}
Which splits the session id apart by '.' and typically - the last part of the
session is the jvmroute (aka the server which served the request)
Anyone with knowledge of JSTL should have no problem incorporating the above
snippet. If you have trouble with the above - seek a trusted consultant for
help. The money will be well spent.
-Tim
Chetan Chheda wrote:
Thanks Tim and Chuck.
Tim,
Is this something a novice programmer like me can add to our Footer.jsp
file? Can you add your recommendation to a code snippet , I would really
appreciate it.
Chuck,
Can you tell me more about 2? How would the sessionId be displayed in the
page with that method? Can you demonstrate that to me with a small example?
3 wont work as most of the users are business users and designers who are not
that savvy.
Thank you so much! Chetan
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
User tracking/monitoring
All, In our production environment we load balance across 4 tomcats that are split among 2 physical servers. This is a high traffic website and we get calls from users for a number of support issues and for most of these issues the first step is to find out what tomcat instance they ended up in. Is there anyway we can add the tomcat sessionID to their page so that they can tell us where they are? Thanks, Chetan
Apache-Tomcat Advanced training courses in the US
All, Can you recommend some of them? Topics that I am looking for are : Mod_jk configuration Tomcat connectors Apache webserver For high traffic scenariors with a performance tuning monitoring perspective .. Thanks, Chetan
Re: Tomcat not closing threads -- SOLVED ??
Pardon my ignorance .. but browsing thru the application directories I found a number of other file types like css, javascript, ser etc and wasnt sure whether these need to go to tomcat or not. Hence - JkMount /LCSW/* ajp13 So I went with with I am sure about, which is image files and html pages. Nobody in my shop knows this side of the application and vendor support is terrible. From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, May 27, 2009 10:31:20 AM Subject: Re: Tomcat not closing threads -- SOLVED ?? -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chetan, On 5/26/2009 2:47 PM, Chetan Chheda wrote: After some digging thru config files setup by the vendor, I think I might have found the root cause ..Correlating the access_log and tomcat logs, I found out that tomcat threads were shooting up whenever a large number of GIF files were being requested. These are the modjk settings in our application config files that are included in httpd.conf JkMount /LCSW/* ajp13 JkMount /LCSW/*.jsp ajp13 JkMount /LCSW/*.jsp/* ajp13 As André points out, the second and third line are superfluous, given the first line. So when an image is accessed via the following URL, http://blahbhal/LCSW/images/header.gif , the above routes were sending it to tomcat. I verified this by shutting down tomcat and accessing the above URL, and was unsuccessful. Now I plan to add the following excludes JkUnMount /*.htm ajp13 JkUnMount /*.html ajp13 JkUnMount /*.png ajp13 JkUnMount /*.gif ajp13 JkUnMount /*.jpg ajp13 Instead of including everything, then excluding certain things, why not just include /only/ the things you /actually/ want Tomcat to handle. Something like: JkMount /LCSW/*.jsp ajp13 I tested the above in a test environment and was able to access a gif file with tomcat down. Try the above. I think you'll like it better, since you won't have to add exceptions for every type of static file you want to serve. my question is, how does the ajp13 connector interpret a gif file/static content request as opposed to a jsp request? mod_jk does not do any interpretation at all. The JkMount directives just set up a table of URL pattern matchers. No content is analyzed or anything like that. In combination with another module, you might be able to sniff some of the /request/ and then conditionally invoke Tomcat (by using André's favorite SetHandler jakarta-servlet trick) but that would be considered super-human efforts to do something that ought to be simple. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkodTrgACgkQ9CaO5/Lv0PAZLACgwBc8vKC66xVRYrd+HJoYtmcf sKwAn0bx7DlKzROMeIV0zhjQ/VSXB0zc =Sxs5 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat not closing threads -- SOLVED ??
This is a vendor supported architecture. They deliver a set of pre-packaged Apache conf files. So apache is a part of their design. On our side, we have multiple httpd's load balancing to multiple tomcats. Also the application is AD integrated and some paths of access lie in the DMZ where only a web tier can reside. Chetan From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, May 27, 2009 11:38:11 AM Subject: RE: Tomcat not closing threads -- SOLVED ?? From: Chetan Chheda [mailto:chetan_chh...@yahoo.com] Subject: Re: Tomcat not closing threads -- SOLVED ?? Pardon my ignorance .. but browsing thru the application directories I found a number of other file types like css, javascript, ser etc and wasnt sure whether these need to go to tomcat or not. Hence - JkMount /LCSW/* ajp13 Let's go back to another of André's questions: why are you using httpd at all? Can you simplify your life by simply removing it and letting Tomcat handle everything directly? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat not closing threads -- SOLVED ??
After some digging thru config files setup by the vendor, I think I might have found the root cause ..Correlating the access_log and tomcat logs, I found out that tomcat threads were shooting up whenever a large number of GIF files were being requested. These are the modjk settings in our application config files that are included in httpd.conf JkMount /LCSW/* ajp13 JkMount /LCSW/*.jsp ajp13 JkMount /LCSW/*.jsp/* ajp13 So when an image is accessed via the following URL, http://blahbhal/LCSW/images/header.gif , the above routes were sending it to tomcat. I verified this by shutting down tomcat and accessing the above URL, and was unsuccessful. Now I plan to add the following excludes JkUnMount /*.htm ajp13 JkUnMount /*.html ajp13 JkUnMount /*.png ajp13 JkUnMount /*.gif ajp13 JkUnMount /*.jpg ajp13 I tested the above in a test environment and was able to access a gif file with tomcat down. my question is, how does the ajp13 connector interpret a gif file/static content request as opposed to a jsp request? Chetan From: Chetan Chheda chetan_chh...@yahoo.com To: Tomcat Users List users@tomcat.apache.org Sent: Monday, May 18, 2009 5:53:07 PM Subject: Re: Tomcat not closing threads Hi martin, Thanks for your reply but you have left me confused Since Apache is setup as below Maxclients = 512 ThreadsPerChild = 256 Wont it restrict the processes to 2 processes with 256 threads each instead of the ServerLimit(16) ? Looking at the OS , I see 2 httpd processes each with 258 threads Thanks, Chetan From: Martin Gainty mgai...@hotmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Monday, May 18, 2009 4:00:16 PM Subject: RE: Tomcat not closing threads good to know With worker mpm the connection_pool_size defaults to ThreadsPerChild ThreadsPerChild that in the final gives the MaxClients connections to the Tomcat. So the default value for connection_pool_size is always ThreadsPerChild For threaded and hybrid servers (e.g. beos or worker) MaxClients restricts the total number of threads that will be available to serve clients. The default value for beos is 50. For hybrid MPMs the default value is 16 (ServerLimit) multiplied by the value of 25 (ThreadsPerChild). Therefore, to increase MaxClients to a value that requires more than 16 processes, you must also raise ServerLimit. /***in your case***/ (MaxClients)MaxClients=16(ServerLimit)*256(ThreadsPerChild)=4096 onto thread calculation: Description:Minimum number of idle threads available to handle request spikes Syntax:MinSpareThreads number Default:See usage for details Context:server config Status:MPM Module:beos, mpm_netware, mpmt_os2, worker Minimum number of idle threads to handle request spikes. Different MPMs deal with this directive differently. worker uses a default of MinSpareThreads 75 and deal with idle threads on a server-wide basis. If there aren't enough idle threads in the server then child processes are created until the number of idle threads is greater than number. mpm_netware uses a default of MinSpareThreads 10 and, since it is a single-process MPM, tracks this on a server-wide bases. beos and mpmt_os2 work similar to mpm_netware. The default for beos is MinSpareThreads 1. For mpmt_os2 the default value is 5. /*lets assume you havent touched this and MinSpareThreads stays at 75***/ MaxSpareThreads(this is the parameter that is plaguing your environment) The range of the MaxSpareThreads value is restricted. Apache will correct the given value automatically according to the following rules: perchild requires MaxSpareThreads to be less or equal than ThreadLimit.mpm_netware wants the value to be greater than MinSpareThreads.For leader, threadpool and worker the value must be greater or equal than the sum of MinSpareThreads and ThreadsPerChild. MaxSpareThreads = MinSpareThreads+ThreadsPerChild MaxSpareThreads =75+256 MaxSpareThreads must be = 331 after all that if you still dont have enough SpareThreads i would up (master thread limit) ThreadLimit to the max The default value for ThreadLimit is 1920 when used with mpm_winnt and 64 when used with the others. There is a hard limit of ThreadLimit 2 (or ThreadLimit 15000 with mpm_winnt) compiled into the server. This is intended to avoid nasty effects caused by typos. /conceivable to start with 1920 and increase by increments of 500 until number of threads sufficient ***/ BTW: the ceiling on ThreadLimit=15000 In your case ThreadLimit=1920 HTH Martin Gainty __ Jogi és Bizalmassági kinyilatkoztatás/Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Ez az üzenet bizalmas. Ha nem ön az akinek szánva
Re: Tomcat not closing threads -- SOLVED ??
Great thanks! I will test out the alternate method ... What I meant when I asked about gif files and tomcat was basically is it an expensive transaction for tomcat ajp13 vs apache vs tomcat's http connector? I guess in other words I am trying to assess the impact my proposed changes in our environment. From: André Warnier a...@ice-sa.com To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, May 26, 2009 3:20:06 PM Subject: Re: Tomcat not closing threads -- SOLVED ?? André Warnier wrote: Chetan Chheda wrote: After some digging thru config files setup by the vendor, I think I might have found the root cause ..Correlating the access_log and tomcat logs, I found out that tomcat threads were shooting up whenever a large number of GIF files were being requested. These are the modjk settings in our application config files that are included in httpd.conf JkMount /LCSW/* ajp13 JkMount /LCSW/*.jsp ajp13 JkMount /LCSW/*.jsp/* ajp13 The above seems totally redundant to me. The first JkMount will capture anything starting with /LCSW/, no ? If yes, then the following two are totally redundant. So when an image is accessed via the following URL, http://blahbhal/LCSW/images/header.gif , the above routes were sending it to tomcat. I verified this by shutting down tomcat and accessing the above URL, and was unsuccessful. Now I plan to add the following excludes JkUnMount /*.htm ajp13 JkUnMount /*.html ajp13 JkUnMount /*.png ajp13 JkUnMount /*.gif ajp13 JkUnMount /*.jpg ajp13 That will work. I tested the above in a test environment and was able to access a gif file with tomcat down. my question is, how does the ajp13 connector interpret a gif file/static content request as opposed to a jsp request? As far as I know, it does not see the difference, nor care about it. It only takes into account the match/unmatch of the request URL. It has no idea of what static or dynamic content is. Separately : For the kind of thing you are doing above, there exists an alternative syntax to the JkMount/JkUnMount pair, which I personally prefer because I find that it better fits in the Apache configuration logic. Here is an example : Location /LCSW SetHandler jakarta-servlet SetEnvIf REQUEST_URI \.(html?|png|gif|jpg)$ no-jk ... /Location (You can of course easily insert other Apache statements inside the same Location block, for example authentication etc.. You find this toward the end of this documentation page : http://tomcat.apache.org/connectors-doc/reference/apache.html The SetHandler jakarta-servlet does essentially the same as the JkMount, for everything that fits under /LCSW. The SetEnvIf then selectively sets the Apache environment variable no-jk (here, whenever the request URI ends in one of the listed extensions). The mod_jk module is always called (because it is the content handler for this section), but it declines to process this URL because the no-jk variable is set. Apache thus processes the request with its own default handler, which serves the content. To see exactly what happens, step by step, set the JkLogLevel to debug, make a request, and look at the jk logfile. Erratum : replace Location /LCSW by Location /LCSW/ in the above. Otherwise, it would also match /LCSW1/ for instance. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
Chris, As with Vishwajit, my tomcat ends up with all threads busy and not serving any new requests. After setting the connectionTimeout the threads are being recycled but apache is not liking .. as per the messsage in mod_jk.log Chetan From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Friday, May 22, 2009 8:37:52 AM Subject: Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chetan, On 5/21/2009 2:08 PM, Chetan Chheda wrote: I am following this thread with great interest. I have a similar issue as Vishwajit and have resorted to adding the connectionTimeout to get rid of a large number of RUNNABLE threads. Why? Are you just offended by the number of threads, or do you have a legitimate resource problem? But mod_jk does not like tomcat threads timing out and logs the message increase the backend idle connection timeout or the connection_pool_minsize in the mod_jk logs which leads me to believe that its apache thats not letting go of the threads in my case. Again, you need to set the Connector's connectionTimeout /and/ your workers' connection_pool_timeout settings to the same time interval (note that they use different semantics... one is in seconds and the other is in ms, so read the documentation carefully). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkoWnKAACgkQ9CaO5/Lv0PBcTwCbBhuJ8/nwYLq/LAxCSVDer35t jAIAn2oUL3on6ki/x9pZHn8n0tLuVS8H =y10X -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity
I am following this thread with great interest. I have a similar issue as Vishwajit and have resorted to adding the connectionTimeout to get rid of a large number of RUNNABLE threads. But mod_jk does not like tomcat threads timing out and logs the message increase the backend idle connection timeout or the connection_pool_minsize in the mod_jk logs which leads me to believe that its apache thats not letting go of the threads in my case. From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Thursday, May 21, 2009 1:05:21 PM Subject: Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Vishwajit, On 5/20/2009 3:01 PM, Pantvaidya, Vishwajit wrote: [Pantvaidya, Vishwajit] Ok so RUNNABLE i.e. persistent threads should not be an issue. The only reason why I thought that was an issue was that I was observing that the none of the RUNNABLE connections were not being used to serve new requests, only the WAITING ones were - and I do know for sure that the RUNNABLE threads were not servicing any existing requests as I was the only one using the system then. It seems pretty clear that this is what your problem is. See if you can follow the order of events described below: 1. Tomcat and Apache httpd are started. httpd makes one or more (persistent) AJP connections to Tomcat and holds them open (duh). Each connection from httpd-Tomcat puts a Java thread in the RUNNABLE state (though actually blocked on socket read, it's not really runnable) 2. Some requests are received by httpd and sent over the AJP connections to Tomcat (or not ... it really doesn't matter) 3. Time passes, your recycle_timeout (300s) or cache_timeout (600s) expires 4. A new request comes in to httpd destined for Tomcat. mod_jk dutifully follows your instructions for closing the connections expired in #3 above (note that Tomcat has no idea that the connection has been closed, and so those threads remain in the RUNNABLE state, not connected to anything, lost forever) 5. A new connection (or multiple new connections... not sure exactly how mod_jks connection expiration-and-reconnect logic is done) is made to Tomcat which allocates a new thread (or threads) which is/are in the RUNNABLE state Rinse, repeat, your server chokes to death when it runs out of threads. The above description accounts for your loss of 4 threads at a time: your web browser requests the initial page followed by 3 other assets (image, css, whatever). Each one of them hits step #4 above, causing a new AJP connection to be created, with the old one still hanging around on the Tomcat side just wasting a thread and memory. By setting connectionTimeout on the AJP Connector, you are /doing what you should have done in the first place, which is match mod_jk cache_timeout with Connector connectionTimeout/. This allows the threads on the Tomcat side to expire just like those on the httpd side. They should expire at (virtually) the same time and everything works as expected. This problem is compounded by your initial configuration which created 10 connections from httpd-Tomcat for every (prefork) httpd process, resulting in 9 useless AJP connections for every httpd process. I suspect that you were expiring 10 connections at a time instead of just one, meaning that you were running out of threads 10 times faster than you otherwise would. Suggestions: 1. Tell your ops guys we know what we're talking about 2. Upgrade mod_jk 3. Set connection_pool_size=1, or, better yet, remove the config altogether and let mod_jk determine its own value 4. Remove all timeouts unless you know that you have a misbehaving firewall. If you do, enable cping/cpong (the preferred strategy by at least one author or mod_jk) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEUEARECAAYFAkoVidEACgkQ9CaO5/Lv0PCzwACYsrhskrNVgJFk6hI1gU+Kkmbe WQCfTNbSLTgNtHcTbTAu5kw5igicNMw= =0EWv -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat not closing threads
threads Chetan, We had similar issue with Sun solaris and weblogic. My system admin found out there is a socket read errors are going on . He used snoop command in solaris. After tuning the TCP parameters the issue got resolved. You might also need to something like that. Thanks Krish On Wed, May 13, 2009 at 11:57 AM, Chetan Chheda chetan_chh...@yahoo.comwrote: I guess I need to ask my question again. Im primarily an HP-UX administrator and recently inherited this web based application ... We are having some issues in accomodating additional user load/functionality. While a part of the team is looking at optimizing code, I am responsible for infrastructure componants. One particular focus area for me is the mod_jk and tomcat configurations. 1. Last of April saw one of the tomcat's stop processing requests. Looking at the mod_jk.log I saw the following errors Unable to get the free endpoint for worker XXX from 37 slots I have setup the status servlet for mod_jk in which I saw that max slots were opened. Thinking that mod_jk needs more number of connections, I basically set it to ThreadsPerChild from worker MPM 2. Now we are randomly seeing that even a simple activity as user login causes a spike in used tomcat connector threads. Eventually all tomcat threads are used and no new requests can be serviced. So, long story short, is my apache, mod_jk and tomcat configuration in sync? Maxclients = 512 ThreadsPerChild = 256 So that means 2 server processes with 256 threads each. I have 2 worker threads in my worker.properties file with connection_pool_size=256. Does this mean that total number of connections into tomcat = 256 * 2(number of workers) * 2(number of server procs) = 1024 ?? Do I need to change the connection_pool_size to 120 . Which means 120*2 *2 = 480 and that leaves the remainder of apache threads for static content? 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in server.xml . Is this why connections remain open? Thanks, Chetan From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, May 12, 2009 10:47:00 PM Subject: RE: Tomcat not closing threads From: Chetan Chheda [mailto:chetan_chh...@yahoo.com] Subject: Re: Tomcat not closing threads I managed to get a thread dump during one such tomcat hangs. Most of the threads are in the following status ... The ones you show are simply waiting for input from httpd; that's a pretty normal state. When looking at a thread dump, it's often the oddball thread that's not doing what the rest are that is causing a problem. However, if all of the threads are just waiting to receive something from httpd, then you may have to look outside of Tomcat for the cause. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org _ Windows Live™: Keep your life in sync. http://windowslive.com/explore?ocid=TXT_TAGLM_BR_life_in_synch_052009
Re: Tomcat not closing threads
HTTPd.conf : Timeout 300 KeepAlive On MaxKeepAliveRequests 100 KeepAliveTimeout 30 IfModule worker.c ThreadLimit 512 StartServers 100 MaxClients 512 MinSpareThreads 129 MaxSpareThreads 256 ThreadsPerChild 256 MaxRequestsPerChild 0 /IfModule Workers.properties: njipuplmapp01:/opt/hpws/apache/conf#cat workers.properties ps=/ workers.tomcat_home=/coach/app/ptc/tomcat1 worker.list=ajp13App11,ajp13App12,loadbalancer,mystatus worker.ajp13App11.port=8010 worker.ajp13App11.host=localhost worker.ajp13App11.type=ajp13 worker.ajp13App11.lbfactor=1 worker.ajp13App11.cache_timeout=900 worker.ajp13App11.socket_timeout=0 worker.ajp13App11.retries=3 worker.ajp13App11.recycle_timeout=900 worker.ajp13App12.port=8011 worker.ajp13App12.host=localhost worker.ajp13App12.type=ajp13 worker.ajp13App12.lbfactor=1 worker.ajp13App12.cache_timeout=900 worker.ajp13App12.socket_timeout=0 worker.ajp13App12.retries=3 worker.ajp13App12.recycle_timeout=900 worker.loadbalancer.type=lb worker.loadbalancer.balanced_workers=ajp13App11,ajp13App12 worker.loadbalancer.method=Request worker.loadbalancer.sticky_session=true worker.inprocess.class_path=$(workers.tomcat_home)$(ps)lib$(ps)tomcat.jar worker.inprocess.cmd_line=start worker.inprocess.stdout=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stdout worker.inprocess.stderr=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stderr worker.mystatus.type=status worker.mystatus.mount=/admin/status/jk mod_jk # Configure mod_jk for basic operation IfModule mod_jk.c JkWorkersFile conf/workers.properties JkLogFile logs/mod_jk.log JkLogLevel info JkShmFile logs/JkShmFile.log # Must use this for proper % escaping (cannot use with Tomcat 3.2.x) JkOptions +ForwardURIEscaped # Uncomment the line below once HP builds with mod_jk 1.2.13 or # higher to ensure that the 'X-Do-not-compress-this' header # behaves as expected. #JkOptions +FlushPackets # Mount all JSP pages with our default worker JkMount /*.jsp loadbalancer JkMount /*.jsp/* loadbalancer /IfModule Connector : Connector acceptCount=100 connectionTimeout=2 disableUploadTimeout=true port=8082 redirectPort=8445 maxSpareThreads=75 maxThreads=150 minSpareThreads=25 /Connector Connector URIEncoding=UTF-8 acceptCount=100 debug=1 port=8012 protocol=AJP/1.3 protocolHandlerClassName=org.apache.jk.server.JkCoyoteHandler redirectPort=8445 tomcatAuthentication=false useBodyEncodingForURI=true /Connector From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Monday, May 18, 2009 12:18:22 PM Subject: Re: Tomcat not closing threads -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chetan, On 5/18/2009 10:43 AM, Chetan Chheda wrote: Our network team is already engaged and looking into this. We had several instances of tomcats locking up today. I have attached one such thread dump here .. It looks like all your request processors are waiting for data to be read from the client. Can you post your current: workers.properties httpd.conf (portions relevant to the worker MPM and mod_jk) Connector - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkoRik4ACgkQ9CaO5/Lv0PDktACfZRY7Eu5R/0nvnyGdOiXD2tID 9RQAoKCgluYtV8ht5/DAvWc1nP4b+yeQ =UycA -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat not closing threads
. (and in the case of ThreadsPerChild parameter MaxClients*64*connection_pool_size) Apache Worker configuration With worker mpm the connection_pool_size defaults to ThreadsPerChild, that in the final gives the MaxClients connections to the Tomcat. So the default value for connection_pool_size is always ThreadsPerChild that with prefork mpm is by design always 1. (in the case of ThreadsPerChild parameter not being null MaxClients*ThreadsPerChild) HTH Martin __ Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni. Date: Mon, 18 May 2009 07:43:04 -0700 From: chetan_chh...@yahoo.com Subject: Re: Tomcat not closing threads To: users@tomcat.apache.org Our network team is already engaged and looking into this. We had several instances of tomcats locking up today. I have attached one such thread dump here .. Can some of you pls take a look and point me into the right direction? From: rad muthu rmuth...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, May 13, 2009 8:05:08 PM Subject: Re: Tomcat not closing threads Chetan, We had similar issue with Sun solaris and weblogic. My system admin found out there is a socket read errors are going on . He used snoop command in solaris. After tuning the TCP parameters the issue got resolved. You might also need to something like that. Thanks Krish On Wed, May 13, 2009 at 11:57 AM, Chetan Chheda chetan_chh...@yahoo.comwrote: I guess I need to ask my question again. Im primarily an HP-UX administrator and recently inherited this web based application ... We are having some issues in accomodating additional user load/functionality. While a part of the team is looking at optimizing code, I am responsible for infrastructure componants. One particular focus area for me is the mod_jk and tomcat configurations. 1. Last of April saw one of the tomcat's stop processing requests. Looking at the mod_jk.log I saw the following errors Unable to get the free endpoint for worker XXX from 37 slots I have setup the status servlet for mod_jk in which I saw that max slots were opened. Thinking that mod_jk needs more number of connections, I basically set it to ThreadsPerChild from worker MPM 2. Now we are randomly seeing that even a simple activity as user login causes a spike in used tomcat connector threads. Eventually all tomcat threads are used and no new requests can be serviced. So, long story short, is my apache, mod_jk and tomcat configuration in sync? Maxclients = 512 ThreadsPerChild = 256 So that means 2 server processes with 256 threads each. I have 2 worker threads in my worker.properties file with connection_pool_size=256. Does this mean that total number of connections into tomcat = 256 * 2(number of workers) * 2(number of server procs) = 1024 ?? Do I need to change the connection_pool_size to 120 . Which means 120*2 *2 = 480 and that leaves the remainder of apache threads for static content? 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in server.xml . Is this why connections remain open? Thanks, Chetan From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, May 12, 2009 10:47:00 PM Subject: RE: Tomcat not closing threads From: Chetan Chheda [mailto:chetan_chh...@yahoo.com] Subject: Re: Tomcat not closing threads I managed to get a thread dump during one such tomcat hangs. Most of the threads are in the following status ... The ones you show are simply waiting for input from httpd; that's a pretty normal state. When looking at a thread dump, it's often the oddball thread that's not doing what the rest are that is causing a problem. However, if all of the threads are just waiting to receive something from httpd, then you may have to look outside
Re: Tomcat not closing threads
Apache 1.3 and Apache 2.x-prefork create a separate child process for each client connection, meaning that if you have default 250 MaxClients, you can end up with MaxClients * connection_pool_size connections to the Tomcat, that would in your case be 25. (and in the case of ThreadsPerChild parameter MaxClients*64*connection_pool_size) Apache Worker configuration With worker mpm the connection_pool_size defaults to ThreadsPerChild, that in the final gives the MaxClients connections to the Tomcat. So the default value for connection_pool_size is always ThreadsPerChild that with prefork mpm is by design always 1. (in the case of ThreadsPerChild parameter not being null MaxClients*ThreadsPerChild) HTH Martin __ Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt uebernehmen. Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le destinataire prévu, nous te demandons avec bonté que pour satisfaire informez l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est interdite. Ce message sert à l'information seulement et n'aura pas n'importe quel effet légalement obligatoire. Étant donné que les email peuvent facilement être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité pour le contenu fourni. Date: Mon, 18 May 2009 07:43:04 -0700 From: chetan_chh...@yahoo.com Subject: Re: Tomcat not closing threads To: users@tomcat.apache.org Our network team is already engaged and looking into this. We had several instances of tomcats locking up today. I have attached one such thread dump here .. Can some of you pls take a look and point me into the right direction? From: rad muthu rmuth...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Wednesday, May 13, 2009 8:05:08 PM Subject: Re: Tomcat not closing threads Chetan, We had similar issue with Sun solaris and weblogic. My system admin found out there is a socket read errors are going on . He used snoop command in solaris. After tuning the TCP parameters the issue got resolved. You might also need to something like that. Thanks Krish On Wed, May 13, 2009 at 11:57 AM, Chetan Chheda chetan_chh...@yahoo.comwrote: I guess I need to ask my question again. Im primarily an HP-UX administrator and recently inherited this web based application ... We are having some issues in accomodating additional user load/functionality. While a part of the team is looking at optimizing code, I am responsible for infrastructure componants. One particular focus area for me is the mod_jk and tomcat configurations. 1. Last of April saw one of the tomcat's stop processing requests. Looking at the mod_jk.log I saw the following errors Unable to get the free endpoint for worker XXX from 37 slots I have setup the status servlet for mod_jk in which I saw that max slots were opened. Thinking that mod_jk needs more number of connections, I basically set it to ThreadsPerChild from worker MPM 2. Now we are randomly seeing that even a simple activity as user login causes a spike in used tomcat connector threads. Eventually all tomcat threads are used and no new requests can be serviced. So, long story short, is my apache, mod_jk and tomcat configuration in sync? Maxclients = 512 ThreadsPerChild = 256 So that means 2 server processes with 256 threads each. I have 2 worker threads in my worker.properties file with connection_pool_size=256. Does this mean that total number of connections into tomcat = 256 * 2(number of workers) * 2(number of server procs) = 1024 ?? Do I need to change the connection_pool_size to 120 . Which means 120*2 *2 = 480 and that leaves the remainder of apache threads for static content? 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in server.xml . Is this why connections remain open? Thanks, Chetan From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, May 12, 2009 10:47:00 PM Subject: RE: Tomcat not closing threads From: Chetan Chheda [mailto:chetan_chh...@yahoo.com] Subject: Re: Tomcat not closing threads I managed to get a thread dump during one such tomcat hangs. Most
Tomcat GC threads
All, We are using -XX:+UseParNewGC option in our Tomcat JVMs with -XX:ParallelGCThreads=16 on a 6 CPU system. What is the real impact of having a higher number of GC threads than the number of CPUs? With the 2 tomcat JVMs running on the box, would you recommend changing it to 6 for each tomcat? Thanks, Chetan
Re: Tomcat not closing threads
I guess I need to ask my question again. Im primarily an HP-UX administrator and recently inherited this web based application ... We are having some issues in accomodating additional user load/functionality. While a part of the team is looking at optimizing code, I am responsible for infrastructure componants. One particular focus area for me is the mod_jk and tomcat configurations. 1. Last of April saw one of the tomcat's stop processing requests. Looking at the mod_jk.log I saw the following errors Unable to get the free endpoint for worker XXX from 37 slots I have setup the status servlet for mod_jk in which I saw that max slots were opened. Thinking that mod_jk needs more number of connections, I basically set it to ThreadsPerChild from worker MPM 2. Now we are randomly seeing that even a simple activity as user login causes a spike in used tomcat connector threads. Eventually all tomcat threads are used and no new requests can be serviced. So, long story short, is my apache, mod_jk and tomcat configuration in sync? Maxclients = 512 ThreadsPerChild = 256 So that means 2 server processes with 256 threads each. I have 2 worker threads in my worker.properties file with connection_pool_size=256. Does this mean that total number of connections into tomcat = 256 * 2(number of workers) * 2(number of server procs) = 1024 ?? Do I need to change the connection_pool_size to 120 . Which means 120*2 *2 = 480 and that leaves the remainder of apache threads for static content? 3. I have worker.XXX.cache_timeout=900 and no connectionTimeout in server.xml . Is this why connections remain open? Thanks, Chetan From: Caldarale, Charles R chuck.caldar...@unisys.com To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, May 12, 2009 10:47:00 PM Subject: RE: Tomcat not closing threads From: Chetan Chheda [mailto:chetan_chh...@yahoo.com] Subject: Re: Tomcat not closing threads I managed to get a thread dump during one such tomcat hangs. Most of the threads are in the following status ... The ones you show are simply waiting for input from httpd; that's a pretty normal state. When looking at a thread dump, it's often the oddball thread that's not doing what the rest are that is causing a problem. However, if all of the threads are just waiting to receive something from httpd, then you may have to look outside of Tomcat for the cause. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat not closing threads
Hello, I am having an issue where I could use some clarity. We are using apache 2.0.59 and tomcat 5.5. We are having some growing pains with vendor specified modjk and tomcat settings. 1. Tomcat opens up a number of threads during some transactions but does not close them. Is this because the connectionTimeout parameter is missing in the AJP connector defination? 2. I do have the cache_timeout (deprecated , I know ) set to 900 secs. What impact does that have on closing idle threads? 3. Today one of the Tomcat's froze and thru jkstatus I saw 409 connections open to it. If my connection pool size is 256 (from modjk.log) why are more connections being opened? Thanks, Chetan My workers.properties file looks like this. ps=/ worker.list=ajp13App31,ajp13App32,loadbalancer,mystatus workers.tomcat_home=/coach/app/ptc/tomcat1 worker.ajp13App31.port=8012 worker.ajp13App31.host=localhost worker.ajp13App31.type=ajp13 worker.ajp13App31.lbfactor=1 worker.ajp13App31.cache_timeout=900 worker.ajp13App31.socket_timeout=0 worker.ajp13App31.retries=3 worker.ajp13App31.recycle_timeout=900 worker.ajp13App32.port=8013 worker.ajp13App32.host=localhost worker.ajp13App32.type=ajp13 worker.ajp13App32.lbfactor=1 worker.ajp13App32.cache_timeout=900 worker.ajp13App32.socket_timeout=0 worker.ajp13App32.retries=3 worker.ajp13App32.recycle_timeout=900 worker.loadbalancer.type=lb worker.loadbalancer.balanced_workers=ajp13App31,ajp13App32 worker.loadbalancer.method=Request worker.loadbalancer.sticky_session=true worker.inprocess.class_path=$(workers.tomcat_home)$(ps)lib$(ps)tomcat.jar worker.inprocess.cmd_line=start worker.inprocess.stdout=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stdout worker.inprocess.stderr=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stderr worker.mystatus.type=status worker.mystatus.mount=/admin/status/jk And here is the AJP13 connector from one of the tomcat's server.xml : Connector URIEncoding=UTF-8 acceptCount=100 debug=1 port=8012 protocol=AJP/1.3 protocolHandlerClassName=org.apache.jk.server.JkCoyoteHandler redirectPort=8445 tomcatAuthentication=false useBodyEncodingForURI=true Apache MPM # worker MPM # StartServers: initial number of server processes to start # MaxClients: maximum number of simultaneous client connections # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # ThreadsPerChild: constant number of worker threads in each server process # MaxRequestsPerChild: maximum number of requests a server process serves IfModule worker.c ThreadLimit 512 StartServers 100 MaxClients 512 MinSpareThreads 129 MaxSpareThreads 256 ThreadsPerChild 256 MaxRequestsPerChild 0 /IfModule
Re: Tomcat not closing threads
Just wondering if anyone got my message below? From: Chetan Chheda chetan_chh...@yahoo.com To: users@tomcat.apache.org Sent: Tuesday, May 12, 2009 12:03:49 PM Subject: Tomcat not closing threads Hello, I am having an issue where I could use some clarity. We are using apache 2.0.59 and tomcat 5.5. We are having some growing pains with vendor specified modjk and tomcat settings. 1. Tomcat opens up a number of threads during some transactions but does not close them. Is this because the connectionTimeout parameter is missing in the AJP connector defination? 2. I do have the cache_timeout (deprecated , I know ) set to 900 secs. What impact does that have on closing idle threads? 3. Today one of the Tomcat's froze and thru jkstatus I saw 409 connections open to it. If my connection pool size is 256 (from modjk.log) why are more connections being opened? Thanks, Chetan My workers.properties file looks like this. ps=/ worker.list=ajp13App31,ajp13App32,loadbalancer,mystatus workers.tomcat_home=/coach/app/ptc/tomcat1 worker.ajp13App31.port=8012 worker.ajp13App31.host=localhost worker.ajp13App31.type=ajp13 worker.ajp13App31.lbfactor=1 worker.ajp13App31.cache_timeout=900 worker.ajp13App31.socket_timeout=0 worker.ajp13App31.retries=3 worker.ajp13App31.recycle_timeout=900 worker.ajp13App32.port=8013 worker.ajp13App32.host=localhost worker.ajp13App32.type=ajp13 worker.ajp13App32.lbfactor=1 worker.ajp13App32.cache_timeout=900 worker.ajp13App32.socket_timeout=0 worker.ajp13App32.retries=3 worker.ajp13App32.recycle_timeout=900 worker.loadbalancer.type=lb worker.loadbalancer.balanced_workers=ajp13App31,ajp13App32 worker.loadbalancer.method=Request worker.loadbalancer.sticky_session=true worker.inprocess.class_path=$(workers.tomcat_home)$(ps)lib$(ps)tomcat.jar worker.inprocess.cmd_line=start worker.inprocess.stdout=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stdout worker.inprocess.stderr=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stderr worker.mystatus.type=status worker.mystatus.mount=/admin/status/jk And here is the AJP13 connector from one of the tomcat's server.xml : Connector URIEncoding=UTF-8 acceptCount=100 debug=1 port=8012 protocol=AJP/1.3 protocolHandlerClassName=org.apache.jk.server.JkCoyoteHandler redirectPort=8445 tomcatAuthentication=false useBodyEncodingForURI=true Apache MPM # worker MPM # StartServers: initial number of server processes to start # MaxClients: maximum number of simultaneous client connections # MinSpareThreads: minimum number of worker threads which are kept spare # MaxSpareThreads: maximum number of worker threads which are kept spare # ThreadsPerChild: constant number of worker threads in each server process # MaxRequestsPerChild: maximum number of requests a server process serves IfModule worker.c ThreadLimit 512 StartServers 100 MaxClients 512 MinSpareThreads 129 MaxSpareThreads 256 ThreadsPerChild 256 MaxRequestsPerChild 0 /IfModule
Re: Tomcat not closing threads
I managed to get a thread dump during one such tomcat hangs. Most of the threads are in the following status ... TP-Processor7 daemon prio=10 tid=00093ab0 nid=86 lwp_id=4866290 runnable [53c0..53c00dc0] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:134) at java.io.FilterInputStream.read(FilterInputStream.java:111) at com.wily.introscope.agent.probe.net.ManagedSocketInputStream.read(ManagedSocketInputStream.java:214) at java.io.BufferedInputStream.fill(BufferedInputStream.java:183) at java.io.BufferedInputStream.read1(BufferedInputStream.java:222) at java.io.BufferedInputStream.read(BufferedInputStream.java:277) - locked 8e0d3a50 (a java.io.BufferedInputStream) at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:598) at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:535) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:663) at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:534) TP-Processor6 daemon prio=10 tid=000938d0 nid=85 lwp_id=4866288 runnable [53cc..53cc0dc0] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:134) at java.io.FilterInputStream.read(FilterInputStream.java:111) at com.wily.introscope.agent.probe.net.ManagedSocketInputStream.read(ManagedSocketInputStream.java:214) at java.io.BufferedInputStream.fill(BufferedInputStream.java:183) at java.io.BufferedInputStream.read1(BufferedInputStream.java:222) at java.io.BufferedInputStream.read(BufferedInputStream.java:277) - locked 8e0d7ed8 (a java.io.BufferedInputStream) at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:598) at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:535) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:663) at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:534) From: Chetan Chheda chetan_chh...@yahoo.com To: Tomcat Users List users@tomcat.apache.org Sent: Tuesday, May 12, 2009 1:32:16 PM Subject: Re: Tomcat not closing threads Just wondering if anyone got my message below? From: Chetan Chheda chetan_chh...@yahoo.com To: users@tomcat.apache.org Sent: Tuesday, May 12, 2009 12:03:49 PM Subject: Tomcat not closing threads Hello, I am having an issue where I could use some clarity. We are using apache 2.0.59 and tomcat 5.5. We are having some growing pains with vendor specified modjk and tomcat settings. 1. Tomcat opens up a number of threads during some transactions but does not close them. Is this because the connectionTimeout parameter is missing in the AJP connector defination? 2. I do have the cache_timeout (deprecated , I know ) set to 900 secs. What impact does that have on closing idle threads? 3. Today one of the Tomcat's froze and thru jkstatus I saw 409 connections open to it. If my connection pool size is 256 (from modjk.log) why are more connections being opened? Thanks, Chetan My workers.properties file looks like this. ps=/ worker.list=ajp13App31,ajp13App32,loadbalancer,mystatus workers.tomcat_home=/coach/app/ptc/tomcat1 worker.ajp13App31.port=8012 worker.ajp13App31.host=localhost worker.ajp13App31.type=ajp13 worker.ajp13App31.lbfactor=1 worker.ajp13App31.cache_timeout=900 worker.ajp13App31.socket_timeout=0 worker.ajp13App31.retries=3 worker.ajp13App31.recycle_timeout=900 worker.ajp13App32.port=8013 worker.ajp13App32.host=localhost worker.ajp13App32.type=ajp13 worker.ajp13App32.lbfactor=1 worker.ajp13App32.cache_timeout=900 worker.ajp13App32.socket_timeout=0 worker.ajp13App32.retries=3 worker.ajp13App32.recycle_timeout=900 worker.loadbalancer.type=lb worker.loadbalancer.balanced_workers=ajp13App31,ajp13App32 worker.loadbalancer.method=Request worker.loadbalancer.sticky_session=true worker.inprocess.class_path=$(workers.tomcat_home)$(ps)lib$(ps)tomcat.jar worker.inprocess.cmd_line=start worker.inprocess.stdout=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stdout worker.inprocess.stderr=$(workers.tomcat_home)$(ps)logs$(ps)inprocess.stderr worker.mystatus.type=status worker.mystatus.mount=/admin/status/jk And here is the AJP13 connector from one of the tomcat's server.xml : Connector URIEncoding=UTF-8 acceptCount=100 debug=1 port=8012 protocol=AJP/1.3 protocolHandlerClassName=org.apache.jk.server.JkCoyoteHandler redirectPort=8445 tomcatAuthentication=false
