Re: Large number of active sessions
Rashmi Rubdi wrote: Google Analytics is Javascript based, it won't show bot activity. I concur. Further, if you've configured a new style xml sitemap with say hourly update instructions then bots will be crawling all over your site with much greater regularity. Analytics the javascript type logging tools are only useful for tracking human use - you'll need to monitor your server logs to get real traffic figures. Some bots also use sessions. I disagree, the bot has no capability to decide to use a session. A bot would only appear to use a session if it was HTTP/1.1 capable, and was handling cookies or encoded URLs properly. Most bots get pages asynchronously, I've observed Googlebot hitting url encoded pages with jsessionids generated days beforehand, during a previous index run. This will trigger a new session as a result, but may account for apparently older creation dates appearing the list of active/recent session. (A guess: I don't know enough about the internals of Tomcat to be sure of that.) p - Original Message From: Christer Nordvik [EMAIL PROTECTED] To: Tomcat Users List users@tomcat.apache.org; [EMAIL PROTECTED] Sent: Tuesday, February 6, 2007 3:46:12 PM Subject: Re: Large number of active sessions 1. Sure. But I have only 100 users. 2. Users: Google Analytics. Sessions: Tomcat manager 3. 5.5.17 4. Yes Guess I have to go the long way of creating my own session listener then, Had hoped there was an easy solution for this. 13000 sessions must indicate that something is very very wrong? I've never had more than 300 visitors according to Google Analytics... -Christer On 2/6/07, Pid [EMAIL PROTECTED] wrote: 1) Lower your session timeout. 2) How are you measuring/calculating statistics? 3) Which version of Tomcat? 4) Are you url encoding all of your links? Christer Nordvik wrote: Hi! I've got a website with pretty low traffic (200 visitors a day). Lately it's been very slow and when I look in Tomcat's manager it says that it has a lot of sessions. Once I had 13000(!) active sessions. The session timeout is set to 30 minutes. How can this happen? Isn't this something that Tomcat should handle itself? Can it be related to my code somehow? I have only basic JSP/Servlet stuff so nothing fancy going on. Any help would be very appreciated! -Christer - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] Bored stiff? Loosen up... Download and play hundreds of games for free on Yahoo! Games. http://games.yahoo.com/games/front - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
HTTPS connector and request processing time
Hi, For some audit requirements in my web application, I would to know for each incoming HTTP/HTTPS request the time the HTTP and the HTTPS connectors process them. The requirement is to measure the processing time of each request and to store them into a file with for each request an ID that uniquely identifies them. By consulting documentations on connectors or the Tomcat API javadoc, I didn't find any tips about how to do this. Is it possible to perform this function? How to do this without modifying the Tomcat code? Thanks, -- Miguel Moquillon [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Can servlets safely spawn threads?
Many thanks! On 29/01/07, Bill Au [EMAIL PROTECTED] wrote: If you are spawning threads from your servlet, make sure that they are cleaned up when the servlet is destroyed. Otherwise, you will have a thread leak. I have ran into this problem when the webapp is reloaded without restarting the server. Each leaked thread has a reference to its classloader, preventing that classloader from being garbage collected. Reload enough time will run the perm space out of memory. Bill On 1/29/07, Christopher Schultz [EMAIL PROTECTED] wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Danny, David Delbecq wrote: 1) Your servlet must always take care to finish all Threads it has spawned. This mean when servlet gets unloaded, you must have provided a mecanism in your servlet to stop all running threads your servlet has created. Remaining Threads not handled by container can either cause tomcat to not stop when requested or prevent garbage collecting of webapp when reloading. Consider using an existing thread pool component. It should have a graceful shutdown capability already built into it. David is right: you have to make sure to shut it down when appropriate. In general, try to avoid spawning your own thread. I agree, which is why I recommend using a thread pool. Yes, you are still technically spawning your own threads (the container will not do it for you), but the point is that you will not be spawning a thread for each request. Set up your thread pool to have a reasonable number of threads for your application. You do not want to have loads of users submit jobs that will each spawn a new thread. If you run out of threads in the thread pool, you simply tell the user to come back later. Management of this type of resource is essential to maintaining a stable and useful system. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFvgK+9CaO5/Lv0PARAoIFAJ45DgOyFV9qxS2e+Qt9uHNTtkWpywCdHbnq pltJVjDsmhZMg0143155k7M= =pbEp -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- http://dannyayers.com - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Multiple services and not restarting
I am running Tomcat 5.5.2 as a stand-alone server with 14 separate services. This is done to run 14 different sites but share 7 ssl certs. The problem I am having is when I run my script to bounce the tomcat server only the first 3 or 4 services get restarted. Does anyone happen to have any explanation for this? Here is my script ** ip=/usr/local/bin/ice tc_dir=/usr/tomcat/apache-tomcat-5.5.20/bin if [ $# -eq 1 ] then tc_oper=$1 else echo -e Enter start, stop, or bounce: \c read tc_oper fi case $tc_oper in start) $tc_dir/startup.sh ;; stop) $tc_dir/shutdown.sh ;; bounce) $tc_dir/shutdown.sh $tc_dir/startup.sh ;; *) echo Invalid operation you must enter stop start or bounce: Exiting. exit ;; esac ** All of the services are exactly the same and here is a snapshot of one from the server.xml file ** Service name=xx Connector address=xxx.xx.xxx.xx port=80 redirectPort=443/ !-- Define a SSL HTTP/1.1 Connector on port 443 -- Connector address=xxx.xx.xxx.xx port=443 maxHttpHeaderSize=8192 maxThreads=150 minSpareThreads=25 maxSpareThreads=75 enableLookups=false disableUploadTimeout=true acceptCount=100 scheme=https secure=true clientAuth=false sslProtocol=SSL keystoreFile=/usr/local/bin/ice/sslcerts/.dewalive keystorePass=tick17 algorithm=IbmX509 / Engine name=xx defaultHost=..xxx Realm className=org.apache.catalina.realm.UserDatabaseRealm resourceName=UserDatabase / !-- This Host is the Virtual Host: -- Host name=..xxx appBase=/usr/x/ Aliasxxx.xx.xxx.xx/Alias Valve className='org.apache.catalina.valves.AccessLogValve' directory='/usr/x//logs' prefix='_access.' suffix='.log' pattern='common'/ Logger className='org.apache.catalina.logger.FileLogger' directory='/usr/x//logs' prefix='_catalina.' suffix='.log' timestamp='true'/ Context path=/ debug=5 docBase=/usr/x/ privileged=true / Context path=/manager debug=5 docBase=/usr/tomcat/apache-tomcat-5.5.20/server/webapps/manager privileged=tr ue / /Host /Engine /Service **
Connection Pool does not disconnect after APP shutdown.
Hi We're using connection pooling in tomcat 5.0.28. Everything works fine. But if we stop the application in the manager console, the connections stay open to the database, and so the corresponding datasource?! Ist this to be expected? Even more surprising is the fact that if we restart the web application, Tomcat is creating a new pool and we've got even more open connections to the database. We configure the connection pool in the corresponding context xml of a webapplication. I've read the tomcat documentation which explains great the creation of a pool, but how to get rid of a connection pool? - Manually destroy the datasource in the destroy of the servlet? (Iiiks!) - Configure tomcat to close corresponding connections and connection pools of a webapp when it's brought down? Thanks for any tips hints Bernhard von Gunten - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat policy file, advice and clarification please
It is likely that some other class in your webapp is on the
stack when the permission check occurs and it causing the
check to fail. You could try .../WEB-INF/- to expand
the grant to the entire webapp and see what happens.
To diagnose this kind of exception, you can add the following
system property to your Tomcat startup:
-Djava.security.debug=access,failure
In the volumes of log output created, search for access
denied. A little beyond where you find this, look for
domain that failed. That will tell you what is missing
the required permission. Note that some permission failures
are normal and won't cause a problem.
Cheers,
Larry
-Original Message-
From: Hugues Ferland [mailto:[EMAIL PROTECTED]
Sent: Tuesday, February 06, 2007 5:30 PM
To: users@tomcat.apache.org
Subject: Tomcat policy file, advice and clarification please
Hi,
I have a Tomcat 5.5 installation on Debian Linux with Java 1.6.
I have a web application that create its own connection to an
Oracle database. The ojdbc14.jar is included in the
WEB-INF/lib directory of the web application.
Now with the default configuration with security enabled, a
SecurityException java.lang.RuntimePermission
getClassLoader is thrown.
I tried to apply a security policy to ojdbc14.jar with grant
AllPermission, but without success. This is what I added to
/etc/tomcat5.5/policy.d/50user.policy:
grant codeBase file:/the path to the web
application/WEB-INF/lib/ojdbc.jar {
permission java.security.AllPermission; }
This did not work. One particularity of the web application
is that I do not deploy it with a war file. I created it
using a context xml file in
${catalina.home}/conf/Catalina/localhost/context name.xml.
Also the docBase point outside the ${catalina.home}.
Of course, I'd like to know what I'm doing wrong... I looked
on google, tomcat-docs, and other places without any luck...
Maybe somebody could point me in the right direction? Or
better yet tell me what is wrong :)
Thanks,
Hugues
-
All new Yahoo! Mail
-
Get news delivered. Enjoy RSS feeds right on your Mail page.
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Tomcat 6.0.x Custom ELResolver
Has anyone gotten a custom ELResolver to work in Tomcat 6.0.x using the existing ELResolver stack? In other words, not entering it into the faces-config.xml, but adding it through the JspApplicationContext. -- View this message in context: http://www.nabble.com/Tomcat-6.0.x-Custom-ELResolver-tf3189253.html#a8853027 Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat policy file, advice and clarification please
Hi Larry,
Thank you very much!
That was it. I'm not sure I understand well what the context of a
checkPermission is made of... hum... Is it that it uses the context from the
web apps to checkPermission...? But then why is checkPermission for my jar in
common/endorsed succeed for getClassLoader but fail for SocketPermission? I
did not find anything for that in Tomcat's policies files.
Thanks again for your help!
Hugues
Larry Isaacs [EMAIL PROTECTED] wrote: It is likely that some other class in
your webapp is on the
stack when the permission check occurs and it causing the
check to fail. You could try .../WEB-INF/- to expand
the grant to the entire webapp and see what happens.
To diagnose this kind of exception, you can add the following
system property to your Tomcat startup:
-Djava.security.debug=access,failure
In the volumes of log output created, search for access
denied. A little beyond where you find this, look for
domain that failed. That will tell you what is missing
the required permission. Note that some permission failures
are normal and won't cause a problem.
Cheers,
Larry
-Original Message-
From: Hugues Ferland [mailto:[EMAIL PROTECTED]
Sent: Tuesday, February 06, 2007 5:30 PM
To: users@tomcat.apache.org
Subject: Tomcat policy file, advice and clarification please
Hi,
I have a Tomcat 5.5 installation on Debian Linux with Java 1.6.
I have a web application that create its own connection to an
Oracle database. The ojdbc14.jar is included in the
WEB-INF/lib directory of the web application.
Now with the default configuration with security enabled, a
SecurityException java.lang.RuntimePermission
getClassLoader is thrown.
I tried to apply a security policy to ojdbc14.jar with grant
AllPermission, but without success. This is what I added to
/etc/tomcat5.5/policy.d/50user.policy:
grant codeBase file:/
application/WEB-INF/lib/ojdbc.jar {
permission java.security.AllPermission; }
This did not work. One particularity of the web application
is that I do not deploy it with a war file. I created it
using a context xml file in
${catalina.home}/conf/Catalina/localhost/.xml.
Also the docBase point outside the ${catalina.home}.
Of course, I'd like to know what I'm doing wrong... I looked
on google, tomcat-docs, and other places without any luck...
Maybe somebody could point me in the right direction? Or
better yet tell me what is wrong :)
Thanks,
Hugues
-
All new Yahoo! Mail
-
Get news delivered. Enjoy RSS feeds right on your Mail page.
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
All new Yahoo! Mail
-
Get news delivered. Enjoy RSS feeds right on your Mail page.
List of modules installed
Hi, Where I can find list of all the installed modules for an Apache installation? Appreciate your help. Thanks Krishna
Re: [OT] List of modules installed
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Krishna, Krishna Paparaju wrote: Where I can find list of all the installed modules for an Apache installation? Appreciate your help. Note that this a mailing list for Apache Tomcat (Java application server) and not Apache httpd (web server). You can find out which modules have been /compiled/ into your httpd like this: $ apache2ctl -l ...but this doesn't show you the modules you are using. In order to find that out, you'll need to look through all of your config files for references to them. You have to look for different things depending on your version of Apache httpd. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFyjvz9CaO5/Lv0PARAiEDAJ4wr6PfNEZItZuBNBPC4Tbq9whwswCdGkDv NcVnmW1CenklVowds/N3tu0= =sycG -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat policy file, advice and clarification please
Hi Huges,
The standard Tomcat 5.5 policy file includes:
grant codeBase file:${catalina.home}/common/- {
permission java.security.AllPermission;
};
which grants permission to the common/endorsed jars.
It is possible for code to programmatically grant itself some
extra permissions, such as permission to use getClassLoader,
an not impose this permission on any of the classes involved
in the calling of this code. In the absence of this programmatic
handling of permissions, all callers must have the permission,
i.e. all classes in the stack above where the permission check
occurs must have the permission. It is likely that you have a
servlet, filter, listener, or some other class in your webapp
that is involved in calling ojdbc.jar. Since it wasn't
being granted the getClassLoader permission, it was causing
the exception once permission was granted to ojdbc.jar.
Cheers,
Larry
-Original Message-
From: Hugues Ferland [mailto:[EMAIL PROTECTED]
Sent: Wednesday, February 07, 2007 3:30 PM
To: Tomcat Users List
Subject: RE: Tomcat policy file, advice and clarification please
Hi Larry,
Thank you very much!
That was it. I'm not sure I understand well what the context
of a checkPermission is made of... hum... Is it that it
uses the context from the web apps to checkPermission...?
But then why is checkPermission for my jar in common/endorsed
succeed for getClassLoader but fail for SocketPermission? I
did not find anything for that in Tomcat's policies files.
Thanks again for your help!
Hugues
Larry Isaacs [EMAIL PROTECTED] wrote: It is likely that
some other class in your webapp is on the stack when the
permission check occurs and it causing the check to fail.
You could try .../WEB-INF/- to expand the grant to the
entire webapp and see what happens.
To diagnose this kind of exception, you can add the following
system property to your Tomcat startup:
-Djava.security.debug=access,failure
In the volumes of log output created, search for access
denied. A little beyond where you find this, look for
domain that failed. That will tell you what is missing the
required permission. Note that some permission failures are
normal and won't cause a problem.
Cheers,
Larry
-Original Message-
From: Hugues Ferland [mailto:[EMAIL PROTECTED]
Sent: Tuesday, February 06, 2007 5:30 PM
To: users@tomcat.apache.org
Subject: Tomcat policy file, advice and clarification please
Hi,
I have a Tomcat 5.5 installation on Debian Linux with Java 1.6.
I have a web application that create its own connection to
an Oracle
database. The ojdbc14.jar is included in the WEB-INF/lib
directory of
the web application.
Now with the default configuration with security enabled, a
SecurityException java.lang.RuntimePermission getClassLoader is
thrown.
I tried to apply a security policy to ojdbc14.jar with grant
AllPermission, but without success. This is what I added to
/etc/tomcat5.5/policy.d/50user.policy:
grant codeBase file:/
application/WEB-INF/lib/ojdbc.jar {
permission java.security.AllPermission; }
This did not work. One particularity of the web application
is that I
do not deploy it with a war file. I created it using a context xml
file in ${catalina.home}/conf/Catalina/localhost/.xml.
Also the docBase point outside the ${catalina.home}.
Of course, I'd like to know what I'm doing wrong... I looked on
google, tomcat-docs, and other places without any luck...
Maybe somebody could point me in the right direction? Or better yet
tell me what is wrong :)
Thanks,
Hugues
-
All new Yahoo! Mail
-
Get news delivered. Enjoy RSS feeds right on your Mail page.
-
To start a new topic, e-mail: users@tomcat.apache.org To
unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
-
All new Yahoo! Mail
-
Get news delivered. Enjoy RSS feeds right on your Mail page.
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
OutOfMemoryError (but not really out of memory?) cause tomcat processes to hang
--- Version Information: --- Red Hat Linux: 2.4.21-4.ELsmp #1 SMP i686 i686 i386 GNU/Linux Apache: 2.0.54 mod_jk: 1.2.20 tomcat: 4.1.34 java: 1.5.0_06 (build 1.5.0_06-b05) --- Problem description: --- Under heavy load, after running for some time, some tomcat instances throw OutOfMemoryErrors and then become unresponsive. Using jmap -heap just after such an error occurs reports plenty of available memory. Watching the heap before the tomcats get into this type of situation reveals no telltale growth in memory usage. We currently have Xmx set to 256M, we have tried increasing and decreasing this value and there is no change in the behavior. Questions: Is it possible that the initial OutOfMemory error occurs when a large garbage collection is taking place and an OutOfMemoryError is thrown before the memory can be reclaimed throwing the whole system into some sort of bad state? If so, how do we combat this behavior? Is one of the tomcat threads running out of memory, and killing itself, thus freeing the memory? If this is what is happening, does anyone have any advice on catching the tomcat memory usage just prior to going into a bad state? Based on other's reports of similar problems we have investigated a number of system resources and their limits (file descriptors, threads, etc) with no luck. (below are some statistics that seem to show that these other areas are not the problem). One perhaps telling piece of information is that once a tomcat has gotten into this state, we find that many connections to apache end up in the SYN_RECV state (as reported by netstat). It appears that tomcat's listener thread is still accepting connections, but something goes wrong in the handoff to the processor threads such that the connection is left in SYN_RECV. This is curious as a stack trace of tomcat's threads report many (20+) processor threads in await() waiting for the next thing to process. I have included as much relevant information as I can think of below, but am happy to provide additional information should anyone have any ideas as to what may be going wrong. We would be very thankful to hear from anyone who may have any experience of similar problems, or guidance with what to try as next steps. --- Version Information: --- Red Hat Linux: 2.4.21-4.ELsmp #1 SMP i686 i686 i386 GNU/Linux Apache: 2.0.54 mod_jk: 1.2.20 tomcat: 4.1.34 java: 1.5.0_06 (build 1.5.0_06-b05) --- System setup: --- We are running apache to serve static content, and mod_jk with balanced workers to forward requests for dynamic content to 5 tomcat instances. The following are the relevant settings: Apache settings (httpd.conf): Timeout 300 KeepAlive On MaxKeepAliveRequests 100 IfModule prefork.c StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 200 MaxRequestsPerChild 100 /IfModule mod_jk settings (worker.properties) worker.list=loadbalancer worker.tomcat1.port=11009 worker.tomcat1.host=localhost worker.tomcat1.type=ajp13 worker.tomcat1.socket_timeout=30 worker.tomcat1.reply_timeout=6 worker.tomcat1.connection_pool_size=1 worker.tomcat1.connection_pool_minsize=1 worker.tomcat1.connection_pool_timeout=300 worker.tomcat1.lbfactor=1 worker.tomcat1.recover_time=600 ### (same settings for tomcat2 - tomcat5) ### worker.loadbalancer.type=lb worker.loadbalancer.balance_workers= tomcat1, tomcat2, tomcat3, tomcat4, tomcat5 tomcat settings (server.xml) Connector className=org.apache.ajp.tomcat4.Ajp13Connector port=11009 minProcessors=5 maxProcessors=100 acceptCount=10 debug=0 enableLookups=false/ After running for some time (anywhere from 20 minutes to 12 hours depending on load), we see one instance of tomcat stop responding. The following are the errors reported in vairous logs. --- catalina.out error messages (stderr/stdout
Re: OutOfMemoryError (but not really out of memory?) cause tomcat processes to hang
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Sharon, Sharon French wrote: Is it possible that the initial OutOfMemory error occurs when a large garbage collection is taking place and an OutOfMemoryError is thrown before the memory can be reclaimed Yes. In fact, this is often when OOMs occur, since the JVM is trying hard to avoid the OOM in the first place. throwing the whole system into some sort of bad state? After an OOM is thrown, the JVM is basically completely hosed. If so, how do we combat this behavior? Fix resource leaks, allocate an appropriate amount of memory, and limit resource use to avoid OOMs in the first place. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFylVl9CaO5/Lv0PARAm/9AJ44AJe4/kUr/nLl9v+g92UbRlQ4HQCcD9YM QEiVxUfHWvr5WV/LBY9jO2o= =f9jZ -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError (but not really out of memory?) cause tomcat processes to hang
Sharon, You have quite a bit of memory in your 'Old Generation'. You need to determine what you're allocating that isn't being released. How many sessions do you have active when the problem occurs? What is your session timeout? What are you putting into your sessions (how much is held by each)? Are you using any caching technologies like ehCache or hibernate caching? not allowing caches to expire can eat up memory in the Old Generation space. We use a product called LambdaProbe to monitor our tomcat instances. It's very helpful to count sessions, see what's in them, and to watch the memory and requests in real-time. If you really *need* everything you're storing and you don't have any low hanging fruit, you may just need to run with more memory. If you could test such a configuration you may find that the memory utilization stabilizes at an acceptable level. -John On 2/7/07, Sharon French [EMAIL PROTECTED] wrote: --- Version Information: --- Red Hat Linux: 2.4.21-4.ELsmp #1 SMP i686 i686 i386 GNU/Linux Apache: 2.0.54 mod_jk: 1.2.20 tomcat: 4.1.34 java: 1.5.0_06 (build 1.5.0_06-b05) --- Problem description: --- Under heavy load, after running for some time, some tomcat instances throw OutOfMemoryErrors and then become unresponsive. Using jmap -heap just after such an error occurs reports plenty of available memory. Watching the heap before the tomcats get into this type of situation reveals no telltale growth in memory usage. We currently have Xmx set to 256M, we have tried increasing and decreasing this value and there is no change in the behavior. Questions: Is it possible that the initial OutOfMemory error occurs when a large garbage collection is taking place and an OutOfMemoryError is thrown before the memory can be reclaimed throwing the whole system into some sort of bad state? If so, how do we combat this behavior? Is one of the tomcat threads running out of memory, and killing itself, thus freeing the memory? If this is what is happening, does anyone have any advice on catching the tomcat memory usage just prior to going into a bad state? Based on other's reports of similar problems we have investigated a number of system resources and their limits (file descriptors, threads, etc) with no luck. (below are some statistics that seem to show that these other areas are not the problem). One perhaps telling piece of information is that once a tomcat has gotten into this state, we find that many connections to apache end up in the SYN_RECV state (as reported by netstat). It appears that tomcat's listener thread is still accepting connections, but something goes wrong in the handoff to the processor threads such that the connection is left in SYN_RECV. This is curious as a stack trace of tomcat's threads report many (20+) processor threads in await() waiting for the next thing to process. I have included as much relevant information as I can think of below, but am happy to provide additional information should anyone have any ideas as to what may be going wrong. We would be very thankful to hear from anyone who may have any experience of similar problems, or guidance with what to try as next steps. --- Version Information: --- Red Hat Linux: 2.4.21-4.ELsmp #1 SMP i686 i686 i386 GNU/Linux Apache: 2.0.54 mod_jk: 1.2.20 tomcat: 4.1.34 java: 1.5.0_06 (build 1.5.0_06-b05) --- System setup: --- We are running apache to serve static content, and mod_jk with balanced workers to forward requests for dynamic content to 5 tomcat instances. The following are the relevant settings: Apache settings (httpd.conf): Timeout 300 KeepAlive On MaxKeepAliveRequests 100 IfModule prefork.c StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 200 MaxRequestsPerChild 100 /IfModule mod_jk settings (worker.properties) worker.list=loadbalancer worker.tomcat1.port=11009 worker.tomcat1.host=localhost worker.tomcat1.type=ajp13 worker.tomcat1.socket_timeout=30 worker.tomcat1.reply_timeout=6
RE: OutOfMemoryError (but not really out of memory?) cause tomcat processes to hang
You have quite a bit of memory in your 'Old Generation'. You need to determine what you're allocating that isn't being released. We have a fairly substantial amount of data that is loaded on startup and remains for the duration. This probably accounts for part of the old-gen usage. I also see that this number increases _and_ decreases over time. My understanding is that this indicates that we have some long-lived objects, but when a full-gc is forced they get cleaned up as well. Am I understanding this correctly? How many sessions do you have active when the problem occurs? I'm not sure - what is the best way to find this out? What is your session timeout? The default of 30 minutes. What are you putting into your sessions (how much is held by each)? Very little - a handful (~5) smallish (100 char) strings. Are you using any caching technologies like ehCache or hibernate caching? not allowing caches to expire can eat up memory in the Old Generation space. Nope. We use a product called LambdaProbe to monitor our tomcat instances. It's very helpful to count sessions, see what's in them, and to watch the memory and requests in real-time. I will check this out. Thanks for the suggestion. If you really *need* everything you're storing and you don't have any low hanging fruit, you may just need to run with more memory. If you could test such a configuration you may find that the memory utilization stabilizes at an acceptable level. This is part of my confusion, turning the Xmx value up seems to have no effect on the problem. A while back I did a bunch of investigation with gclogging and each tomcat instance seemed to level off it's usage at around 120M. So my guess is that we are not dealing with a standard growing of objects over time, but perhaps a case of something going awry that throws us into the OutOfMemory case. But I'm struggling with how to catch that something. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError (but not really out of memory?) cause tomcat processes to hang
Hello Sharon, just some thoughts On 2/7/07, Sharon French [EMAIL PROTECTED] wrote: --- Version Information: --- Red Hat Linux: 2.4.21-4.ELsmp #1 SMP i686 i686 i386 GNU/Linux Apache: 2.0.54 mod_jk: 1.2.20 tomcat: 4.1.34 java: 1.5.0_06 (build 1.5.0_06-b05) apart from the problem, pretty old setup :-) We currently have Xmx set to 256M, we have tried increasing and decreasing this value and there is no change in the behavior. I assume the Xms value is set to the same value as Xmx? If not, at least earlier jdks tendered to through outofmemory during heap resize. Do you have a graph on memory consumption (System.freeMemory()) ? If yes, is it linear? The changes in Xmx values, did they had any impact on the duration of the servers good state under load? Questions: Is it possible that the initial OutOfMemory error occurs when a large garbage collection is taking place and an OutOfMemoryError is thrown before the memory can be reclaimed throwing the whole system into some sort of bad state? yes. If so, how do we combat this behavior? Give the system enough memory. Enough means, that it always have enough to serve the requests during a gc run. You should study your memory consumption (for example with System.freeMemory output or something) and configure your memory values to be sufficent. I don't know which treshhold is suitable for you, I prefer to have at least 100MB in the VM just in case :-). Is one of the tomcat threads running out of memory, and killing itself, thus freeing the memory? If this is what is happening, does anyone have any advice on catching the tomcat memory usage just prior to going into a bad state? You could try some gc options. How much memory does your machine have? I also assume you run 32 bit? Based on other's reports of similar problems we have investigated a number of system resources and their limits (file descriptors, threads, etc) with no luck. (below are some statistics that seem to show that these other areas are not the problem). Hmm, call me silly, but I would increase PermGen space too. just in case. We are running apache to serve static content, and mod_jk with balanced workers to forward requests for dynamic content to 5 tomcat instances. The following are the relevant settings: Your instances are all on the same machine? Why? Btw, can you reproduce the error in testing environment? Could it be that you simply have a memory leak? regards Leon - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError (but not really out of memory?) cause tomcatprocesses to hang
From: Christopher Schultz [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError (but not really out of memory?) cause tomcatprocesses to hang Sharon French wrote: Is it possible that the initial OutOfMemory error occurs when a large garbage collection is taking place and an OutOfMemoryError is thrown before the memory can be reclaimed Yes. In fact, this is often when OOMs occur, since the JVM is trying hard to avoid the OOM in the first place. Uhhh - no. The JVM will not return an OOME until the GC is completely finished and the request still cannot be satisfied. Even with concurrent GC, failing allocation requests are suspended until a full GC has completed. The exception to the above is if a request is made that could never be satisfied; that will be given an OOME immediately. After an OOM is thrown, the JVM is basically completely hosed. Definitely not true. The JVM is still functional, although the app (Tomcat, in this case) may not be able to proceed. If so, how do we combat this behavior? Fix resource leaks, allocate an appropriate amount of memory, and limit resource use to avoid OOMs in the first place. Very good advice. If you can, use a profiler to find out what (and who) is consuming the heap. However, the 64m given to the PermGen is rather small for an app server, given the number of classes that must be loaded; for that matter, the 256m for the whole heap is pretty tiny these days. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError (but not really out of memory?) cause tomcatprocesses to hang
Uhhh - no. The JVM will not return an OOME until the GC is completely finished and the request still cannot be satisfied. Even with concurrent GC, failing allocation requests are suspended until a full GC has completed. The exception to the above is if a request is made that could never be satisfied; that will be given an OOME immediately. This is very helpful to know. So if after receiving such OOMEs, I run jmap -heap and there is plenty of memory, I'm assuming that whomever was sucking up the memory has either given it back or died off - right? (The dying off would line up with fact that the number of processors tomcat says he's started is less than the number of threads I currently see running.) Now it's down to how do I catch the culprit? Since I don't see steady memory growth, I believe that there must be some error condition that is handled improperly that throws us into some sort of memory cosumption craziness. Since by the time the symptom is present, there is plenty of memory available, I'm having trouble catching the case. Any suggestions? After an OOM is thrown, the JVM is basically completely hosed. Definitely not true. The JVM is still functional, although the app (Tomcat, in this case) may not be able to proceed. This also lines up with my observations. The first several OOMEs don't seem to trip us up, however at some point we hit a threshold that does hose us. Given that after such an error occurs, we have plently of memory, and plenty of threads sitting in await(), any ideas on why the connections to apache end up in SYN_RECV? Am I right in thinking that this indicates that the tomcat listener thread has accepted the connection but then fails to pass it off to a processor thread to handle? Or does the pass off happen but for some reason the socket is hosed? Any thoughts here? If so, how do we combat this behavior? Fix resource leaks, allocate an appropriate amount of memory, and limit resource use to avoid OOMs in the first place. However, the 64m given to the PermGen is rather small for an app server. We don't seem to ever get to a usage of 20M for PermGen, and the OutOfMemoryErrors always indicate java heap space. Is this error message reliable? Can I count on the fact that it's really heap space I'm out of (I've seen reports of this indicating other problems, but all seem to be pre-1.5 jvm). Thanks for the help thus far! - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError (but not really out of memory?) cause tomcat processes to hang
I assume the Xms value is set to the same value as Xmx? If not, at least earlier jdks tendered to through outofmemory during heap resize. I will give this a shot. Thanks. The changes in Xmx values, did they had any impact on the duration of the servers good state under load? Nope. This (along with the lack of memory growth over time) leads me to believe that it's not a normal memory leak, but more a poorly handled error case somewhere that sends us into a tailspin which eventually results in the OOME. (I've done a good bit of monitoring of the heap and things grow and shrink over time, and then seemingly out of the blue we'll see an OOME.) How much memory does your machine have? 4G (currently each of the 5 tomcats has Xmx=256M, apache runs on the same machine) I also assume you run 32 bit? yes Your instances are all on the same machine? Why? Because that's the machine that we've got available right now ;) Btw, can you reproduce the error in testing environment? Could it be that you simply have a memory leak? For all my efforts I cannot reproduce the error on a staging machine. Though I wish I could... - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError (but not really out of memory?) cause tomcatprocesses to hang
From: Sharon French [mailto:[EMAIL PROTECTED] Subject: RE: OutOfMemoryError (but not really out of memory?) cause tomcatprocesses to hang I'm assuming that whomever was sucking up the memory has either given it back or died off - right? Not necessarily. The OOME could have been presented due to an unreasonably large request that could never be satisifed. The memory won't be given back until another GC has run, although minor GCs run quite frequently in a HotSpot JVM. Now it's down to how do I catch the culprit? The first step is to turn on -verbose:gc so you can get a better idea of what's happening when. You might want to update the JVM to the current 1.5.0_11 level, just to make sure it's not some sort of heap management problem in your relatively old level. You should have stack traces to go with the OOME reports; where do these show the execution to be? Given that after such an error occurs, we have plently of memory, and plenty of threads sitting in await(), any ideas on why the connections to apache end up in SYN_RECV? Sorry, I have no experience with httpd. the OutOfMemoryErrors always indicate java heap space. Is this error message reliable? It should be in 1.5; 1.4 gave no clue as to the actual underlying clause. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError (but not really out of memory?) cause tomcatprocesses to hang
We've seen errors where individual queries are improperly bounded or unbounded allowing requests to attempt to retrieve unreasonable amounts of data. A single request like this can take an otherwise stable system and throw it into all sorts of bad behaviors and error conditions. To assist with this debugging you may try to run your jvm with gc logging turned on, and log the http requests coming in. If you then have an OOM crash you can correlate the spike in memory allocation (on a timeline) with certain requests that may have caused it. you can use tools like gcviewer to post-process your gc log files to see what was going on with your memory over time. -John On 2/7/07, Sharon French [EMAIL PROTECTED] wrote: Uhhh - no. The JVM will not return an OOME until the GC is completely finished and the request still cannot be satisfied. Even with concurrent GC, failing allocation requests are suspended until a full GC has completed. The exception to the above is if a request is made that could never be satisfied; that will be given an OOME immediately. This is very helpful to know. So if after receiving such OOMEs, I run jmap -heap and there is plenty of memory, I'm assuming that whomever was sucking up the memory has either given it back or died off - right? (The dying off would line up with fact that the number of processors tomcat says he's started is less than the number of threads I currently see running.) Now it's down to how do I catch the culprit? Since I don't see steady memory growth, I believe that there must be some error condition that is handled improperly that throws us into some sort of memory cosumption craziness. Since by the time the symptom is present, there is plenty of memory available, I'm having trouble catching the case. Any suggestions? After an OOM is thrown, the JVM is basically completely hosed. Definitely not true. The JVM is still functional, although the app (Tomcat, in this case) may not be able to proceed. This also lines up with my observations. The first several OOMEs don't seem to trip us up, however at some point we hit a threshold that does hose us. Given that after such an error occurs, we have plently of memory, and plenty of threads sitting in await(), any ideas on why the connections to apache end up in SYN_RECV? Am I right in thinking that this indicates that the tomcat listener thread has accepted the connection but then fails to pass it off to a processor thread to handle? Or does the pass off happen but for some reason the socket is hosed? Any thoughts here? If so, how do we combat this behavior? Fix resource leaks, allocate an appropriate amount of memory, and limit resource use to avoid OOMs in the first place. However, the 64m given to the PermGen is rather small for an app server. We don't seem to ever get to a usage of 20M for PermGen, and the OutOfMemoryErrors always indicate java heap space. Is this error message reliable? Can I count on the fact that it's really heap space I'm out of (I've seen reports of this indicating other problems, but all seem to be pre-1.5 jvm). Thanks for the help thus far! - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- John Hayward
RE: OutOfMemoryError (but not really out of memory?) cause tomcatprocesses to hang
You should have stack traces to go with the OOME reports; where do these show the execution to be? Nope - no stack trace :(. http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4753347 seems to indicate that this was missed in the 1.5.0 release and came in with 1.5.1. It should be in 1.5; 1.4 gave no clue as to the actual underlying clause. Again, thanks for the info. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError (but not really out of memory?) cause tomcatprocesses to hang
From: Sharon French [mailto:[EMAIL PROTECTED] Subject: RE: OutOfMemoryError (but not really out of memory?) cause tomcatprocesses to hang Nope - no stack trace :(. http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4753347 seems to indicate that this was missed in the 1.5.0 release and came in with 1.5.1. Even the bug report is out of date - 1.5.1 was cancelled quite a while ago, in order to concentrate on stabilizing Java SE 6. Since that's now recently released, you could try that, if you're really gutsy... - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: OutOfMemoryError (but not really out of memory?) cause tomcatprocesses to hang
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chuck, Caldarale, Charles R wrote: In fact, this is often when OOMs occur, since the JVM is trying hard to avoid the OOM in the first place. Uhhh - no. The JVM will not return an OOME until the GC is completely finished and the request still cannot be satisfied The collector itself requires the creation of Java objects on the heap. If the collector does not have enough memory to do the collection, it can easily barf. After an OOM is thrown, the JVM is basically completely hosed. Definitely not true. The JVM is still functional, although the app (Tomcat, in this case) may not be able to proceed. Fair enough. But if Tomcat isn't happy, nobody's happy ;) However, the 64m given to the PermGen is rather small for an app server It's clear from the OP's heap stats that the Perm space is busting. Odd, though: the MaxPermSpace is 64MB, but she is topping out at 20MB or so. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.6 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFym3I9CaO5/Lv0PARAo5MAJ9pBpTS+bi8FIl0UPm/I3IJAxi1WQCdFAWN PbonyKzrLbbTfztMBXyxX1A= =IkDu -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: OutOfMemoryError (but not really out of memory?) cause tomcatprocessesto hang
From: Christopher Schultz [mailto:[EMAIL PROTECTED] Subject: Re: OutOfMemoryError (but not really out of memory?) cause tomcatprocessesto hang The collector itself requires the creation of Java objects on the heap. Not really. The generational HotSpot collector does reserve a portion of the heap into which it copies existing live objects (the To space of the YoungGen area), but no real allocations occur there, nor are any new objects created during the process. Once the copy is done, another portion of the YoungGen space is reserved for use by the next GC. Fair enough. But if Tomcat isn't happy, nobody's happy ;) Depends on who you are; when you look at it from the guts of the JVM (where I hang out), it ain't my problem :-) It's clear from the OP's heap stats that the Perm space is busting. Odd, though: the MaxPermSpace is 64MB, but she is topping out at 20MB or so. The boundary between the spaces is not readily movable, so the max often can't be put to use once the heap is initialized. As you know, running out of PermGen is a pretty typical cause of OOMEs, especially with a relatively small total heap such as this. Need to set -XX:PermSize as well as -XX:MaxPermSize to insure you really get the amount you want. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
[OT]Re: Large number of active sessions
I agree that many have experienced the jsessionid on URLs accessed by bots, perhaps it happens in a Struts environment or some other environment. In certain set-ups (only JSPs, no servlets, no MVC - session is removed and created freshly on each JSP page) the jsessionid doesn't appear on URLs accessed by bots like Googlebot even when the JSP page uses sessions, and that confuses me. I've verified the absense of jsession id in the URLs accessed by Googlebot in the server logs. This inconsistent appearance of jsessionid for certain set-ups doesn't bother me, it's just a matter of curiosity. -Rashmi - Original Message From: Pid [EMAIL PROTECTED] Some bots also use sessions. I disagree, the bot has no capability to decide to use a session. A bot would only appear to use a session if it was HTTP/1.1 capable, and was handling cookies or encoded URLs properly. Most bots get pages asynchronously, I've observed Googlebot hitting url encoded pages with jsessionids generated days beforehand, during a previous index run. This will trigger a new session as a result, but may account for apparently older creation dates appearing the list of active/recent session. (A guess: I don't know enough about the internals of Tomcat to be sure of that.) p Finding fabulous fares is fun. Let Yahoo! FareChase search your favorite travel sites to find flight and hotel bargains. http://farechase.yahoo.com/promo-generic-14795097 - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Tomcat setup / config (System.exit if exception thrown)
When starting up Tomcat, I would like to do a System.exit() if an exception of any kind is thrown. Is there a way to configure this? We are using a large number of frameworks (Spring, Hibernate, log4j) and I want the developers to pay attention right away if there is something wrong and to get it fixed rather than ignoring them (or missing them). I want to shut things down hard and fast if something is wrong during development and force everyone to throw their hands up and be stuck (and make noise) until it is fixed. Any ideas would be appreciated. L -- View this message in context: http://www.nabble.com/Tomcat-setup---config-%28System.exit-if-exception-thrown%29-tf3191721.html#a8860501 Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
