Thank you so much for your tips, Rainer! The websites went down yet again. Increasing the java heap size took care of the OutOfMemoryError, but the number of httpd processes keep increasing until the websites crash. I haven't added any new code in the past few months, hence I am surprised why the requests are getting stuck. Here's a link to the tomcat thread dumps: http://pastebin.com/m17eea139
Please let me know if you cannot view it and I will email the relevant portion of the catalina.out file to you. Is there an easy way to find out what code is causing the requests to get stuck? Thank you! Joe On Sun, Oct 4, 2009 at 2:36 PM, Rainer Jung <rainer.j...@kippdata.de> wrote: > Hi Joe, > > On 04.10.2009 21:45, Joe Hansen wrote: >> Rainer, Thank you so much for your kind reply! >> >> I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am >> hoping that would fix the issue. I had configured our webserver to use >> Jasig's Central Authentication System (CAS). Recently I increased the >> session timeout from 30 minutes to 4 hours. I am guessing that must >> have had an impact on the number of tickets that the CAS could store >> in the Java's memory space. >> >> I did run the kill -QUIT command against the tomcat process. It did >> generate a huge output in the catalina.out file. I am unable to >> decipher it. I do not want to post it to the mailing list because its >> very long. Would you be able to please tell me what should I be >> looking for within this long thread dump? > > Can you put it somewhere on the web, so we can look at it, or are you > afraid there is something private in there? You could use pastebin or > something similar in case you do not have a public web server yourself. > > If you don't want to post in public, you can also mail it to me, I will > post the result, in case I find something relevant. > > Regards, > > Rainer > >> On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung <rainer.j...@kippdata.de> wrote: >>> On 03.10.2009 20:07, Joe Hansen wrote: >>>> Hey All, >>>> >>>> I get this error (java.lang.OutOfMemoryError: Java heap space) after >>>> my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running >>>> for a few hours. This problem started just since two days. Never had >>>> this issue before! >>>> >>>> I have also noticed that as soon as I startup the server, 9 httpd >>>> processes start. Number of httpd processes keep on increasing until I >>>> get the OutOfMemoryError. >>>> $ps -aef | grep httpd >>>> root 31984 1 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31987 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31988 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31989 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31990 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31991 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31992 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31993 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> apache 31994 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>> >>> Sounds like requests get stuck or responses are only returned very slowly. >>> >>> I would take thread dumps during the time requests pile up (e.g. httpd >>> process count increases). Thread dumps are generated by "kil -QUIT" >>> against the Tomcat process. Result is written to catalina.out. Always >>> take afew thread dumps shortly after each other, e.g. 3 dumps each 3 >>> seconds apart from the previous one, so that you can find out, if a >>> status in a dump is pure coincidence or lasts for somewhat longer. >>> >>>> $ps -aef | grep tomcat >>>> root 31949 1 43 11:23 pts/0 00:00:58 /usr/java/jdk/bin/java >>>> -Djava.u >>>> il.logging.manager=org.apache.juli.ClassLoaderLogManager >>>> -Djava.util.logging.co >>>> fig.file=/usr/lib/apache-tomcat/conf/logging.properties >>>> -Djava.endorsed.dirs=/u >>>> r/lib/apache-tomcat/common/endorsed -classpath >>>> :/usr/lib/apache-tomcat/bin/boot >>>> trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar >>>> -Dcatalina.base=/us >>>> /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat >>>> -Djava.io.tmpdir=/usr >>>> lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start >>> >>> There is no Java memory configuration included above (i.e. al defaults). >>> It might well be, that you have to explicitely set heap size, perm size >>> and if you like also eden and semi spaces. >>> >>>> Can someone on this list please help me resolve this issue. >>>> >>>> Thanks you, >>>> Joe >>> >>> Regards, >>> >>> Rainer >>> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
