On 05.10.2009 18:58, Joe Hansen wrote: > Thank you so much for your tips, Rainer! > > The websites went down yet again. Increasing the java heap size took > care of the OutOfMemoryError, but the number of httpd processes keep > increasing until the websites crash. I haven't added any new code in > the past few months, hence I am surprised why the requests are getting > stuck. Here's a link to the tomcat thread dumps: > http://pastebin.com/m17eea139
Just tried to look at it, but pastebin replies with: ==================== Down for maintenance - 6th Oct 2009 Pastebin.com is getting an unprecedented amount of traffic due to a news story in which some leaked Hotmail passwords have been pasted on this site ... ==================== Let's see, when they will be up again. They're running Apache 1.3.33 ... > Please let me know if you cannot view it and I will email the relevant > portion of the catalina.out file to you. Is there an easy way to find > out what code is causing the requests to get stuck? > > Thank you! > > Joe > > > On Sun, Oct 4, 2009 at 2:36 PM, Rainer Jung <rainer.j...@kippdata.de> wrote: >> Hi Joe, >> >> On 04.10.2009 21:45, Joe Hansen wrote: >>> Rainer, Thank you so much for your kind reply! >>> >>> I have increased the java heap size to 512MB (-Xms512m -Xmx512m). I am >>> hoping that would fix the issue. I had configured our webserver to use >>> Jasig's Central Authentication System (CAS). Recently I increased the >>> session timeout from 30 minutes to 4 hours. I am guessing that must >>> have had an impact on the number of tickets that the CAS could store >>> in the Java's memory space. >>> >>> I did run the kill -QUIT command against the tomcat process. It did >>> generate a huge output in the catalina.out file. I am unable to >>> decipher it. I do not want to post it to the mailing list because its >>> very long. Would you be able to please tell me what should I be >>> looking for within this long thread dump? >> >> Can you put it somewhere on the web, so we can look at it, or are you >> afraid there is something private in there? You could use pastebin or >> something similar in case you do not have a public web server yourself. >> >> If you don't want to post in public, you can also mail it to me, I will >> post the result, in case I find something relevant. >> >> Regards, >> >> Rainer >> >>> On Sat, Oct 3, 2009 at 12:24 PM, Rainer Jung <rainer.j...@kippdata.de> >>> wrote: >>>> On 03.10.2009 20:07, Joe Hansen wrote: >>>>> Hey All, >>>>> >>>>> I get this error (java.lang.OutOfMemoryError: Java heap space) after >>>>> my Apache 2.0/Tomcat 5.5/mod_jk installation has been up and running >>>>> for a few hours. This problem started just since two days. Never had >>>>> this issue before! >>>>> >>>>> I have also noticed that as soon as I startup the server, 9 httpd >>>>> processes start. Number of httpd processes keep on increasing until I >>>>> get the OutOfMemoryError. >>>>> $ps -aef | grep httpd >>>>> root 31984 1 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>>> apache 31987 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>>> apache 31988 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>>> apache 31989 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>>> apache 31990 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>>> apache 31991 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>>> apache 31992 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>>> apache 31993 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>>> apache 31994 31984 0 11:23 ? 00:00:00 /usr/sbin/httpd >>>> >>>> Sounds like requests get stuck or responses are only returned very slowly. >>>> >>>> I would take thread dumps during the time requests pile up (e.g. httpd >>>> process count increases). Thread dumps are generated by "kil -QUIT" >>>> against the Tomcat process. Result is written to catalina.out. Always >>>> take afew thread dumps shortly after each other, e.g. 3 dumps each 3 >>>> seconds apart from the previous one, so that you can find out, if a >>>> status in a dump is pure coincidence or lasts for somewhat longer. >>>> >>>>> $ps -aef | grep tomcat >>>>> root 31949 1 43 11:23 pts/0 00:00:58 /usr/java/jdk/bin/java >>>>> -Djava.u >>>>> il.logging.manager=org.apache.juli.ClassLoaderLogManager >>>>> -Djava.util.logging.co >>>>> fig.file=/usr/lib/apache-tomcat/conf/logging.properties >>>>> -Djava.endorsed.dirs=/u >>>>> r/lib/apache-tomcat/common/endorsed -classpath >>>>> :/usr/lib/apache-tomcat/bin/boot >>>>> trap.jar:/usr/lib/apache-tomcat/bin/commons-logging-api.jar >>>>> -Dcatalina.base=/us >>>>> /lib/apache-tomcat -Dcatalina.home=/usr/lib/apache-tomcat >>>>> -Djava.io.tmpdir=/usr >>>>> lib/apache-tomcat/temp org.apache.catalina.startup.Bootstrap start >>>> >>>> There is no Java memory configuration included above (i.e. al defaults). >>>> It might well be, that you have to explicitely set heap size, perm size >>>> and if you like also eden and semi spaces. >>>> >>>>> Can someone on this list please help me resolve this issue. >>>>> >>>>> Thanks you, >>>>> Joe >>>> >>>> Regards, >>>> >>>> Rainer --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
