Tomcat 4.1.31 crashing with memory errors, crashing with no errors and shutting down cleanly without manual intervention
*We are having an issue with the tomcat service crashing version 4.1.31, sometimes with these memory errors and sometimes not. We have a backup but once the load moves to that server the backup crashes also almost immediately after the load balancer switches. This has happened multiple times to us over the past few days with no changes made to the server, just a slight increase in load. We have no option to upgrade right now since this is included in an enterprise application. We have the heap space set to the maximum that it will allow us to. The vendor recommended 1536 but this setting caused tomcat to not start at all, ~1200 was the highest I was able to get it to go. I watched it crash yesterday and it did a clean shutdown with no errors which was differrent. The vendor doesn't have a fix yet and I wanted to see what you all think. There are other large logs, if there is something specific that we would need to look at then let me know and I will try and get it out. Thanks, Sean Dec 5, 2007 4:11:05 PM - Error creating event for domain ss-ls01p: java.lang.OutOfMemoryError: Java heap space at java.util.ArrayList.init(ArrayList.java:113) at java.util.ArrayList.init(ArrayList.java:120) at com.aprisma.spectrum.app.event.web.model.BackEndEventDataModel.makeEventObject (BackEndEventDataModel.java:566) at com.aprisma.spectrum.app.event.web.model.BackEndEventDataModel.applyEventUpdate (BackEndEventDataModel.java:748) at com.aprisma.spectrum.app.event.web.model.BackEndEventDataModel$QueuedEventUpdate.run (BackEndEventDataModel.java:116) at com.aprisma.util.thread.JobQueue.runJobThread(JobQueue.java:232) at com.aprisma.util.thread.JobQueue.access$000(JobQueue.java:38) at com.aprisma.util.thread.JobQueue$JobRunnable.run(JobQueue.java:47) at java.lang.Thread.run(Thread.java:595) java.lang.OutOfMemoryError: Java heap space at com.aprisma.spectrum.core.idl.CsCAttribute.CsCAttrValueHelper.read( CsCAttrValueHelper.java:53) at com.aprisma.spectrum.core.idl.CsCAttribute.CsCAttrValListHelper.read( CsCAttrValListHelper.java:56) at com.aprisma.spectrum.core.idl.CsCEventDomainPackage.CsCEventHelper.read( CsCEventHelper.java:64) at com.aprisma.spectrum.core.idl.CsCEventDomainPackage.CsCEventListHelper.read( CsCEventListHelper.java:63) at com.aprisma.spectrum.core.idl.CsCEventWatchCBPOA._invoke( CsCEventWatchCBPOA.java:66) at com.aprisma.spectrum.core.idl.CsCEventWatchCBPOA._invoke( CsCEventWatchCBPOA.java:51) at com.inprise.vbroker.poa.POAImpl.invoke(POAImpl.java:2822) at com.inprise.vbroker.poa.ActivationRecord.invoke(ActivationRecord.java :186) at com.inprise.vbroker.GIOP.GiopProtocolAdapter.doRequest( GiopProtocolAdapter.java:838) at com.inprise.vbroker.GIOP.GiopProtocolAdapter.dispatchMessage( GiopProtocolAdapter.java:1120) at com.inprise.vbroker.orb.TPDispatcherImpl$TPDispatcher.run( TPDispatcherImpl.java:100) at com.inprise.vbroker.orb.ThreadPool$PoolWorker.run(ThreadPool.java:76) java.lang.OutOfMemoryError: Java heap space Dec 5, 2007 4:12:21 PM - EventFormatHelper: Error parsing event table enum file nmiEventType: java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space Exception in thread HostConfig[localhost] java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space Exception in thread CORBAMonitorPool-243: IDLE java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space Exception in thread VBJ ThreadPool Worker java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space Exception in thread VBJ ThreadPool Worker java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space Exception in thread StandardManager[/examples] java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space Exception in thread CORBAMonitorPool-247: IDLE Exception in thread VBJ ThreadPool Worker java.lang.NullPointerException at org.apache.coyote.tomcat4.OutputBuffer.realWriteChars( OutputBuffer.java:563) at org.apache.tomcat.util.buf.CharChunk.flushBuffer(CharChunk.java:435) at org.apache.coyote.tomcat4.OutputBuffer.close(OutputBuffer.java:268) at org.apache.coyote.tomcat4.CoyoteResponse.finishResponse( CoyoteResponse.java:438) at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java :153) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java :799) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection (Http11Protocol.java:705) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java :577) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run( ThreadPool.java:683) at java.lang.Thread.run(Thread.java:595) java.lang.OutOfMemoryError: Java heap space java.lang.OutOfMemoryError: Java heap space
Re: Tomcat 4.1.31 crashing with memory errors, crashing with no errors and shutting down cleanly without manual intervention
Pete, Thanks for the quick response. Your evaluation is similar to what I have been saying to my counterparts in regards to load balancing. The back-end servers seem to be responding in a timely fashion right now. We have performance data from the time period and nothing seems abnormal. I think this can be attributed mostly to the increased load, on that that day due to shift switchover we have about 50% more users. Wednesday was the most that we have ever had. Thanks for the suggestions, you certainly helped me backup my theory to the powers that be. Regards, Sean On Dec 6, 2007 10:43 AM, Peter Crowther [EMAIL PROTECTED] wrote: From: Peter Crowther Looks like that slight increase in load has tipped you over from being just-about-alright to just-about-failing. If you can't increase heap space, can't decrease load and can't alter the application, your only remaining choice is to add capacity: install another server and load-balance across N+1 servers rather than N servers. Also remember that if you have plenty of spare RAM on the machines hosting Tomcat, and it's just maximum heap size that's the issue, then you might be able to run two Tomcat processes (on different ports) on the same server, avoiding the need to deploy new hardware. Bad Netiquette to follow up my own answer, but I missed a point. Often, the amount of heap space required is related to the number of concurrent requests being processed. If this is the case for you, and you're using back-end services like relational databases, web services and the like, then you might be able to salvage the situation by speeding up one or more of those back-end services so that requests complete faster and hence there are fewer concurrent requests waiting on the Tomcat webapps. It's another option to consider if you have some control over these services but none over the Tomcat app. - Peter - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Any sufficiently advanced technology is indistinguishable from magic. - Arthur C. Clarke
Re: Tomcat 4.1.31 crashing with memory errors, crashing with no errors and shutting down cleanly without manual intervention
Andrew, The performance data that we have is for the backend servers. We are monitoring the normal things cpu, memory, load swap, context switches, threads etc etc on the system but nothing specific to the tomcat service, just the overall health of the system. It is not easy to put a lot of profiling on a production server, it may be a good thing to do to find out the issue but not something that anyone wants on in production. Cataline.out from one of the crashes is below : 2007-12-05 13:42:50 CoyoteAdapter An exception or error occurred in the container during the request processing java.lang.OutOfMemoryError: Java heap space at org.apache.tomcat.util.buf.ByteChunk.toString(ByteChunk.java:457) at org.apache.tomcat.util.buf.MessageBytes.toString(MessageBytes.java :192) at org.apache.tomcat.util.http.MimeHeaders.getHeader(MimeHeaders.java :293) at org.apache.coyote.Request.getHeader(Request.java:345) at org.apache.coyote.tomcat4.CoyoteAdapter.postParseRequest( CoyoteAdapter.java:201) at org.apache.coyote.tomcat4.CoyoteAdapter.service(CoyoteAdapter.java :150) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java :799) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection (Http11Protocol.java:705) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java :577) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run( ThreadPool.java:683) at java.lang.Thread.run(Thread.java:595) 2007-12-05 13:43:55 CoyoteAdapter An exception or error occurred in the container during the request processing java.lang.OutOfMemoryError: Java heap space 2007-12-05 13:44:08 CoyoteAdapter An exception or error occurred in the container during the request processing java.lang.OutOfMemoryError: Java heap space 2007-12-05 13:44:21 CoyoteAdapter An exception or error occurred in the container during the request processing java.lang.OutOfMemoryError: Java heap space 2007-12-05 13:44:21 CoyoteAdapter An exception or error occurred in the container during the request processing java.lang.OutOfMemoryError: Java heap space 2007-12-05 13:44:34 CoyoteAdapter An exception or error occurred in the container during the request processing java.lang.OutOfMemoryError: Java heap space 2007-12-05 13:44:46 CoyoteAdapter An exception or error occurred in the container during the request processing java.lang.OutOfMemoryError: Java heap space Sun 1.4.2 Jvm A stack trace was not done yet. backend stats for cpu/memory/disk look normal we collect them every 15 minutes. The thing I am wondering also is why it shuts down cleanly sometimes. Is there something internally that triggers that with this version of Tomcat, would an out of heap memory situation cause that. Maybe its in one of the logs but I am still going through all of them. There are multiple megs of them. Sean On Dec 6, 2007 11:23 AM, Andrew Miehs [EMAIL PROTECTED] wrote: On 06/12/2007, at 5:12 PM, Peter Crowther wrote: From: Sean Carnes [mailto:[EMAIL PROTECTED] The back-end servers seem to be responding in a timely fashion right now. We have performance data from the time period and nothing seems abnormal. I unfortunately missed the first part of this thread ... If you are having problems and your performance data show nothing abnormal, then you either do not have enough data or you do not have a problem.. :-) What errors are you seeing? (What is in catalina.out)? Are you running out of threads? (you are probably runing JVM 1.42 based on the version of tomcat you are running - Sun and IBM JVM 1.42 used to respond with Out of memory when you were out of threads)... Have you done a stack trace on the tomcat? Do you have disk performance stats from the backend as well as CPU, and load? At the moment, it could be anything, but if you say you have 50% more users, this something could very easily be the effect of a little more load, which brings the whole thing to a standstill Cheers Andrew -- Any sufficiently advanced technology is indistinguishable from magic. - Arthur C. Clarke
Re: Tomcat 4.1.31 crashing with memory errors, crashing with no errors and shutting down cleanly without manual intervention
Andrew, Believe me I wish this was on Linux s many things would be so much easier for someone who is used to unix, but unfortunately we are on windows server 2k3 the decision was made before I got here. We are moving to either linux or Solaris in the future but that will take a while. The highest that we could set the heap was to 1200. I tried higher and it would not start. It also seemed somewhat unstable above 1024 which was the previous setting, slowness updating the client and other things. The company that develops the enterprise s/w that uses tomcat said that settings over 1024 were unstable so my feeling was confirmed by them. We use an snmp agent to our nms to get system statistics. There was nothing out of the ordinary, other than tomcat using about 1298M which is the most that we have seen it use. We have everything up and running now and we are load balancing which is how it should have been set up in the first place. The memory usage of tomcat has dropped ~40% since we made that change. It was normally using between 600M 800M now its down to about 4-500M give or take. I can't run ps on Windows unfortunately.As far as the JVM settings are concerned I will have to see where they are set. Since this is not just tomcat things are not all where they normally are but I will check. As far as it dumping core we have 2 .hprof files in our directory which is what the vendor had us send in so that they can look through them each one of those came when it had full thread dump in the log files. The other time we saw no .hprof but the message in the stdout.log that says tomcat is shutting down. Regards, Sean On Dec 6, 2007 12:42 PM, Andrew Miehs [EMAIL PROTECTED] wrote: Do you also have performance data for the front end machines? What OS are you running? Would definitely recommending installing sar (or sysstat package) if you are running linux. If Linux, which kernel? If it really is heap, have a look at: http://hausheer.osola.com/docs/5 for a simple description on how to fix this... Below is the google link which shows how this was found http://www.google.com/search?rls=en-usq=java.lang.OutOfMemoryError:+Java+heap+spaceie=UTF-8oe=UTF-8 The thing I am wondering also is why it shuts down cleanly sometimes. Is there something internally that triggers that with this version of Tomcat, would an out of heap memory situation cause that. Maybe its in one of the logs but I am still going through all of them. There are multiple megs of them. Are you sure it shuts down cleanly - this seems strange - you may want to chnage `ulimit -c 1000` so that you can see whether it really core dumps or not what does `ps auxH` report on your machine? How much memory is the thing using? How high have you configured your memory settings for the JVM? Cheers Andrew -- Any sufficiently advanced technology is indistinguishable from magic. - Arthur C. Clarke
