Re: Tomcat dies suddenly

Carl Sat, 30 Jan 2010 09:33:55 -0800

George,

Thanks for your reply.


I will check that IPV6 is disabled... can't remember for certain.

I have had the problem on both openSuse and Slackware which has lead me tobelieve the problem is in the JVM. I guess I could try compiling the JVMfrom source on the target machine.

I ran memTest86 for 30 hours on the first server I encountered the problemwith (the first server with Slackware and 64 bit java that I brought up) andnothing showed up. I then, redid that machine with openSuse (thought theproblem might be Slackware) but openSuse failed as quickly.


Thanks,

Carl

----- Original Message -----From: "George Sexton" <geor...@mhsoftware.com>

To: "'Tomcat Users List'" <users@tomcat.apache.org>
Sent: Saturday, January 30, 2010 11:46 AM
Subject: RE: Tomcat dies suddenly

I've had this happen. Finally, I got a stack trace. In my case, there
appears to be a bug in GLIBC, so when a reverse IP address lookup is done
AND there is an IPV6 entry, it causes a problem.

The solution appeared to be disabling IPV6 since I'm not using it. This is
on OpenSuSE.

It would be worth checking.

You might also run MemTest86+ on it if you haven't already.

George Sexton
MH Software, Inc.
http://www.mhsoftware.com/
Voice: 303 438 9585

-----Original Message-----
From: Carl [mailto:c...@etrak-plus.com]
Sent: Friday, January 29, 2010 12:54 PM
To: Tomcat Users List
Subject: Tomcat dies suddenly

6-7 weeks ago, we built up some new servers and started having sudden
failures... Tomcat just stops with no error message, no system error
messages, nothing that I have been able to find so far.

To refresh everyone's memory, this is a new server, a Dell T110 with a
Xeon 3440 processor and 4GB memory.  I have turned off both the turbo
mode and hyperthreading.

The environment:

64 bit Slackware Linux

java version "1.6.0_17"
Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode)

Tomcat: apache-tomcat-6.0.20

These are the current JAVA_OPTS="-Xms1024m -Xmx1024m -XX:PermSize=368m
-XX:MaxPermSize=368m"

I have observed the memory usage and general performance with Java
VisualVM and have seen nothing strange.  I thought I was seeing GC as
memory usage was going up and down but in fact it was mostly people
coming onto the system and leaving it.  After several hours, the memory
settles to a baseline of about 375MB.  Forced GC never takes it below
that value and the ups and downs from the people coming onto and
leaving the system also returns it to pretty much that value.  The
maximum memory used never was above 700MB for the entire day.

The server runs well, idling along at 2-5% load, except for a quick
spike during GC, serving jsp's, etc. at a reasonable speed.  Without
warning and with no tracks in any log (Tomcat or system) or to the
console, Tomcat just shuts down.  I can usually simply restart it as
the ports used by Tomcat are closed... today, I needed to run
shutdown.sh before I could run startup.sh (startup.sh gave no errors
but would not start Tomcat until I ran shutdown.sh and that process put
nothing in the logs... this is the first time this has happened.)

Sometimes, the system will run for a week, sometimes for only several
hours, sometimes only for a few minutes.  Today, it ran until about
1:00PM and has been down four times since then.

The failure (Tomcat shutting down) is not always the same place in the
code (I have some debugging messages going to catalina.out.)

Load does not seem to make a difference.

I have tried another sever (Dell T105, AMD processor, 6GB memory) and
have observed the same results.   I have run memTest86 on the T110 for
about 30 hours and it showed nothing.

I rebuilt the T110 with SUSE linux, Java 1.6.18 and Tomcat 6.0.24... it
lasted 15 minutes.  I have used the same server.xml on all the
installs:

<Server port="8005" shutdown="SHUTDOWN">

<Listener className="org.apache.catalina.core.AprLifecycleListener"
SSLEngine="on" />

<Listener className="org.apache.catalina.core.JasperListener" />

<Listener
className="org.apache.catalina.mbeans.ServerLifecycleListener" />

<Listener
className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener"
/>

<!-- Global JNDI resources

Documentation at /docs/jndi-resources-howto.html

-->

<GlobalNamingResources>

<!-- Editable user database that can also be used by

UserDatabaseRealm to authenticate users

-->

<Resource name="UserDatabase" auth="Container"

type="org.apache.catalina.UserDatabase"

description="User database that can be updated and saved"

factory="org.apache.catalina.users.MemoryUserDatabaseFactory"

pathname="conf/tomcat-users.xml" />

</GlobalNamingResources>

<!-- A "Service" is a collection of one or more "Connectors" that share

a single "Container" Note: A "Service" is not itself a "Container",

so you may not define subcomponents such as "Valves" at this level.

Documentation at /docs/config/service.html

-->

<Service name="Catalina">

<!--The connectors can use a shared executor, you can define one or
more named thread pools-->

<!--

<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"

maxThreads="150" minSpareThreads="4"/>

-->

<!-- A "Connector" represents an endpoint by which requests are
received

and responses are returned. Documentation at :

Java HTTP Connector: /docs/config/http.html (blocking & non-blocking)

Java AJP Connector: /docs/config/ajp.html

APR (HTTP/AJP) Connector: /docs/apr.html

Define a non-SSL HTTP/1.1 Connector on port 8080

-->

<Connector port="8080" protocol="HTTP/1.1"

maxHttpHeaderSize="8192"

maxThreads="600" minSpareThreads="25" maxSpareThreads="75"

enableLookups="false" redirectPort="8443" scheme="http"
acceptCount="100"

connectionTimeout="20000" disableUploadTimeout="true" />

<!-- A "Connector" using the shared thread pool-->

<!--

<Connector executor="tomcatThreadPool"

port="8080" protocol="HTTP/1.1"

connectionTimeout="20000"

redirectPort="8443" />

-->

<!-- Define a SSL HTTP/1.1 Connector on port 8443

This connector uses the JSSE configuration, when using APR, the

connector should be using the OpenSSL style configuration

described in the APR documentation -->

<Connector port="8443" maxHttpHeaderSize="8192"

maxThreads="600" minSpareThreads="25" maxSpareThreads="75"

enableLookups="false" disableUploadTimeout="true"

acceptCount="100" scheme="https" secure="true"

clientAuth="false" sslProtocol="TLS" SSLEnabled="true"

keystoreFile="/usr/local/certs/tomcat_keystore.ks"
keystorePass="jellybean"/>

<!-- Define a SSL HTTP/1.1 Connector on port 443 -->

<Connector port="443" maxHttpHeaderSize="8192"

maxThreads="600" minSpareThreads="25" maxSpareThreads="75"

enableLookups="false" disableUploadTimeout="true"

acceptCount="100" scheme="https" secure="true"

clientAuth="false" sslProtocol="TLS" SSLEnabled="true"

keystoreFile="/usr/local/certs/tomcat_keystore.ks"
keystorePass="jellybean"/>

<!--

<Connector port="8443" protocol="HTTP/1.1" SSLEnabled="true"

maxThreads="150" scheme="https" secure="true"

clientAuth="false" sslProtocol="TLS" />

-->

<!-- Define an AJP 1.3 Connector on port 8009 -->

<Connector port="8009"

enableLookups="false" redirectPort="443" protocol="AJP/1.3" />

<!-- An Engine represents the entry point (within Catalina) that
processes

every request. The Engine implementation for Tomcat stand alone

analyzes the HTTP headers included with the request, and passes them

on to the appropriate Host (virtual host).

Documentation at /docs/config/engine.html -->

<!-- You should set jvmRoute to support load-balancing via AJP ie :

<Engine name="Catalina" defaultHost="localhost" jvmRoute="jvm1">

-->

<Engine name="Catalina" defaultHost="localhost">

<!--For clustering, please take a look at documentation at:

/docs/cluster-howto.html (simple how to)

/docs/config/cluster.html (reference documentation) -->

<!--

<Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster"/>

-->

<!-- The request dumper valve dumps useful debugging information about

the request and response data received and sent by Tomcat.

Documentation at: /docs/config/valve.html -->

<!--

<Valve className="org.apache.catalina.valves.RequestDumperValve"/>

-->

<!-- This Realm uses the UserDatabase configured in the global JNDI

resources under the key "UserDatabase". Any edits

that are performed against this UserDatabase are immediately

available for use by the Realm. -->

<Realm className="org.apache.catalina.realm.UserDatabaseRealm"

resourceName="UserDatabase"/>

<!-- Define the default virtual host

Note: XML Schema validation will not work with Xerces 2.2.

-->

<Host name="localhost" appBase="webapps"

unpackWARs="true" autoDeploy="true" deployOnStartup="true"

xmlValidation="false" xmlNamespaceAware="false">

<!-- SingleSignOn valve, share authentication between web applications

Documentation at: /docs/config/valve.html -->

<!--

<Valve className="org.apache.catalina.authenticator.SingleSignOn" />

-->

<!-- Access log processes all example.

Documentation at: /docs/config/valve.html -->

<!--

<Valve className="org.apache.catalina.valves.AccessLogValve"
directory="logs"

prefix="localhost_access_log." suffix=".txt" pattern="common"
resolveHosts="false"/>

-->

</Host>

</Engine>

</Service>

</Server>

When Tomcat shuts down, the memory that it was using seems to still be
held (as seen from top) but it is nowhere near the machine physical
memory.

The application has been running on an older server (Dell 600SC, 32 bit
Slackware, 2GB memory) for several years and, while the application
will throw exceptions now and then, it never crashed.  This lead me to
believe the problem had something to do with the 64 bit JVM but, with
without seeing errors anywhere, I can't be certain and don't know what
I can do about it except go back to 32 bit.

One time, I observed the heap and permGen memory usage with Visual JVM.
It was running around 600MB before I forced a GC and 375MB afterward.
Speed was good.  Memory usage from top was 2.4GB.  Five minutes later,
Tomcat stopped leaving no tracks that I could find.  The memory usage
from top was around 2.4GB.  The memory usage from Visual JVM was still
showing 400MB+ although the Tomcat process was gone.  I restarted
Tomcat (did not reboot) so Tomcat had been shutdown gracefully enough
to close the ports (8080, 8443, 443.)  Tomcat stayed up for less than
an hour (under light load) and stopped again.  The memory used
according to top was less than 3GB but I didn't get the exact number.
I restarted it again (no server reboot) and it ran for the rest of the
night (light load) and top was showing 3.3GB for memory in the morning.

Anyone have any ideas how I might track this problem down?

Thanks,

Carl



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat dies suddenly

Reply via email to