From the original posting:
This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB
memory. I have turned off both the turbo mode and hyperthreading.
The environment:
64 bit Slackware Linux
java version "1.6.0_17"
Java(TM) SE Runtime Environment (build 1.6.0_17-b04)
Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode)
Tomcat: apache-tomcat-6.0.20
These are the current JAVA_OPTS="-Xms1024m -Xmx1024m -XX:PermSize=368m
-XX:MaxPermSize=368m"
In the previous posting, I noted that I have observed the memory usage
and
general performance with Java VisualVM and have seen nothing strange. GC
seems to be performing well and the memory rarely gets anywhere near the
max. New information: I thought I was seeing GC as memory usage was
going
up and down but in fact it was mostly people coming onto the system and
leaving it. After several hours, the memory settles to a baseline of
about
375MB. Forced GC never takes it below that value and the ups and downs
from
the people coming onto and leaving the system also returns it to pretty
much
that value. The maximum memory used never was above 700MB for the entire
day.
The server runs well, idling along at 2-5% load, except for a quick spike
during GC, serving jsp's, etc. at a reasonable speed. Without warning
and
with no tracks in any log (Tomcat or system) or to the console, the JVM
will
just go away, disappear. New information: The JVM does not just go away
but
somehow Tomcat shutsdown as the ports used by Tomcat are closed (pointed
out
by Konstantin.) Sometimes, the system will run for a week, sometimes for
only several hours. Initially, I thought the problem was the turbo or
hyperthreading but, no, the problem persists.
When Tomcat shuts down, the memory that it held is still being held (as
seen from top) but it is nowhere near the machine physical memory.
The application has been running on an older server (Dell 600SC, 32 bit
Slackware, 2GB memory) for several years and, while the application will
throw exceptions now and then, it never crashed. This lead me to believe
the problem had something to do with the 64 bit JVM but, with without
seeing
errors anywhere, I can't be certain and don't know what I can do about it
except go back to 32 bit.
New information.
Last evening, I observed the heap and permGen memory usage with Visual
JVM.
It was running around 600MB before I forced a GC and 375MB afterward.
Speed was good. Memory usage from top was 2.4GB. Five minutes later,
Tomcat stopped leaving no tracks that I could find. The memory usage
from
top was around 2.4GB. The memory usage from Visual JVM was still showing
400MB+ although the Tomcat process was gone. I restarted Tomcat (did not
reboot) so Tomcat had been shutdown gracefully enough to close the ports
(8080, 8443, 443.) Tomcat stayed up for less than an hour (under light
load) and stopped again. The memory used according to top was less than
3GB
but I didn't get the exact number. I restarted it again (no server
reboot)
and it ran for the rest of the night (light load) and top was showing
3.3GB
for memory this morning.
I brought up a new server last night and have switched to that server for
production (same Linux, JDK, server.xml, JAVA_OPTS, etc.). It would seem
if
the problem is with my application or the JVM, that the problem will
follow
me to the new server.
Anyone have any ideas how I might track this problem down?
Thanks,
Carl
