Ok, fishing in the dark really, but trying to accumulate more clues.
What about the other questions (how is Tomcat started) ?
And, looking at your original "ps" list, some vague thing :
all these ghost processes seem to
a) not be doing much (CPU time 0), even after a long time.
b) all be started around the clock hour:00, or the hour:30, give or take a
couple of minutes.
Got anything particular that would run at approximately 30 minutes interval ?
(Like some monitoring process which sends a request every 30 minutes just to check ? And
if so : has this been added just before the problem started ? and what request does it send ?)
As an alternative to netstat under Solaris, there is the "lsof" utility.
It has a million options to sort through, but it certainly can tell you which process owns
which TCP socket.
The point here is to try to figure out what these ghosts are doing and which configuration
settings they use. They seem to be using the same command-line settings as the original,
but we only see a small part (there may be other switches on your version of "ps" which
list the whole command line of each process.).
You cannot have several processes opening the same TCP port in "LISTEN" mode.
If all these processes use the same config and try to open the same ports (Tomcat opens
several), all but the first one should fail, and that should normally leave a trace in the
Tomcat logs.
Else, they are not real processes and really are threads, no matter what "ps"
is saying.
You could also try taking some thread dumps of the Tomcat JVM (see the Tomcat FAQ), to see
if some thread-id there match the PID's seen in your ps displays.
Dave Stubbs wrote:
Solaris environment is....
SunOS ********** 5.10 Generic_127127-11 sun4v sparc
SUNW,SPARC-Enterprise-T5120
System = SunOS
Node = **********
Release = 5.10
KernelID = Generic_127127-11
Machine = sun4v
BusType = <unknown>
Serial = <unknown>
Users = <unknown>
OEM# = 0
Origin# = 1
NumCPU = 64
-----Original Message-----
From: André Warnier [mailto:a...@ice-sa.com]
Sent: 09 September 2011 10:25
To: Tomcat Users List
Subject: Re: Tomcat 5.5.27 - Solaris 10 - Tomcat process starts additional
copies of itself with only 1 thread and no CPU
Intriguing.
For a complement of information :
- what is the exact platform ?
- is Tomcat started via jsvc ?
- as a matter of fact, *how* is the "main" Tomcat being started ?
What does the command "netstat -pan | grep LISTEN" show when you have such
multiple copies running ?
(the form of the command will depend a bit on the platform; the command
above, under Linux, shows the ports on which processes are listening, and
the corresponding PID's and maybe program name)
Dave Stubbs wrote:
We're experiencing a problem on both production and test systems that
has only started to occur in the last few weeks (no new version of any
software have been put live to anyone's knowledge) that has us quite
stumped.
We are seeing tomcat starting up additional copies of itself, each new
copy is allocating a chunk of storage, it only starts 1 thread,
nothing gets written to any logs and no CPU is being listed as having been
used.
This keeps happening, and if it's not monitored the system memory all
gets taken up and we start to get SIGBUS errors.
We've tried turning all logging on, there don't appear to be any
software related events at the heart of the issue, and we can't see
any system triggers.
When we list the system tasks we see that the ghost process PPID is
the same as the PID of the still running main tomcat instance. Here is
list of what happened when it wasn't picked up for a while.
tomcat 21115 21114 0 14:46:44 ? 193:04 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
773 21115 0 17:29:27 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
4270 21115 0 18:29:29 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
5988 21115 0 18:59:32 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
27209 21115 0 01:01:02 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
13155 21115 0 05:32:09 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
16785 21115 0 06:34:52 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
12948 21115 0 20:58:50 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
20054 21115 0 23:00:21 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti tomcat
23730 21115 0 00:03:47 ? 0:00 /usr/java/bin/java
-Djava.util.logging.config.file=/opt/apache-tomcat -5.5.27-ti
The main process 21115 starts off fine, and is even still running with
no problems the next day, but we keep getting these ghost processes
(for want of a better description).
We have multiple tomcat servers running on the same system, but this
is the only one experiencing the problem, all of the others are
unaffected.
The web apps themselves are pretty standard, servlets, JSPs, web
services, Oracle DB connection and a JMX link to an Apache MQ server
for monitoring purposes. This isn't even a heavily used application.
The web has come up empty for us, we can't find any information on
this fault at all, so would really appreciate any help you may be able
to give us.
Regards
Dave
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
