Re: How ot monitor hung tomcat/apache processes?
On 09/22/2010 03:42 PM, Darryl Lewis wrote:
Are you trying to monitor from the same computer that tomcat is running on?
That's not a good idea. What happens if the entire system crashes...you won't
get any data/alerts. End to end uses another machine to monitor the first.
You could monitor catalina.out for errors, but I don't think you'll capture
every possibility.
What would happen if the system stops serving pages (or serves them slow)? That
won't show in the logs.
What port is your application running on? You can use (from another machine)
wget {servername}:8080.
Well, on the other hand, what happens, if the machine doing the
monitoring crashes? Again, you're out of alerts.
So, monitoring is something you build in layers. And have one point of
your monitoring infrastructure be something that is seen if it goes
down. Or alternatively, have some weird setup where two (or more)
things are monitoring each other.
As for end-to-end monitoring, it's nice for response time statistics,
but if your infrastructure is anything beyond simple, you'll want to
monitor distinct components. Consider a situation where authentication
is done by some kind of front-end proxy. You have end-to-end monitoring
in place, so all your requests go through the front-end. Now, monitoring
shows an error. Is the error with the front-end or with the application
server? Or perhaps in the network between the two? Or in the network
between the monitoring server and the front-end? Add another layer
(database?), and you'll agani have more possibilities what could be
wrong -- and more or less, with end-to-end monitoring you'll only get a
notification that something is wrong with this page.
Further, with end-to-end monitoring combined with redundant systems, the
alert will generate when the last piece of redundancy is gone - i.e.
when the whole redundant service is down, not when it gets degraded
(which would be the correct point to react).
So, something running on your Tomcat machine and doing requests for
HEAD / would be sufficient to see that the Tomcat itself is not
completely hung. When you make such a program simple enough (but at the
same time foolproof enough), and set it to run regularly using the
operating system task scheduler (cron in Unix-like systems), you'll have
a good point solution that tells you exactly that the Tomcat is not
working.
As to finding out what is wrong with a non-working Tomcat, I've been
pretty successful in generating thread dumps from hung Tomcat java
processes, and seeing from them what has been executing, and also rather
often what has caused the hang. For example, some versions of Oracle
JDBC drivers.. .
--
..Juha
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: How ot monitor hung tomcat/apache processes?
In my experience, the PID can still exist of tomcat but a Java heap crash has stopped it responding. Checking a PID will not check if the application is responding. You're better to do some sort of end to end monitoring On 22/09/10 10:03 PM, Mendiratta, Shashank shashank_mendira...@intuit.com wrote: Hi , I am working on a monitoring system to find out hung tomcat/apache processes . By this I mean if the PID exists and still the apache / tomcat is not responding that die to memory leak or variety of other reasons . Is their a tool to find this . Regards Shashank
RE: How ot monitor hung tomcat/apache processes?
Hi Darryl, Yes This is the same problem I am facing. Sorry I am kind of new to it but can you tell me what kind of end to end monitoring should I do ? Regards Shashank -Original Message- From: Darryl Lewis [mailto:darryl.le...@unsw.edu.au] Sent: Wednesday, September 22, 2010 5:38 PM To: Tomcat Users List Subject: Re: How ot monitor hung tomcat/apache processes? In my experience, the PID can still exist of tomcat but a Java heap crash has stopped it responding. Checking a PID will not check if the application is responding. You're better to do some sort of end to end monitoring On 22/09/10 10:03 PM, Mendiratta, Shashank shashank_mendira...@intuit.com wrote: Hi , I am working on a monitoring system to find out hung tomcat/apache processes . By this I mean if the PID exists and still the apache / tomcat is not responding that due to memory leak or variety of other reasons . Is their a tool to find this . Regards Shashank - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: How ot monitor hung tomcat/apache processes?
It depends on the application you are running, but a simple test would be to access the webpage (ensuring part of it is served from Tomcat, not apache) and check for an expected response. For example, a simple jsp page that prints out ok You can then do a wget, and check for that string. Cheers. On 22/09/10 10:13 PM, Mendiratta, Shashank shashank_mendira...@intuit.com wrote: Hi Darryl, Yes This is the same problem I am facing. Sorry I am kind of new to it but can you tell me what kind of end to end monitoring should I do ? Regards Shashank -Original Message- From: Darryl Lewis [mailto:darryl.le...@unsw.edu.au] Sent: Wednesday, September 22, 2010 5:38 PM To: Tomcat Users List Subject: Re: How ot monitor hung tomcat/apache processes? In my experience, the PID can still exist of tomcat but a Java heap crash has stopped it responding. Checking a PID will not check if the application is responding. You're better to do some sort of end to end monitoring On 22/09/10 10:03 PM, Mendiratta, Shashank shashank_mendira...@intuit.com wrote: Hi , I am working on a monitoring system to find out hung tomcat/apache processes . By this I mean if the PID exists and still the apache / tomcat is not responding that due to memory leak or variety of other reasons . Is their a tool to find this . Regards Shashank
RE: How ot monitor hung tomcat/apache processes?
HI Darryl , Thanx , about that here the outbound port 80 is blocked so we cannot wget , moreover this wont solve the problem as to why the the services are getting hung. Well I had an idea, please critic it. Why not monitor the server.log file if we get some kind of error. We send an alert and then restart the service . Befire that we have to make a repository of types of error that can occur Please do comment Regards Shashank From: Darryl Lewis [mailto:darryl.le...@unsw.edu.au] Sent: Wednesday, September 22, 2010 5:54 PM To: Mendiratta, Shashank; Tomcat Users List Subject: Re: How ot monitor hung tomcat/apache processes? It depends on the application you are running, but a simple test would be to access the webpage (ensuring part of it is served from Tomcat, not apache) and check for an expected response. For example, a simple jsp page that prints out ok You can then do a wget, and check for that string. Cheers. On 22/09/10 10:13 PM, Mendiratta, Shashank shashank_mendira...@intuit.com wrote: Hi Darryl, Yes This is the same problem I am facing. Sorry I am kind of new to it but can you tell me what kind of end to end monitoring should I do ? Regards Shashank -Original Message- From: Darryl Lewis [mailto:darryl.le...@unsw.edu.au] Sent: Wednesday, September 22, 2010 5:38 PM To: Tomcat Users List Subject: Re: How ot monitor hung tomcat/apache processes? In my experience, the PID can still exist of tomcat but a Java heap crash has stopped it responding. Checking a PID will not check if the application is responding. You're better to do some sort of end to end monitoring On 22/09/10 10:03 PM, Mendiratta, Shashank shashank_mendira...@intuit.com wrote: Hi , I am working on a monitoring system to find out hung tomcat/apache processes . By this I mean if the PID exists and still the apache / tomcat is not responding that due to memory leak or variety of other reasons . Is their a tool to find this . Regards Shashank
Re: How ot monitor hung tomcat/apache processes?
Are you trying to monitor from the same computer that tomcat is running on?
That's not a good idea. What happens if the entire system crashes...you won't
get any data/alerts. End to end uses another machine to monitor the first.
You could monitor catalina.out for errors, but I don't think you'll capture
every possibility.
What would happen if the system stops serving pages (or serves them slow)? That
won't show in the logs.
What port is your application running on? You can use (from another machine)
wget {servername}:8080.
On 22/09/10 10:30 PM, Mendiratta, Shashank shashank_mendira...@intuit.com
wrote:
HI Darryl ,
Thanx , about that here the outbound port 80 is blocked so we cannot wget ,
moreover this wont solve the problem as to why the the services are getting
hung.
Well I had an idea, please critic it. Why not monitor the server.log file if we
get some kind of error. We send an alert and then restart the service . Befire
that we have to make a repository of types of error that can occur
Please do comment
Regards
Shashank
From: Darryl Lewis [mailto:darryl.le...@unsw.edu.au]
Sent: Wednesday, September 22, 2010 5:54 PM
To: Mendiratta, Shashank; Tomcat Users List
Subject: Re: How ot monitor hung tomcat/apache processes?
It depends on the application you are running, but a simple test would be to
access the webpage (ensuring part of it is served from Tomcat, not apache) and
check for an expected response.
For example, a simple jsp page that prints out ok
You can then do a wget, and check for that string.
Cheers.
On 22/09/10 10:13 PM, Mendiratta, Shashank shashank_mendira...@intuit.com
wrote:
Hi Darryl,
Yes This is the same problem I am facing. Sorry I am kind of new to it
but can you tell me what kind of end to end monitoring should I do ?
Regards
Shashank
-Original Message-
From: Darryl Lewis [mailto:darryl.le...@unsw.edu.au]
Sent: Wednesday, September 22, 2010 5:38 PM
To: Tomcat Users List
Subject: Re: How ot monitor hung tomcat/apache processes?
In my experience, the PID can still exist of tomcat but a Java heap
crash has stopped it responding.
Checking a PID will not check if the application is responding.
You're better to do some sort of end to end monitoring
On 22/09/10 10:03 PM, Mendiratta, Shashank
shashank_mendira...@intuit.com wrote:
Hi ,
I am working on a monitoring system to find out hung tomcat/apache
processes .
By this I mean if the PID exists and still the apache / tomcat is not
responding that due to memory leak or variety of other reasons . Is
their a tool to find this .
Regards
Shashank
RE: How ot monitor hung tomcat/apache processes?
I get you point . well we are using dynatrace to keep a track on how
slow the server is serving pages and for the entire system metrics
monitoring we are using monit and both are done externally . The
firewall has closed all the outbound ports so wget is not possible.
Its just that we have to monitor the JVM so that it does not hang but
yes you are right with dynatrace just has a monitoring window and it
does not restart the service. So I have to come up with a cript also
which how the system is serving pages if its to slow . then to restart
the JVM .
From: Darryl Lewis [mailto:darryl.le...@unsw.edu.au]
Sent: Wednesday, September 22, 2010 6:12 PM
To: Mendiratta, Shashank; Tomcat Users List
Subject: Re: How ot monitor hung tomcat/apache processes?
Are you trying to monitor from the same computer that tomcat is running
on? That's not a good idea. What happens if the entire system
crashes...you won't get any data/alerts. End to end uses another machine
to monitor the first.
You could monitor catalina.out for errors, but I don't think you'll
capture every possibility.
What would happen if the system stops serving pages (or serves them
slow)? That won't show in the logs.
What port is your application running on? You can use (from another
machine) wget {servername}:8080.
On 22/09/10 10:30 PM, Mendiratta, Shashank
shashank_mendira...@intuit.com wrote:
HI Darryl ,
Thanx , about that here the outbound port 80 is blocked so we cannot
wget , moreover this wont solve the problem as to why the the services
are getting hung.
Well I had an idea, please critic it. Why not monitor the server.log
file if we get some kind of error. We send an alert and then restart the
service . Befire that we have to make a repository of types of error
that can occur
Please do comment
Regards
Shashank
From: Darryl Lewis [mailto:darryl.le...@unsw.edu.au]
Sent: Wednesday, September 22, 2010 5:54 PM
To: Mendiratta, Shashank; Tomcat Users List
Subject: Re: How ot monitor hung tomcat/apache processes?
It depends on the application you are running, but a simple test would
be to access the webpage (ensuring part of it is served from Tomcat, not
apache) and check for an expected response.
For example, a simple jsp page that prints out ok
You can then do a wget, and check for that string.
Cheers.
On 22/09/10 10:13 PM, Mendiratta, Shashank
shashank_mendira...@intuit.com wrote:
Hi Darryl,
Yes This is the same problem I am facing. Sorry I am kind of new to it
but can you tell me what kind of end to end monitoring should I do ?
Regards
Shashank
-Original Message-
From: Darryl Lewis [mailto:darryl.le...@unsw.edu.au]
Sent: Wednesday, September 22, 2010 5:38 PM
To: Tomcat Users List
Subject: Re: How ot monitor hung tomcat/apache processes?
In my experience, the PID can still exist of tomcat but a Java heap
crash has stopped it responding.
Checking a PID will not check if the application is responding.
You're better to do some sort of end to end monitoring
On 22/09/10 10:03 PM, Mendiratta, Shashank
shashank_mendira...@intuit.com wrote:
Hi ,
I am working on a monitoring system to find out hung tomcat/apache
processes .
By this I mean if the PID exists and still the apache / tomcat is not
responding that due to memory leak or variety of other reasons . Is
their a tool to find this .
Regards
Shashank
Re: How ot monitor hung tomcat/apache processes?
On 22/09/2010 13:08, Darryl Lewis wrote: In my experience, the PID can still exist of tomcat but a Java heap crash has stopped it responding. Checking a PID will not check if the application is responding. Obviously, I exist whether Tomcat is running or not. p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: How ot monitor hung tomcat/apache processes?
On 22/09/2010 13:54, Mendiratta, Shashank wrote: So I have to come up with a cript also which how the system is serving pages if its to slow . then to restart the JVM . Really? That sounds, well, a little drastic... p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: How ot monitor hung tomcat/apache processes?
On 22/09/2010 13:03, Mendiratta, Shashank wrote: Hi , I am working on a monitoring system to find out hung tomcat/apache processes. Instead of looking for problems so you can restart the JVM, trying looking at the actual problem and fixing it, then you won't have to invent crazy solutions. By this I mean if the PID exists and still the apache / tomcat is not responding that die to memory leak or variety of other reasons . The PID may still exist, even if the process is dead. Is their a tool to find this . There are lots of tools. It depends on whether you want to pay for one or if you're trying to get this for free. p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
RE: How ot monitor hung tomcat/apache processes?
Hi .. can you share the script ?? -Original Message- From: Pid [mailto:p...@pidster.com] Sent: Wednesday, September 22, 2010 9:52 PM To: Tomcat Users List Subject: Re: How ot monitor hung tomcat/apache processes? On 22/09/2010 13:54, Mendiratta, Shashank wrote: So I have to come up with a cript also which how the system is serving pages if its to slow . then to restart the JVM . Really? That sounds, well, a little drastic... p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: How ot monitor hung tomcat/apache processes?
On 22/09/2010 17:29, Mendiratta, Shashank wrote: Hi .. can you share the script ?? Which script? You wrote it, according to the below: On 22/09/2010 13:54, Mendiratta, Shashank wrote: So I have to come up with a cript also which how the system is serving pages if its to slow . then to restart the JVM . ? p 0x62590808.asc Description: application/pgp-keys signature.asc Description: OpenPGP digital signature
Re: How ot monitor hung tomcat/apache processes?
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Shashank,
On 9/22/2010 8:30 AM, Mendiratta, Shashank wrote:
Thanx , about that here the outbound port 80 is blocked so we cannot
wget , moreover this wont solve the problem as to why the the services
are getting hung.
Hmm. Can you monitor from the server itself? That's not unusual to do.
Also, connections to localhost:80 usually work even when software-based
firewalls are in place, since the local host is usually considered trusted.
Well I had an idea, please critic it. Why not monitor the server.log
file if we get some kind of error. We send an alert and then restart the
service . Befire that we have to make a repository of types of error
that can occur
We have one particularly poorly-written webapp that has a habit of
running out of memory. We have segregated it into it's own Tomcat
instance and actually do scan the log file for errors in the way you
describe.
The script is essentially this:
grep -m 1 OutOfMemoryError ${LOGFILE} /dev/null
if [ $? == 0 ] ; then
# notify an administrator
fi
It's not particularly elegant, but it gets the job done.
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkyaS0wACgkQ9CaO5/Lv0PCxXQCgwIlct+hqxxejBAEUAPw8+gXj
EiAAoImkWA55dP3Nw8iuWIqM2P/N7Hvk
=avt1
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: How ot monitor hung tomcat/apache processes?
Hi,
You can try several approaches (I'll list 2 that I'm aware of):
1) Automatic restarts on OutOfMemory errors:
Add the following to CATALINA_OPTS:
-XX:OnOutOfMemoryError=/usr/sbin/restart_tcserver
Write your restart_tcserver (you may send an e-mail notification from it
etc.)
2) This is what I do (please critisice/suggest improvements to this
approach):
I've got 2 servers with Tomcat+Apache httpd with heartbeat beetween them:
I'm running this little script every 15 min. via cron:
# cat /srv/scripts/test_live.sh
#!/bin/bash
SERVICE_HTTPD=$(ps -ef | grep -v grep | grep -c httpd)
SERVICE_TOMCAT=$(ps -ef | grep -v grep | grep -c tomcat)
SERVICE_HEARTBEAT=$(ps -ef | grep -v grep | grep -c heartbeat)
SERVICE_STATUS=$(/srv/scripts/check_http.pl -H confluence-server.myorg.com
-u /blank.html)
# While testing, please uncomment the following echo statements
if [ $SERVICE_HTTPD -ne 0 -a $SERVICE_TOMCAT -ne 0 -a $SERVICE_STATUS =
Status: OK ]
then
# echo SERVICE_HTTPD and SERVICE_TOMCAT and SERVICE_STATUS
are OK, everything is fine
exit
elif [ $SERVICE_HEARTBEAT -ne 0 ]
then
echo The following output triggered failover:
SERVICE_HTTPD=$SERVICE_HTTPD , SERVICE_TOMCAT=$SERVICE_TOMCAT ,
SERVICE_STATUS=$SERVICE_STATUS , failing over to spare server
echo The following output triggered failover:
SERVICE_HTTPD=$SERVICE_HTTPD , SERVICE_TOMCAT=$SERVICE_TOMCAT ,
SERVICE_STATUS=$SERVICE_STATUS , failing over to a spare server at `date` |
/bin/mailx -s Server `uname -n` encountered a problem, failing over to a
spare server at `date` lkolchin at gmail dot com
/etc/init.d/heartbeat stop
else
# echo This server probably failed over to the spare one,
nothing to do
exit
fi
-
If Tomcat+Apache running and application responsive ($SERVICE_STATUS) do
nothing if at least one of those conditions is not true, failover to a spare
server.
check_http.pl - This is a perl script (from Nagios Plugin I believe)-
## check_http.pl
## Copyright (c) 2008, Oliver Wittenburg oli...@wiburg.de
##
## This program is free software: you can redistribute it and/or modify it
under
## the terms of the GNU General Public License as published by the Free
Software
...
Cheers,
Leon Kolchinsky
On Thu, Sep 23, 2010 at 04:30, Christopher Schultz
ch...@christopherschultz.net wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Shashank,
On 9/22/2010 8:30 AM, Mendiratta, Shashank wrote:
Thanx , about that here the outbound port 80 is blocked so we cannot
wget , moreover this wont solve the problem as to why the the services
are getting hung.
Hmm. Can you monitor from the server itself? That's not unusual to do.
Also, connections to localhost:80 usually work even when software-based
firewalls are in place, since the local host is usually considered trusted.
Well I had an idea, please critic it. Why not monitor the server.log
file if we get some kind of error. We send an alert and then restart the
service . Befire that we have to make a repository of types of error
that can occur
We have one particularly poorly-written webapp that has a habit of
running out of memory. We have segregated it into it's own Tomcat
instance and actually do scan the log file for errors in the way you
describe.
The script is essentially this:
grep -m 1 OutOfMemoryError ${LOGFILE} /dev/null
if [ $? == 0 ] ; then
# notify an administrator
fi
It's not particularly elegant, but it gets the job done.
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkyaS0wACgkQ9CaO5/Lv0PCxXQCgwIlct+hqxxejBAEUAPw8+gXj
EiAAoImkWA55dP3Nw8iuWIqM2P/N7Hvk
=avt1
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
