Lisa Woodring wrote:
...
> In order to monitor
> the availability of the HTTPS/AJP port (Apache-->Tomcat), our
> monitoring software opens a port to verify that this works -- but then
> does not follow that up with an actual request. This happens every 2
> minutes.
...
This sounds like the perfect recipe for simulating a DOS attack. Your monitoring system
is forcing Tomcat to allocate a thread to process the request which should subsequently
arrive on that connection, yet that request never comes; so basically this thread is
wasted, until the ConnectionTimeout triggers (after 20 seconds, according to your HTTP
connector settings).
...
The thread count grows over time (goes up to 130-150 threads after 2
hours). Setting 'connectionTimeout' (as opposed to the default of
never timing out) does seems to help "some"
Have you tried setting it shorter ? 20000 = 20000 ms = 20 seconds. That is still quite
long if you think about a legitimate browser/application making a connection, and then
sending a request on that connection. Why would it wait so long ? A browser would never
do that : it would open a connection to the server when it needs to send a request, and
then send the request immediately, as soon as the connection is established.
In other words : anything which opens a HTTP connection to your server, and then waits
more than 1 or 2 seconds before sending a request on that connection, is certainly not a
browser.
And it probably is either a program designed to test or attack your server, or else a
badly-designed monitoring system.. ;-)
-- the # of threads isn't
quite as bad (only 60-80 threads after 2 hours). However, the CPU
Idle % is still not good -- was only 10% idle with default tomcat
settings, is something like 40% idle with current settings. Also
tried setting Apache's 'KeepAliveTimeout = 5' (currently set to 15)
but this did not make any difference.
Note : this value is in milliseconds. setting it to 5 or 15 is almost equivalent to
disabling keep-alive altogether. 3000 may be a reasonable value.
KeepAlive only happens after at least one request has been received and processed, waiting
for another (possible) request on the same connection. If there is never any request sent
on that connection, then it would not apply here, and only the connectionTimeout would apply.
Note that my comments above are relative to your HTTP Connector.
For the AJP Connector, other circumstances apply.
If you are using AJP, it implies that there is a front-end server, using a module such as
mod_jk or mod_proxy_ajp to connect to Tomcat's AJP Connector.
In that case, you should probably leave Tomcat's connectionTimeout to its default value,
and let the front-end server handle such things as the connection timeout and the
keep-alive timeout. The connector module on the front-end server will manage these
connections to Tomcat, and it may pre-allocate some connections, to constitute a pool of
available connections for when it actually does need to send a request to Tomcat over one
such connection. Timing out these connections at the Tomcat level may thus be
contra-productive, forcing the front-end to re-create them constantly.
Is there some configuration we can set to make Tomcat tolerant of this
monitoring? (We have tried setting connectionTimeout &
keepAliveTimeout on the Connector. And we have tried putting the
Connector behind an Executor with maxIdleTime.)
OR, should we modify our monitoring somehow? And if so, suggestions?
I would think so. Have your monitoring send an actual request to Tomcat (and read the
response); even a request that results in an error would probably be better than no
request at all. But better would be to request something real but small, which at the
Tomcat level would be efficient to respond to (e.g. not a 5 MB image file).
Create a little webapp which just responds "I'm fine" (*), and check that response in your
monitor. It will tell you not only that Tomcat has opened the port, but also that Tomcat
webapps are actually working (and how quickly it answers).
And do not try to monitor the AJP port directly. Monitor a request to the front-end, which
should arrive to Tomcat via the AJP port. The AJP connector module on the front-end will
respond with its own error, if it cannot talk to Tomcat.
(*) actually, there may even exist some built-in mechanism in Tomcat, designed precisely
for such kind of usage (or at least usable for it).
Any of the experts on the list ? does the standard vanilla Tomcat offer some URL which can
be called, and triggers some small efficient response readable by a monitoring program ?
...
* Running on Linux CentOS release 5.9
* running Apache in front of Tomcat for authentication, using mod_jk
* Tomcat 8.0.14
relevant sections of tomcat/conf/server.xml:
------------------------------------------------------------------------
<Executor name="tomcatThreadPool" namePrefix="catalina-exec-"
maxThreads="250" minSpareThreads="20" maxIdleTime="60000" />
<Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1"
connectionTimeout="20000" redirectPort="8443" />
<Connector executor="tomcatThreadPool" port="8009" protocol="AJP/1.3"
redirectPort="8443" maxThreads="256"
connectionTimeout="3000" keepAliveTimeout="60000" />
----------------------------------------------------------------------------
If interested, I can provide graphing of the machine's thread count,
cpu idle%, and cpu load.
Any suggestions would be most welcome.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
