Hi Tim,
[EMAIL PROTECTED] schrieb:
Setup: TC 4.1.18, IIS5, mod_jk 1.2.25, JDK 1.3.1, Windows 2000
1 LB pointing to 1 worker TC, same server as IIS. Because of 3rd party
license issues, we're stuck with using 1 TC only.
Usually the tomcat runs with ~ 280 threads, but soon after a bounce its
getting 400+ threads, and pretty soon I get errors such as this -
[Error] ThreadPool- - Caught exception executing
org.apache.jk.common.SocketAcceptor... <java.lang.OutOfMemoryError
unable to create new native thread>
OK, usually when you run OutOfMemory during thread creation, this will
also mean, that your Tomcat doesn't accept any new connections any more.
With the old style connector, whenever a new connection comes in, Tomcat
accepts the connection on the thread, that waits for new connections and
lets this thread work on the requests coming in via this connection.
It then looks for another thread to again accept new connections. If the
pool is empty and the maximum number of threads is not reached, it will
try to create a new thread. If there is not enough memory for this,
thread creation will fail and from that on, there will be no more thread
accepting any new connection. This effectvely means, that Tomcat will
still answer the open requests, but no more new requests are handeld -
until Tomcat restart.
That's the reson, why you should always limit the number of threads in
the pool. It's better to not accept new connections, if your threads
reahed the defined limit, than to create to many threads and by this
break the service unrecoverable.
How many threads will work? It depends on the OS. On Windows something
between 400 and 500, on Linux it depends on 32/64 Bits and the thread
library. On Solaris Sparc there is no such trouble. It's mostly a
question of memory adressing for thread stacks.
I would test, if 400 work, and then limit the pool to 400. How do you
test? Simply configure the pool to 400 minimum and maximum and see, if
you can start Tomcat and if all threads get created. Then you know, that
your platform can create that many threads inside the JVM.
So far this is totally independent of mod_jk/isapi plugin.
Shortly after, mod_jk declares the worker to be in an Error state, and
all my site serves is the 'Service Temporarily Unavailable' page. The
isapi_redirect.log is dumping out hundreds of messages including 'all
tomcat instances are busy or in error state'. This is like a couple of
minutes after TC is restarted !
The cat, that you get many such errors, once your Tomcat isn't accepting
any more connections seems normal. The fact, that *all* tomcats are busy
seems normal too, since you only have one.
Why will mod_jk put your Tomcat into error state. although it still has
400 working connections (but cannot open a new one): because that's the
way it's implemented. If we try to open a new connection and we get an
error (e.g. a network timeout), we declare the backend as broken (error
state).
worker.properties has
worker.wkr.connection_pool_size=200
worker.wkr.connection_pool_timeout=300
Hmmm. This would mean, that the web server only opens up max. 200
connections to Tomcat. If we assume, that there is only one web server
in front of Tomcat and no traffic is going to Tomcat from somewhere else
(other connectors etc.), we should not see more than 200 and a few
threads inside Tomcat. This contradicts your observations.
It's possible, that old connections are not closed correctly on the
Tomcat side, if a firewall between web server and Tomcat drops idle
connections. Usually one uses a connectionTimeout on the Tomcat side
together with the connection pool timeout on the mod_jk side.
See:
http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html
- When mod_jk shows a state of ERR/FRC, what exactly does the forced
recovery attempt to do ?
If a balancer has to put all members into error state, it could either
always return an error, or it could say: I've got no good backend, so
lets try to use the bad ones anyhow. Force recovery means: no backend is
OK, then use a bad one. In your case (one backend) it means, that
although the backend goes into error, we still send the requests there.
- Our webapp in TC works hard on startup to cache frequent content into
memory. That means bouncing the Tomcat is painful for us during the day.
Fix an upper thread limit. If your app behaves well, then you shouldn't
have to restart. Of course if the problem is bad performance and your
ressources are exhausted, that problm will only end, if the load
decreases, or the performance gets better. But usually the container
will then proceed as normal, without restart.
In the case described abive, when the container could not start a new
thread for socket acept, it will be broken and a restart is needed.
I wonder anectdotally if the FRC messages from mod_jk are causing it
hassle ? Moreover in the first ~15minutes of the webapp, response times
will be longer than usual. Is this causing mod_jk to give up? If so
whats the best tuning approach?
It will only give up on long response times when a reply_timeout is set.
- Is there a simple way in mod_jk to throttle traffic incident on tomcat
?
No. There are other modules, but most of them throttle traffic per
client IP. Search for httpd modules with "throttle" or "bandwidth". You
could code a servlet filter though. Throttling inside Tomat is less
stable than in front of it, but you can have some easy logic there, whom
you want to send the throttle page (error page), e.g. to many users
logged on (or to many sessions) and the request doesn't belong to a
session. etc. Ooops, sorry, you are on IIS, forgot about that.
- Lastly, can the 'Service Temporarily Unavailable' page be customised ?
I'm presuming that its being served by mod_jk internals rather than IIS
?
Anything inside IIS? It would be ErrorPage with httpd.
regards
Tim
Regards,
Rainer
---------------------------------------------------------------------
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
