Roman Zhovtulya [mailto:[EMAIL PROTECTED] asked:
> > Normally the performance is really good, but about once a day it gets
> > really slow (around 1-5 minutes to display the page). It also recovers
> > autocatically to normal response time when you simply wait
> > 5-10 minutes.
On Mon, Nov 08, 2004 at 11:16:55PM -0000, Steve Kirk wrote:
> My favourite answer to any query about a server being painfully slow: does
> the slowdown coincide with a slowdown or outage of your DNS service? This
> can have suprisingly big impact if TC is doing reverse lookups of the IP of
> every request.
>
> At the risk of asking the obvious, are you sure that some other heavy
> process is not kicking off at that time, either on one of the machines you
> mention, or some other machine on the network?
If it's not external load (as SteveK suggests), nor internal load,
it might be resource contention.
1) how many apache processes do you have?
I had a problem on a redhat box, apparently the default apache install
was 10 processes. Needless to say, as soon as the traffic went up a
bit, all 10 processes were busy and subsequent requests had to wait
for a free process - and wait, and wait, and wait...
Does each page load actually load, or does the browser time out while
trying to get a connection?
If the page takes 5 minutes but actually does load (and if the delay
is not because it's trying to shove a lot of data into the page), then
it's probably *not* a lack of apache processes. But it's easy enough
to check on, so check on it anyway.
2) If it's not contention for apache processes, it might be some sort
of contention inside tomcat - either a thread locking issue somewhere,
or getting at the database, or some other resource.
3) It may not be actual load, but more time delays. For example,
anything in your system that hits an outsid server (message queue,
email) may take a couple of seconds to complete. All of those secodns
add up. They may also lead to resource contention.
One of my earlier servlet projects, the "super high reliability"
message queue (MQ Series - this was pre-JMS) performed as
advertised... but the "super high reliability" big iron it was
connecting to was shut down for a couple hours of scheduled
maintenance every sunday morning at 1am (which, of course, I found out
about one sunday morning at 2am). The message queue's default timeout
was kind of high (25 seconds), so the thread just sat there waiting
for the answer, holding open one of the apache processes all the
while. Soon enough, all the apache processes were busy waiting for
answers from the tomcat threads, which were waiting for answers from
their message queue requests...
The right way to resolve this sort of problem depends a lot on the
circumstances, so look into that and post here with further questions.
4) Just a wild idea, but check on garbage-collection. Maybe put a
couple of logging calls in the pages, so you can see the memory usage
in the log:
Runtime.getRuntime().maxMemory() // maximum memory allocation
Runtime.getRuntime().totalMemory() // memory actually allocated by JVM
Runtime.getRuntime().freeMemory() // allocated but not in use by objects
The gotcha to watch out for is that the system will quite happily
allocate memory without doing garbage-collection, until it hits
maxMemory(). Then it'll do garbage collection, but it may take a
while.
You can't really eliminate the delay, but you can move it around
and/or spread it out. There are command-line arguments to tweak how
garbage collection is done. At the time I learned this lesson, I
don't think those command-line arguments were available, so I used a
call to System.gc() instead. But I'd strongly suggest looking into
them before you go mucking with System.gc().
System.gc() isn't guaranteed to do anything. Whether it does anything
or not depends on your platform, but it's supposed to sort of "nudge"
the garbage collector and say "now might be a good time to clean up."
If you have any sort of consistent spot in your app where a bunch of
objects become superfluous (in the one where I learned this lesson, it
was an object cache that expired objects every minute), putting in a
call to System.gc() might really even out the memory usage.
5) Do some very rough thumbnail profiling; print in
System.currentTimeInMillis() to the log at a number of strategic
points. Then after a slowdown, look at the log, try to trace a single
request through the process and see where the big delay was, then take
a microscope to that portion of the application.
--
Steven J. Owens
[EMAIL PROTECTED]
"I'm going to make broad, sweeping generalizations and strong,
declarative statements, because otherwise I'll be here all night and
this document will be four times longer and much less fun to read.
Take it all with a grain of salt." - http://darksleep.com/notablog
---------------------------------------------------------------------
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
