So, I've been doing thread stacktrace dumps, but what I found doesn't make a lot of sense to me.
After restarting tomcat there were 5 processor threads all blocked on a monitor:
"Ajp13Processor[8085][0]" daemon prio=1 tid=0x0x4f801b28 nid=0x45ae waiting on m
onitor [bd9ff000..bd9ff8ac]
at java.lang.Object.wait(Native Method)
- waiting on <0x446602c0> (a org.apache.ajp.tomcat4.Ajp13Processor)
at java.lang.Object.wait(Object.java:426)
at org.apache.ajp.tomcat4.Ajp13Processor.await(Ajp13Processor.java:305)
- locked <0x446602c0> (a org.apache.ajp.tomcat4.Ajp13Processor)
at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:490)
at java.lang.Thread.run(Thread.java:536)
I presume these are waiting for the scheduler thread to pass them a request.
A while later a bunch of background threads get spawned and I do another dump/trace,
now there are a whole bunch of threads in a runnable state trying to read request packets:
"Ajp13Processor[8085][21]" daemon prio=1 tid=0x0x8289b38 nid=0x70d6 runnable [ba
fff000..bafff8ac]
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:116)
at org.apache.ajp.Ajp13.readN(Ajp13.java:429)
at org.apache.ajp.Ajp13.receive(Ajp13.java:469)
at org.apache.ajp.Ajp13.receiveNextRequest(Ajp13.java:274)
at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:371
)
at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:495)
at java.lang.Thread.run(Thread.java:536)
Despite the name receiveNextRequest, it looks like that method is being called to receive each request including the first. So, the ajp13 processor is sitting blocked waiting for a full ajp13 packet from apache and apache is not sending data and not closing the connection/socket. From the comments in the code it sounds like apache restarts periodically, closing the sockets and unblocking the read calls, at which point these processors return to the pool - so if apache restarts before you reach your threads limit you are ok.
What I don't understand is how apache can be starting up ajp13 connections to tomcat and then not sending a full packet without this causing user visible failures. Shouldn't there be users getting page loads timing out on them? That doesn't seem to be happening.
Does this indicate a fault in apache or is it normal for apache to send partial requests without closing the connection?
Any further guidance?
Thanks,
Lindsay
Glenn Nielsen wrote:
One way to start debugging this type of problem is to tell the java process running Tomcat to do a Thread Stacktrace Dump.kill -QUIT <java processid> Then analyze the stack traces for all threads. Regards, Glenn Lindsay Patten wrote:Hi,
I thought I would take a different tack on my problem with the Ajp13Processor using up all of its threads problem.
Does anyone know if the Ajp13Processor has a timeout when it is looking for a worker to handle a request? If the worker threads were all swapped out and taking a long time to get going and respond, would the main thread start a new background thread instead of waiting? Or does it just have a list of available workers and blocks until the worker it selects responds?
If it is the former that would explain my problem since I have experienced my processes getting swapped out and taking a long time to respond. The question would then become: is a way to specify how long the scheduler should wait before starting a new thread? At least I would have something I could take to my service provider. On the other hand if I knew for sure that the scheduler thread didn't have a timeout on worker threads than I could concentrate on looking elsewhere. Does anyone know? Or know where to look or ask short of reading the source?
Thanks,
Lindsay
Lindsay Patten wrote:
Hi,
I am having a problem where tomcat keeps intermitantly starting new ajp13 processor threads, eventually it reaches the max and starts refusing connections. My hosting provider (linux machine) provides a private version of tomcat for each user with a single apache web server, there are typically a large number of processes on the machine (>4000). My particular tomcat server (v4.0.3) is not being loaded at all but the ajp13processor threads don't seem to not get reused under some circumstances that I don't understand. I have added logging statements in my jsp pages and the pages appear to run to completion. Each jsp page accesses some info from a mysql db and displays it (*Summary) or takes some data from a form and sticks it in the db (Page). The system will often run for several hours and hundreds of requests without starting any threads, but then sometimes, as below, it starts new threads for almost every request for a while. Is there a way to determine if the threads are getting hung up or returned to the pool? If a thread were swapped out and taking a long time to get swapped back in would tomcat start a new thread? The ps command indicates that the processes are there and sleeping. Any help or debugging pointers would be greatly appreciated. Thanks - Lindsay
2002-10-18 03:50:45 Ajp13Processor[8085][14] Starting background thread
2002-10-18 03:50:53 Ajp13Processor[8085][15] Starting background thread
2002-10-18 03:50:59 NationalSummary start
2002-10-18 03:51:00 NationalSummary end
2002-10-18 03:51:10 DistrictSummary start
2002-10-18 03:51:10 DistrictSummary end
2002-10-18 03:51:14 Ajp13Processor[8085][16] Starting background thread
2002-10-18 03:51:14 SubdistrictSummary start 42
2002-10-18 03:51:14 SubdistrictSummary end 42
2002-10-18 03:51:23 Ajp13Processor[8085][17] Starting background thread
2002-10-18 03:51:23 PageFrame start
2002-10-18 03:51:23 PageFrame end
2002-10-18 03:51:24 Ajp13Processor[8085][18] Starting background thread
2002-10-18 03:51:24 Page start 206
2002-10-18 03:51:24 Page end 206
2002-10-18 03:53:32 Ajp13Processor[8085][19] Starting background thread
2002-10-18 03:53:32 Page start 206
2002-10-18 03:53:33 Page end 206
2002-10-18 03:53:56 Ajp13Processor[8085][20] Starting background thread
2002-10-18 03:53:56 NationalSummary start
2002-10-18 03:53:56 NationalSummary end
2002-10-18 03:54:42 Page start 206
2002-10-18 03:54:42 Page end 206
2002-10-18 03:55:06 DistrictSummary start
2002-10-18 03:55:06 DistrictSummary end
2002-10-18 03:55:24 Ajp13Processor[8085][21] Starting background thread
2002-10-18 03:55:24 NationalSummary start
2002-10-18 03:55:24 NationalSummary end
2002-10-18 03:55:34 DistrictSummary start
2002-10-18 03:55:34 DistrictSummary end
2002-10-18 03:55:41 Page start 206
2002-10-18 03:55:41 Page end 206
2002-10-18 03:56:52 Ajp13Processor[8085][22] Starting background thread
2002-10-18 03:56:52 Page start 206
2002-10-18 03:56:52 Page end 206
2002-10-18 03:58:16 Page start 206
2002-10-18 03:58:16 Page end 206
2002-10-18 03:59:31 Page start 206
2002-10-18 03:59:31 Page end 206
2002-10-18 04:00:24 Page start 206
2002-10-18 04:00:24 Page end 206
2002-10-18 04:01:34 Page start 206
2002-10-18 04:01:34 Page end 206
--
To unsubscribe, e-mail: <mailto:tomcat-user-unsubscribe@;jakarta.apache.org>
For additional commands, e-mail: <mailto:tomcat-user-help@;jakarta.apache.org>
--
To unsubscribe, e-mail: <mailto:tomcat-user-unsubscribe@;jakarta.apache.org>
For additional commands, e-mail: <mailto:tomcat-user-help@;jakarta.apache.org>
--
To unsubscribe, e-mail: <mailto:tomcat-user-unsubscribe@;jakarta.apache.org>
For additional commands, e-mail: <mailto:tomcat-user-help@;jakarta.apache.org>
-- To unsubscribe, e-mail: <mailto:tomcat-user-unsubscribe@;jakarta.apache.org> For additional commands, e-mail: <mailto:tomcat-user-help@;jakarta.apache.org>
