DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-12-11 15:23 --- The bug reporting system is for reporting bugs in the Tomcat code itself, not for resolving configuration problems. Please ask this question on the [EMAIL PROTECTED] email list. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-12-10 15:18 --- Glenn I was wondering, what is your hardware configuration, and what settings did you use for -Xmx? Also, did you run incgc? Thanks, Jim -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-04-29 20:03 --- FYI - I was experiencing a similar problem with Tomcat 4.0.3 on Solaris 8, where if there were more simultaneous connections to the server than specified in maxProcessors, instead of queueing the connections, there were connection timeout messages and blank HTML pages being returned to the client. A workaround was to increase maxProcessors to something like 320 - that allowed me to have 300 simultaneous requests/connections. But I tried Coyote as suggested, and with maxProcessors set to only 15, I was able to get 300+ simultaneous connections without error. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-04-29 20:13 --- I have a question: which version of Coyote are you using ? I'm asking that because only the recent versions are using the TC 3.3 TP code. So if you're using one of the older versions which have the older thread pool, it would indicate that the problem is with GC (Coyote greatly reduces GC). -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG
RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735.
ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND
INSERTED IN THE BUG DATABASE.
http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735
HTTP connector running out of processors under heavy load
--- Additional Comments From [EMAIL PROTECTED] 2002-04-24 17:04 ---
I would recommend that you dump the stack for all running threads when
you experience this problem. This can help identify what is causing the
problem. By reviewing the stack dump for each thread you can determine
whether the problem is due to Tomcat or your application.
On unix you do a kill -QUIT {tomcat java pid) to cause the thread stack's
to be dumped to catalina .out.
A Processor for Tomcat runs your application code, delays in your code
can cause additional processor threads to be created to handle new requests.
Possible application or configuration problems which can delay requests:
Connection delays due to networked services such as a db.
Connection delays due to running out of pooled resources.
Thread synchronization deadlocks.
A cascading affect where many new processors get created due to
excessively long JVM Garbage Collections. start java with
-verbose:gc to detect this.
--
To unsubscribe, e-mail: mailto:[EMAIL PROTECTED]
For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-04-19 10:35 --- Coyote 1.0 Beta 7 or later now uses different thread pooling code (from Tomcat 3.3). Could you test using it ? If you're not experiencing the bug anymore, then it is likely the problem is with Tomcat 4 TP code. http://jakarta.apache.org/builds/jakarta-tomcat-connectors/coyote/release/v1.0- b8/ To configure Coyote with SSL, use: Connector className=org.apache.coyote.tomcat4.CoyoteConnector port=443 minProcessors=5 maxProcessors=75 enableLookups=true acceptCount=10 debug=0 scheme=https secure=true useURIValidationHack=false Factory className=org.apache.coyote.tomcat4.CoyoteServerSocketFactory clientAuth=false protocol=TLS / /Connector Note that the factory className is different. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load [EMAIL PROTECTED] changed: What|Removed |Added CC||[EMAIL PROTECTED] Status|RESOLVED|REOPENED Resolution|WORKSFORME | --- Additional Comments From [EMAIL PROTECTED] 2002-04-17 18:50 --- I can confirm that we are definately getting this same problem in our production environment. This is a major and urgent issue for us, our business is riding on it. We are using: Tomcat 4.0.3 Solaris 2.7 Heap 128 min, 256 max JSK 1.3.0.2 Thread pool min 5, 255 max We actually run our of heap at around 140 threads, but are seeing the same pattern. - Things are fine for a while - At a seemingly randon amount of time (usualy several hours) after restart, the Catalina log shows thread creation going crazy: 2002-04-17 01:54:03 HttpProcessor[443][11] Starting background thread 2002-04-17 04:51:02 HttpProcessor[443][12] Starting background thread 2002-04-17 04:52:45 HttpProcessor[443][13] Starting background thread 2002-04-17 04:52:45 HttpProcessor[443][14] Starting background thread 2002-04-17 04:52:45 HttpProcessor[443][15] Starting background thread 2002-04-17 04:52:45 HttpProcessor[443][16] Starting background thread 2002-04-17 04:52:45 HttpProcessor[443][17] Starting background thread 2002-04-17 04:52:45 HttpProcessor[443][18] Starting background thread 2002-04-17 04:52:45 HttpProcessor[443][19] Starting background thread 2002-04-17 04:52:45 HttpProcessor[443][20] Starting background thread 2002-04-17 04:52:46 HttpProcessor[443][21] Starting background thread .. .. .. .. 2002-04-17 04:56:43 HttpProcessor[443][127] Starting background thread 2002-04-17 04:56:43 HttpProcessor[443][128] Starting background thread 2002-04-17 04:56:43 HttpProcessor[443][129] Starting background thread 2002-04-17 06:20:20 HttpConnector Opening server socket on all host IP addresses 2002-04-17 06:20:20 HttpConnector Opening server socket on all host IP addresses The gap in the log at 4:56 is the server getting an out-of-memory exception and becoming unresponsive to further resquests (including shutdown). We have to kill manually and restart. During this same time people, we see a huge jump in the amount of memory being used by the JVM for this process. I guess that's why we run out of heap, before we can run out of threads. Sorry for the lack of appropriate detail, I am not the sys admin. Please let me know what better information we can provide. This is happening daily for us, in our (24x7) prod system -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-04-17 20:32 --- Did you read the comments above ? If you get the OutOfMemory first (before the thread problems), it could be that the server runs out of memory because of too many active sessions (or something like that). -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-04-17 21:43 --- Yes, I read the entire thread. We keep an eye on the number of active sessions through the manager/list command are these are by no means extreme. In fact this problem tends *not* to happen during our periods of highest load, where we see our active session count at it's max. It seems to be in the morning when a bunch of people log in at once. We are tracing process memory usage and see it double at the same time as the rush of new threads in the pool: 04-17-02:04:30:00 VSZ RSS 194208 169088 04-17-02:04:40:00 VSZ RSS 194272 169160 04-17-02:04:50:00 VSZ RSS 194272 169160 04-17-02:05:00:00 VSZ RSS 342440 315240 04-17-02:05:10:00 VSZ RSS 342080 314880 04-17-02:05:20:00 VSZ RSS 342080 314880 VSZ=Virtual set size RSS=Resident set size Our problem (this morning) occured at 04:55. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-04-09 21:14 --- Created an attachment (id=1511) server.xml -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load [EMAIL PROTECTED] changed: What|Removed |Added Status|RESOLVED|REOPENED Resolution|WORKSFORME | --- Additional Comments From [EMAIL PROTECTED] 2002-04-09 21:16 --- I am experiencing this problem on a regular basis. The specifics of my setup: linux (kernel 2.2.19) glibc 2.1.3 j2sdk1.4.0 Tomcat 4.0.3 Max Heap: 512MB Thread Stack: 512kb Physical RAM: 1GB -server only runs tomcat Tomcat is running on port 80 with the HTTP 1.1 connector only. No other connectors are configured. I have attached a copy of my server.xml to this bug report. The specific problem I see, is that upon launching of tomcat, everything works fine and all dynamic content is served properly. Then after a seemingly random amount of time, I begin to get the following in my catalina log file: 2002-04-09 19:23:02 HttpProcessor[80][80] Starting background thread 2002-04-09 19:23:03 HttpProcessor[80][81] Starting background thread 2002-04-09 19:23:05 HttpProcessor[80][82] Starting background thread etc up to 400 (my max processors) Once I get to 400 processors I start getting the following error for all additional requests: 2002-04-09 19:42:05 HttpConnector[80] No processor available, rejecting this con nection Manual review of the log file reveals no other information. However, I can confirm that for each new request sent to the server, a new background thread is created. So it seems that the existing threads aren't reused or killed, and new ones are created which eventually hit the maximum. I'm in desperate shape, please help. To correct the problem, I basically must restart the server. The error occurs about every 2-3 hours on my system. It mostly seems to happen during the day which leads me to believe it is load related. When the error occurs the system has plenty of resources available both CPU and RAM. Also, it is worth noting that I can't do a graceful shutdown of tomcat when this error occurs. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load [EMAIL PROTECTED] changed: What|Removed |Added Status|REOPENED|RESOLVED Resolution||WORKSFORME --- Additional Comments From [EMAIL PROTECTED] 2002-04-09 21:32 --- As this is not reproducable for me, I'm afraid I can't help much. It's also only the second time I get that particular report. Hopefully the upcoming Coyote release will fix it for you. As Glenn pointed out in his analysis, it could also be a problem with the GC (and then, after there are too many threads, the VM is sort of messed up). You should lower the number of maxProcessors; 400 is a lot more than what the system can handle, so it wouldn't help (150 looks more reasonable). If you're having problems with the standalone processor because the traffic is too high, maybe you also could give mod_jk a try. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load [EMAIL PROTECTED] changed: What|Removed |Added Status|NEW |RESOLVED Resolution||WORKSFORME --- Additional Comments From [EMAIL PROTECTED] 2002-02-28 01:00 --- Since no one has been able to reproduce it for the HTTP connector (and the thread pooling code), I'll mark this bug as worksforme. I've added additional code which could make the connector more robust (handling of unexpected exceptions and errors), but I now highly doubt there's any problem with the threading code itself. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-02-20 19:22 --- I've just committed a fix for a problem which could cause this bug with the HTTP/1.1 connector. Could you test again with the latest code (form HEAD) ? -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-02-20 21:51 --- The problem I was having with the Ajp connector looks like it is resolved. The java -Xmx max memory usage was set too high. With memory required by other processes on the system, the JVM would hit a memory usage threshhold below the -Xmx setting where it would have to start using swap. Using swap does not make for efficient GC. To debug GC problems start the JVM with the -verbose:gc arg. This doesn't add alot of overhead, or too much logging. But does give you alot of valuable information. The problem I had looks like it is resolved, although I am a bit embarressed that I didn't find it earlier. Thanks to all those who assisted. Now back to the original reported bug with HTTP processors. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-02-21 03:01 --- After further review of the Tomcat 4 logs, here is another indicator and/or impact of GC delays. If you see the Ajp13Proccessor throw IOExceptions for Broken Pipe (Unix) or Socket Write Error (Windows) this could be caused by GC delays. While running Tomcat with -verbose:gc I noticed that GC delays 1 second could cause the above exception to be thrown. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-02-19 04:42 --- I think we are getting close to identifying how this bug behaves. Everytime you get the following Exception: java.io.IOException: Broken pipe at java.net.SocketOutputStream.socketWrite(Native Method) at java.net.SocketOutputStream.write(SocketOutputStream.java:96) at org.apache.ajp.Ajp13.send(Ajp13.java:525) at org.apache.ajp.RequestHandler.finish(RequestHandler.java:496) at org.apache.ajp.Ajp13.finish(Ajp13.java:395) at org.apache.ajp.tomcat4.Ajp13Response.finishResponse(Ajp13Response.java:192) at org.apache.ajp.tomcat4.Ajp13Processor.process(Ajp13Processor.java:453) at org.apache.ajp.tomcat4.Ajp13Processor.run(Ajp13Processor.java:540) at java.lang.Thread.run(Thread.java:484) Ajp13Connection spawns a new Ajp13Processor, if you have reached maxProccessors, a connection is rejected. In a long test run I am doing I see this maybe once an hour. A bunch of the above exceptions, with an identical number of attempts to spawn a new Ajp13Processor. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-02-19 06:21 --- Interesting, but looking at the code, I don't see any problem it would cause (the exception gets caught right away, and the processor should get recycled normally; I don't see any side effect or problems it would cause). -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-02-17 15:57 --- it's not surprising that the ajp connector/processors are exhibiting the same behaviour as the http connector/processors, considering that the ajp versions were created by copying, then modifying the http versions :) -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-02-17 23:36 --- I did a closer review of the apache access_log and the tomcat 4 catalina_log. When the Ajp13Connector starts spawning additional Ajp13Proccessors all of a sudden there is a 1:1 correlation between apache requests which get forwarded to tomcat and the startup of Ajp13Processors. Prior to the the runaway creation of Ajp13Processors catalina had run 2 hours without creating any new ones. So it looks like some bug causes existing processors to no longer be available for use, and any new ones created to only be used once. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-02-18 00:30 --- Based on the apache logs it looks like when the Ajp13Processor's become unavailable for reuse they have successfully handled a request. I can tell this due to entries existing in the apache access_log for image files, etc. related to a request forwarded to Tomcat via mod_jk being requested. So the problem which is causing the Ajp13Processor's to hang must be in Ajp13Proccessor code after a request has been handled and before it calls connector.recycle(this) to make itself available to process another Ajp request. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
Re: DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
Looking at o.a.ajp.tc4.Ajp13Connector, I can see that curProcessors is never decremented, which is probably what is causing this. Unlike in the 3.3 branch, Ajp13Processor is fanatical about not dying. I haven't looked at the C code side, but if it is anything like the 3.3 branch, it will close the connection to TC when the user hits the stop button, but in the j-t-c branch the Processor won't die. Before anyone's feathers get ruffled, my comparisons to 3.3 are simple because that is the code-base I know (I've spent a great deal of time fixing variants of 4518 on 3.3, and I still can't figure out how to fix it on Catalina ;). - Original Message - From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Sunday, February 17, 2002 3:36 PM Subject: DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-02-17 23:36 --- I did a closer review of the apache access_log and the tomcat 4 catalina_log. When the Ajp13Connector starts spawning additional Ajp13Proccessors all of a sudden there is a 1:1 correlation between apache requests which get forwarded to tomcat and the startup of Ajp13Processors. Prior to the the runaway creation of Ajp13Processors catalina had run 2 hours without creating any new ones. So it looks like some bug causes existing processors to no longer be available for use, and any new ones created to only be used once. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
--- Additional Comments From [EMAIL PROTECTED] 2002-02-18 00:30 --- Based on the apache logs it looks like when the Ajp13Processor's become unavailable for reuse they have successfully handled a request. I can tell this due to entries existing in the apache access_log for image files, etc. related to a request forwarded to Tomcat via mod_jk being requested. So the problem which is causing the Ajp13Processor's to hang must be in Ajp13Proccessor code after a request has been handled and before it calls connector.recycle(this) to make itself available to process another Ajp request. that would be somewhere in Ajp13Processor.process()... if you can reproduce this consistently, can you turn up the logging for the ajp13 connector, then send me (seguin at apache.org) the all of the pertinent log files and the captured stdout/stderr - unfortunately, some of the ajp stuff logs to stdout :( - from a run where this problem occurs? thanks. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load [EMAIL PROTECTED] changed: What|Removed |Added Severity|Major |Blocker Component|HTTP/1.1 Connector |Catalina Priority|Other |High Version|4.0.2 Beta 1|4.0.2 Final --- Additional Comments From [EMAIL PROTECTED] 2002-02-18 05:52 --- Recategorizing bug, and upgrading priority. The 1 request - 1 new processor situation Glenn experienced could indeed be an important clue. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
Re: DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
Yeah, well, I can confirm that o.a.c.connector.http.HttpConnector is just a broken as o.a.ajp.tc4.Ajp13Connector. However, the o.a.c.connector.http.HttpProcessor isn't quite as brain-dead as it's ajp13 counterpart (but it still doesn't recycle). The problem is that nothing handles dying threads. - Original Message - From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Sunday, February 17, 2002 9:52 PM Subject: DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load [EMAIL PROTECTED] changed: What|Removed |Added -- -- Severity|Major |Blocker Component|HTTP/1.1 Connector |Catalina Priority|Other |High Version|4.0.2 Beta 1|4.0.2 Final --- Additional Comments From [EMAIL PROTECTED] 2002-02-18 05:52 --- Recategorizing bug, and upgrading priority. The 1 request - 1 new processor situation Glenn experienced could indeed be an important clue. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-02-17 03:50 --- When trying to put a new site into production today I ran into a similar problem at least four times using mod_jk and Ajp13. Each time I had to shutdown and restart Tomcat. Everything would work fine for several hours. Tomcat would end up with ~50 Ajp13Processors. maxProcessors was 75. Then something would change. All of a sudden Tomcat would start creating additional Ajp13 processors one after another, in a few minutes hitting the max of 75, then it started rejecting connections. Because of the repeatable nature of this, and after reviewing logs, this doesn't appear to be due to a sudden increase in traffic to the site. The Apache server has the following config: Solaris 7, Apache 1.3.22, mod_jk built from cvs ~ 1 week ago. Tomcat is running on a different server. Solaris 8, Tomcat 4.1-dev built from cvs ~ 3-4 weeks ago, jk and ajp jars built from cvs ~ 1 week ago. Java(TM) 2 Runtime Environment, Standard Edition (build 1.3.1_02-b02) I reviewed all the logs including the apache mod_jk.log and could not find anything obvious that may have triggered this behaviour. FYI, we have another site running with an identical Apache/Tomcat config, but much lower volume. It has been running fine for over a week. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load [EMAIL PROTECTED] changed: What|Removed |Added CC||[EMAIL PROTECTED] --- Additional Comments From [EMAIL PROTECTED] 2002-02-07 16:21 --- *** Bug 6260 has been marked as a duplicate of this bug. *** -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load [EMAIL PROTECTED] changed: What|Removed |Added CC||[EMAIL PROTECTED] --- Additional Comments From [EMAIL PROTECTED] 2002-01-29 21:07 --- We are testing our application with the MS Application Stress tool. When we run tests for a long time (a weekend or so) then sometimes a new HTTP Processor is created (log/catalina_xxx.txt). Because the stress tool always runs with a fixed number of users the number of processors shouldn't increase. After such a test it isn't possible to shutdown the tomcat (without killing the processes). I don't know if it's an error in our software or in tomcat. I believe it's in tomcat but I have no arguments... I will continue to search the cause. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
DO NOT REPLY [Bug 5735] - HTTP connector running out of processors under heavy load
DO NOT REPLY TO THIS EMAIL, BUT PLEASE POST YOUR BUG RELATED COMMENTS THROUGH THE WEB INTERFACE AVAILABLE AT http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735. ANY REPLY MADE TO THIS MESSAGE WILL NOT BE COLLECTED AND INSERTED IN THE BUG DATABASE. http://nagoya.apache.org/bugzilla/show_bug.cgi?id=5735 HTTP connector running out of processors under heavy load --- Additional Comments From [EMAIL PROTECTED] 2002-01-29 21:11 --- About the shutdown problem, it may be another problem which was fixed. Please try to see if 4.0.2 b2 works better for that. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
