RE: 3.2.2b3 mod_jk gets stuck in readFully
After quite a bit of struggle, I think I found out what is going on. The problem is that the default configuration of Tomcat does not have enough threads in its thread pool for the default configuration of Apache. This issue would only be apparent if many Apache children were in use. The result was that any Apache children over the number of Tomcat threads would hang waiting for Tomcat to respond to requests. Tomcat would not respond until threads became available, which could be quite a long time if Apache children were not dying off (ie, because load was increasing during the day). I was wrong about the threads being stuck in readFully. The real problem is that not enough threads existed at all (ie, the thread handling socket accept would be blocked). The simplest workaround is to change the AJP13 connector to SimpleTcpConnector rather than PoolTcpConnector in server.xml. I strongly suggest that thread pool exhaustion emit a log message, since this was quite difficult to track down. Additionally, it would be better for the default configuration to be more robust. Bill --- Marc Saegesser [EMAIL PROTECTED] wrote: I finally got some time to look at this and I think I can duplicate the problem your seeing. Hopefully, its the problem your seeing, or else we have two serious problems. __ Do You Yahoo!? Yahoo! Auctions - buy the things you want at great prices http://auctions.yahoo.com/
RE: 3.2.2b3 mod_jk gets stuck in readFully
If 100 is a constraint for the pool size, it should be stated in the Tomcat User's manual, since there it is explained how to increase the pool, but no max is given. I observed Exceptions when using more than 100 threads in the pool, coming from one or two arrays which have fixed size 100 in the tomcat code. I could not reproduce these errors after I updated from 3.2 to 3.2.2 and to 3.3. Is there a reason, why I could stably use a bigger pool with these version? I put 240 concurrent requests on apache. Which was the apache version you used, when the hang problem occured? At 16:47 22.04.01 , you wrote: Two things. First, the other problem that I was seeing turned out to be an Apache problem. I switched to Apache 1.3.19 and my thread hang problems went away. That problem seemed to be a synchronization thing that occurred if requests showed up too close together. As for the thread pool stuff. By default, Tomcat 3.2.x thread pools create 10 threads. This can be changed using the min_spare_threads parameter. The pools will grow as needed up to the maximum number of threads allowed (100, by default). You can increase the maximum number of allowed threads using the max_threads parameter. See if this fixes your problem better than using SimpleTcpConnector. -Original Message- From: Pogo Com [mailto:[EMAIL PROTECTED]] Sent: Saturday, April 21, 2001 5:39 PM To: Marc Saegesser; [EMAIL PROTECTED] Subject: RE: 3.2.2b3 mod_jk gets stuck in readFully After quite a bit of struggle, I think I found out what is going on. The problem is that the default configuration of Tomcat does not have enough threads in its thread pool for the default configuration of Apache. This issue would only be apparent if many Apache children were in use. The result was that any Apache children over the number of Tomcat threads would hang waiting for Tomcat to respond to requests. Tomcat would not respond until threads became available, which could be quite a long time if Apache children were not dying off (ie, because load was increasing during the day). I was wrong about the threads being stuck in readFully. The real problem is that not enough threads existed at all (ie, the thread handling socket accept would be blocked). The simplest workaround is to change the AJP13 connector to SimpleTcpConnector rather than PoolTcpConnector in server.xml. I strongly suggest that thread pool exhaustion emit a log message, since this was quite difficult to track down. Additionally, it would be better for the default configuration to be more robust. Bill --- Marc Saegesser [EMAIL PROTECTED] wrote: I finally got some time to look at this and I think I can duplicate the problem your seeing. Hopefully, its the problem your seeing, or else we have two serious problems. __ Do You Yahoo!? Yahoo! Auctions - buy the things you want at great prices http://auctions.yahoo.com/
RE: 3.2.2b3 mod_jk gets stuck in readFully
Thanks for your help, Marc. Would it be possible to log a message to tomcat.log if the thread pool gets exhausted? I believe the default Apache installation calls for 256 children, so busy sites are going to run into this. A log message suggesting to increase max_threads could save a lot of aggravation! Bill --- Marc Saegesser [EMAIL PROTECTED] wrote: As for the thread pool stuff. By default, Tomcat 3.2.x thread pools create 10 threads. This can be changed using the min_spare_threads parameter. The pools will grow as needed up to the maximum number of threads allowed (100, by default). You can increase the maximum number of allowed threads using the max_threads parameter. See if this fixes your problem better than using SimpleTcpConnector. __ Do You Yahoo!? Yahoo! Auctions - buy the things you want at great prices http://auctions.yahoo.com/
RE: 3.2.2b3 mod_jk gets stuck in readFully
I finally got some time to look at this and I think I can duplicate the problem your seeing. Hopefully, its the problem your seeing, or else we have two serious problems. I'm running Apache 1.3.9 (I'm too lazy to update) and mod_jk using AJP12 on Win2000. I'm testing with http://localhost:8081/examples/jsp/dates/date.jsp. If I use JMeter to hit this URL on a single thread it works fine. As soon as I start more than one thread the requests start to hang. I can also duplicate the problem by hitting the same URL from two machines and then holding down the F5 key on both browsers. Things get stuck, but after some time the requests seem to become unstuck and things start working again. I'm not sure how much time I'm going to have to investigate the problem this weekend (my honey-do list is pretty long), but I'll be looking hard at it on Monday. This is a show stopper bug for 3.2.2. -Original Message- From: Pogo Com [mailto:[EMAIL PROTECTED]] Sent: Thursday, April 19, 2001 7:19 PM To: [EMAIL PROTECTED] Subject: 3.2.2b3 mod_jk gets stuck in readFully I've been testing Tomcat 3.2.2b3 and Apache 1.3 on Solaris, connected with mod_jk. Things are generally working, but there is a serious problem that occurs under load. The problem is that certain Apache children get stuck talking to Tomcat. The children are always requesting JSP pages. Using the Apache status display, a stuck child looks like: 141-0 29388 0/1/1 W 0.00 1092 0 0.0 0.01 0.01 154.x.x.x bingoe01.mysite.com GET /ad/loading-applet.jsp?tabl=1site=pogoscrn=motormindanam It is not uncommon to have half of the Apache children stuck like this. On the tomcat side, the threads appear at this location: "Thread-256" (TID:0xd020a0, sys_thread_t:0xd01fd8, state:R, thread_t: t@263, threadID:0x63cb1dd8, stack_bottom:0x63cb2000, stack_si ze:0x2) prio=5 [1] java.net.SocketInputStream.socketRead(Native Method) [2] java.net.SocketInputStream.read(SocketInputStream.java:85) [3] org.apache.tomcat.service.connector.TcpConnector.receiveFully(TcpC onnector.java:148) [4] org.apache.tomcat.service.connector.TcpConnector.receive(TcpConnec tor.java:118) [5] org.apache.tomcat.service.connector.Ajp13ConnectionHandler.process Connection(Ajp13ConnectionHandler.java:109) [6] org.apache.tomcat.service.TcpWorkerThread.runIt(PoolTcpEndpoint.java:393) [7] org.apache.tomcat.util.ThreadPool$ControlRunnable.run(ThreadPool.java:475) [8] java.lang.Thread.run(Thread.java:478) Generally, the number of threads inside receiveFully matches the number of stuck Apache children. Losing Apache children like this eventually starves the server of ability to do useful work, even though there is a lot of idle time on the machine. Is this a known issue? Thanks, Bill Lipa __ Do You Yahoo!? Yahoo! Auctions - buy the things you want at great prices http://auctions.yahoo.com/
