Re: ThreadPool logFull -- BufferedInputStream
This problem was resolved (thanks to a suggestion from Mladen Turk) by adding connectionTimeout="2" to the AJP connector in the server.xml file. --patty On Mon, 18 Apr 2005, Patty O'Reilly wrote: > Date: Mon, 18 Apr 2005 13:53:24 -0700 (PDT) > From: Patty O'Reilly <[EMAIL PROTECTED]> > Reply-To: Tomcat Users List > To: Tomcat Users List > Subject: ThreadPool logFull -- BufferedInputStream > > I haven't heard anything back so I thought I'd rephrase my question. > > I'm running tomcat-5.0.30 with apache 2.0.52 and j2sdk1.4.2_04 on > Red Hat AS 3 update 2. I am seeing runaway thread creation where > threads continue to accumulate until they are maxed out. > > After doing a thread dump, which I have very little experience > reading, it appears that there may be a socket bug in either Java > or RedHat, but I'm not sure. > > Most threads are locked on BufferedInputStream. This is what a typical > thread looks like: > > "TP-Processor104" daemon prio=1 tid=0x080dc218 nid=0x4c87 runnable > [ad1ff000..a$ > at java.net.SocketInputStream.socketRead0(Native Method) >at java.net.SocketInputStream.read(SocketInputStream.java:129) >at java.io.BufferedInputStream.fill(BufferedInputStream.java:183) >at java.io.BufferedInputStream.read1(BufferedInputStream.java:222) >at java.io.BufferedInputStream.read(BufferedInputStream.java:277) >- locked <0x339bd098> (a java.io.BufferedInputStream) >at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:598) >at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:535) >at > org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:6$ > at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866) >at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.$ > at java.lang.Thread.run(Thread.java:534) > > The last thing called appears to be SocketInputStream.socketRead0 > (Native Method). The application is a simple jsp using taglibs. > > Can anyone shed some light on this. I'm brand new to Tomcat. > > Thanks > > --patty > > On Wed, 13 Apr 2005, Patty O'Reilly wrote: > > > Date: Wed, 13 Apr 2005 16:13:40 -0700 (PDT) > > From: Patty O'Reilly <[EMAIL PROTECTED]> > > Reply-To: Tomcat Users List > > To: tomcat-user@jakarta.apache.org > > Subject: ThreadPool logFull > > > > Hi, > > > > I'm running tomcat 5.0.30. > > > > After running for a week or so the server begins to slow down and > > finally crashes because it is out of threads. I can see the java > > processes accumulating day by day. > > > > Apr 8, 2005 3:11:16 PM org.apache.tomcat.util.threads.ThreadPool logFull > > SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads > > (200) > > or check the servlet status > > > > I'm pretty new to tomcat. Not sure if the error is in my apache > > server's workers.properties, or the tomcat servers server.xml, or > > > > Can anyone point me in the right direction. This service is lightly > > loaded. 200 threads should be plenty. > > > > --patty > > > > > > - > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat connectors
Mladen, You are my hero! Setting connectionTimeout="2" to the AJP connector has resolved my accumulating threads and connections problem. --patty On Sat, 23 Apr 2005, Mladen Turk wrote: > Date: Sat, 23 Apr 2005 10:21:27 +0200 > From: Mladen Turk <[EMAIL PROTECTED]> > Reply-To: Tomcat Users List > To: Tomcat Users List > Subject: Re: tomcat connectors > > Patty O'Reilly wrote: > > Thanks very much for responding. I'm still stuggling with an ever > > growing number of connections to the apache server and an ever growing > > number of threads on my tomcat server. I'm fairly certain it is > > not the application. I have all three timeouts set but no joy. > > > > worker.ifa-prod.jweb1.cachesize=1 > > worker.ifa-prod.jweb1.cache_timeout=600 > > worker.ifa-prod.jweb1.socket_keepalive=1 > > worker.ifa-prod.jweb1.socket_timeout=300 > > worker.ifa-prod.jweb1.recycle_timeout=300 > > > > You are probably using prefork mpm, so there is no way to > control the number of connections to Tomcat in any way, trough > mod_jk. You can limit it only by setting MaxClients in the httpd.conf > to the maximum number desired. > Because of pre-forking mechanism each child process will eventually > establish a single connection to Tomcat, thus the number of connections > will rise from StartServers to MaxClients. > > So there are two solutions for prefork. > 1. Make maxThreads==MaxClients > 2. Add connectionTimeout="2" to AJP connector. > > The leter one will recycle inactive connections, but degrade > the performance. > > Regards, > Mladen. > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat connectors
First off, I really appreaciate your help. Many thanks. It looks like I am using the prefork module. $ ./httpd -l Compiled in modules: core.c prefork.c http_core.c mod_so.c MaxClients is set to 150 on apache. maxThreads in the tomcat server.xml is set to 150 as well, along with aditional threads: I am going to change AJP connector and add the connectionTimeout that you suggested like this: I'm also going to bump maxThreads to 175. Tonight we are going to do some load testing. Fingers crossed. On Sat, 23 Apr 2005, Mladen Turk wrote: > Date: Sat, 23 Apr 2005 10:21:27 +0200 > From: Mladen Turk <[EMAIL PROTECTED]> > Reply-To: Tomcat Users List > To: Tomcat Users List > Subject: Re: tomcat connectors > > Patty O'Reilly wrote: > > Thanks very much for responding. I'm still stuggling with an ever > > growing number of connections to the apache server and an ever growing > > number of threads on my tomcat server. I'm fairly certain it is > > not the application. I have all three timeouts set but no joy. > > > > worker.ifa-prod.jweb1.cachesize=1 > > worker.ifa-prod.jweb1.cache_timeout=600 > > worker.ifa-prod.jweb1.socket_keepalive=1 > > worker.ifa-prod.jweb1.socket_timeout=300 > > worker.ifa-prod.jweb1.recycle_timeout=300 > > > > You are probably using prefork mpm, so there is no way to > control the number of connections to Tomcat in any way, trough > mod_jk. You can limit it only by setting MaxClients in the httpd.conf > to the maximum number desired. > Because of pre-forking mechanism each child process will eventually > establish a single connection to Tomcat, thus the number of connections > will rise from StartServers to MaxClients. > > So there are two solutions for prefork. > 1. Make maxThreads==MaxClients > 2. Add connectionTimeout="2" to AJP connector. > > The leter one will recycle inactive connections, but degrade > the performance. > > > Regards, > Mladen. > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: tomcat connectors
Thanks very much for responding. I'm still stuggling with an ever growing number of connections to the apache server and an ever growing number of threads on my tomcat server. I'm fairly certain it is not the application. I have all three timeouts set but no joy. worker.ifa-prod.jweb1.cachesize=1 worker.ifa-prod.jweb1.cache_timeout=600 worker.ifa-prod.jweb1.socket_keepalive=1 worker.ifa-prod.jweb1.socket_timeout=300 worker.ifa-prod.jweb1.recycle_timeout=300 --patty On Fri, 22 Apr 2005, Lionel Farbos wrote: > Date: Fri, 22 Apr 2005 16:03:59 +0200 > From: Lionel Farbos <[EMAIL PROTECTED]> > To: Tomcat Users List > Cc: [EMAIL PROTECTED] > Subject: Re: tomcat connectors > > Hi, > > As described here : > http://jakarta.apache.org/tomcat/connectors-doc/config/workers.html > the goal of socket_timeout is very different than recyle_timeout : > > socket_timeout is a timeout during the activity (between apache and tomcat) > recycle_timeout is a timeout after the activity; when the exchanges between > apache and tomcat are finished, the socket remains open during > recycle_timeout of inactivity (this feature is close to cache_timeout). > > Regards. > On Wed, 20 Apr 2005 17:30:32 -0700 (PDT) > "Patty O'Reilly" <[EMAIL PROTECTED]> wrote: > > > The descriptions of recycle_timeout and socket_timeout seem very > > similar in the Jakarta Tomcat Connector doc. Does anyone know when > > you would use one over the other, or should both directives be > > used together? > > > > I'm using 1.2.8 for Apache 2.0.52 + RHAS3. > > > > Many thanks > > --patty > > > > - > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
tomcat connectors
The descriptions of recycle_timeout and socket_timeout seem very similar in the Jakarta Tomcat Connector doc. Does anyone know when you would use one over the other, or should both directives be used together? I'm using 1.2.8 for Apache 2.0.52 + RHAS3. Many thanks --patty - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: AW: All threads (250) are currently busy
Can you share your apache connector config? On Wed, 20 Apr 2005, Zsolt Koppany wrote: > Date: Wed, 20 Apr 2005 11:08:33 +0200 > From: Zsolt Koppany <[EMAIL PROTECTED]> > Reply-To: Tomcat Users List > To: 'Tomcat Users List' > Subject: AW: All threads (250) are currently busy > > We had the same problem, but after configuring the apache connector > correctly the problem disappeared. > > Zsolt > > > -Ursprüngliche Nachricht- > > Von: Paul Grimwood [mailto:[EMAIL PROTECTED] > > Gesendet: Mittwoch, 20. April 2005 06:07 > > An: tomcat-user@jakarta.apache.org > > Betreff: All threads (250) are currently busy > > > > Tomcat hangs intermittently (1 to 10 days) with the following message in > > catalina.out > > > > 20/04/2005 13:48:09 org.apache.tomcat.util.threads.ThreadPool logFull > > SEVERE: All threads (250) are currently busy, waiting. Increase maxThreads > > (250) or check the servlet status > > > > > > We are running tomcat 5.5.7 on jdk1.5.0_02 under Redhat Linux Fedora Core > > 2 against an Oracle9 DB. > > > > I have seen various posts to this with suggestions including setting Linux > > threads with LD_ASSUME_KERNEL (tried but problem still exists) and setting > > connection timeout in server.xml from 0 to 6 (but ours is set at 2 > > already). And we have already upgraded to latest Tomcat and JRE to no > > avail. Despite reviewing the mailing list over the past 18 to 24 months, i > > am none the wiser. > > > > The problem started around the time we deployed the code onto a new server > > running a later Linux version but this could be a red herring and I > > suspect it is a database connection issue. Has anyone got any ideas? > > > > - > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
ThreadPool logFull -- BufferedInputStream
I haven't heard anything back so I thought I'd rephrase my question. I'm running tomcat-5.0.30 with apache 2.0.52 and j2sdk1.4.2_04 on Red Hat AS 3 update 2. I am seeing runaway thread creation where threads continue to accumulate until they are maxed out. After doing a thread dump, which I have very little experience reading, it appears that there may be a socket bug in either Java or RedHat, but I'm not sure. Most threads are locked on BufferedInputStream. This is what a typical thread looks like: "TP-Processor104" daemon prio=1 tid=0x080dc218 nid=0x4c87 runnable [ad1ff000..a$ at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:183) at java.io.BufferedInputStream.read1(BufferedInputStream.java:222) at java.io.BufferedInputStream.read(BufferedInputStream.java:277) - locked <0x339bd098> (a java.io.BufferedInputStream) at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:598) at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:535) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:6$ at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.$ at java.lang.Thread.run(Thread.java:534) The last thing called appears to be SocketInputStream.socketRead0 (Native Method). The application is a simple jsp using taglibs. Can anyone shed some light on this. I'm brand new to Tomcat. Thanks --patty On Wed, 13 Apr 2005, Patty O'Reilly wrote: > Date: Wed, 13 Apr 2005 16:13:40 -0700 (PDT) > From: Patty O'Reilly <[EMAIL PROTECTED]> > Reply-To: Tomcat Users List > To: tomcat-user@jakarta.apache.org > Subject: ThreadPool logFull > > Hi, > > I'm running tomcat 5.0.30. > > After running for a week or so the server begins to slow down and > finally crashes because it is out of threads. I can see the java > processes accumulating day by day. > > Apr 8, 2005 3:11:16 PM org.apache.tomcat.util.threads.ThreadPool logFull > SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads > (200) > or check the servlet status > > I'm pretty new to tomcat. Not sure if the error is in my apache > server's workers.properties, or the tomcat servers server.xml, or > > Can anyone point me in the right direction. This service is lightly > loaded. 200 threads should be plenty. > > --patty > > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: ThreadPool logFull
I did the thread dump. Cool. There are 25 threads that look like this: "http-8100-Processor13" daemon prio=1 tid=0x083c6090 nid=0x99a in Object.wait() [bcdff000..bcdff8d0] at java.lang.Object.wait(Native Method) - waiting on <0x33feebc0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Object.wait(Object.java:429) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656) - locked <0x33feebc0> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable) at java.lang.Thread.run(Thread.java:534) There is 1 thread that look like this: "http-8100-Processor25" daemon prio=1 tid=0x084badf8 nid=0x9a6 runnable [bb5ff000..bb5ff8d0] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:353) - locked <0x33ef71f0> (a java.net.PlainSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:448) at java.net.ServerSocket.accept(ServerSocket.java:419) at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60) at org.apache.tomcat.util.net.PoolTcpEndpoint.acceptSocket(PoolTcpEndpoint.java:368) at org.apache.tomcat.util.net.TcpWorkerThread.runIt(PoolTcpEndpoint.java:549) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:534) There are 102 of these runnable processes and two waiting: "TP-Processor39" daemon prio=1 tid=0x0809de40 nid=0x35d0 runnable [b53ff000..b53ff8d0] at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:129) at java.io.BufferedInputStream.fill(BufferedInputStream.java:183) at java.io.BufferedInputStream.read1(BufferedInputStream.java:222) at java.io.BufferedInputStream.read(BufferedInputStream.java:277) - locked <0x3449e188> (a java.io.BufferedInputStream) at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:598) at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:535) at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:663) at org.apache.jk.common.SocketConnection.runIt(ChannelSocket.java:866) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684) at java.lang.Thread.run(Thread.java:534) The total number of processes is now at 143 for this tomcat instance. Yesterday evening it was at around 95. Most of the hits are the result of our monitoring system. I don't know how to make sense of the thread dumps. The application was written by one of our developers. It performs a simple export compliance task. The app simply grabs the user's name and checks to see if it is in a database, if it isn't, the app displays an agreement, which the users either accepts or rejects. If the user accepts the agreement, their name is added to the database. After that, the user is redirected to either a logout page if they don't accept the agreement, or they are redirected to another server, if their name appears in the database. Can you help with the tread dump? --patty On Thu, 14 Apr 2005, David Rees wrote: > Date: Thu, 14 Apr 2005 04:56:17 -0700 > From: David Rees <[EMAIL PROTECTED]> > Reply-To: Tomcat Users List , > David Rees <[EMAIL PROTECTED]> > To: Tomcat Users List > Subject: Re: ThreadPool logFull > > On 4/13/05, Patty O'Reilly <[EMAIL PROTECTED]> wrote: > > > > I'm running tomcat 5.0.30. > > > > After running for a week or so the server begins to slow down and > > finally crashes because it is out of threads. I can see the java > > processes accumulating day by day. > > Send the process a -QUIT signal if on Linux, do the equivalent on > Windows and you can get stack traces showing where each thread is > getting stuck instead of returning to the thread pool for new > processing. > > > I'm pretty new to tomcat. Not sure if the error is in my apache > > server's workers.properties, or the tomcat servers server.xml, or > > It's probably not a Tomcat issue, but your something in your > application which is getting hung up. > > -Dave > > - > To unsubscribe, e-mail: [EMAIL PROTECTED] > For additional commands, e-mail: [EMAIL PROTECTED] > > - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
ThreadPool logFull
Hi, I'm running tomcat 5.0.30. After running for a week or so the server begins to slow down and finally crashes because it is out of threads. I can see the java processes accumulating day by day. Apr 8, 2005 3:11:16 PM org.apache.tomcat.util.threads.ThreadPool logFull SEVERE: All threads (200) are currently busy, waiting. Increase maxThreads (200) or check the servlet status I'm pretty new to tomcat. Not sure if the error is in my apache server's workers.properties, or the tomcat servers server.xml, or Can anyone point me in the right direction. This service is lightly loaded. 200 threads should be plenty. --patty - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
