subject:"Tomcat threads"

Hi Charles,

Let me explain the scenario. When tomcat gets a request, it does a socket
send to some other process to handle the request and then respond. This
would happen fast. But assuming 20k client requests come in at the same
time, the server would try to allocate 20k threads and handle it. Mostly the
system would trash and go down.

But if it was to be handled gracefully, we can have the threads pick up the
request throw it in a queue and let the worker threads work and respond. Let
me know if I am missing something here.

Bharath


On Tue, Mar 2, 2010 at 7:22 PM, Caldarale, Charles R 
chuck.caldar...@unisys.com wrote:

  From: Bharath Vasudevan [mailto:bharath@gmail.com]
  Subject: Re: Tomcat threads
 
  Why is it illlogical?

 40+ years of system architecture experience.

  If the number of requests increases, the number of threads
  that can be handled by the system goes down.

 You'll have to explain that one, since it doesn't make sense as written.

  The context switches and the pain to handle the switches makes
  handling of the requests in lesser threads which is scalable.

 Nor does that.  What are you categorizing as a lesser thread?  What makes
 those any more scalable than any other kind of thread?  You're free to set
 the thread pool limit to any value you want, including infinity.  Setting
 the limit to larger than what the logical and physical resources of the
 system you're running on can handle will induce performance problems (e.g.,
 page thrashing, excessive GCs) or outright failures as soon as someone
 decides to start pounding on your server.

  - Chuck


 THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
 MATERIAL and is thus for use only by the intended recipient. If you received
 this in error, please contact the sender and delete the e-mail and its
 attachments from all computers.


 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat threads

2010-03-03 Thread Mark Thomas

On 03/03/2010 17:21, Bharath Vasudevan wrote:
 Hi Charles,
 
 Let me explain the scenario. When tomcat gets a request, it does a socket
 send to some other process to handle the request and then respond. This
 would happen fast. But assuming 20k client requests come in at the same
 time, the server would try to allocate 20k threads and handle it. Mostly the
 system would trash and go down.
 
 But if it was to be handled gracefully, we can have the threads pick up the
 request throw it in a queue and let the worker threads work and respond. Let
 me know if I am missing something here.

Like the fact that Tomncat is open source and you could just look at the
source code for the various connectors and figure this all out for
yourself? You don't even have to read the source. The connectors
documentation covers this pretty well.

Mark



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat threads

Thanks Bill. Comet is something that I can dig into :).

On Tue, Mar 2, 2010 at 7:21 PM, Bill Barker billwbar...@verizon.net wrote:



 Caldarale, Charles R chuck.caldar...@unisys.com wrote in message
 news:99c8b2929b39c24493377ac7a121e21f96cb817...@usea-exch8.na.uis.unisys.com.
 ..

  From: Bharath Vasudevan [mailto:bharath@gmail.com]
 Subject: Re: Tomcat threads

 If we get a request on a thread, let some other thread do
 the work for it and store the response info. The thread
 which does the work writes the response on that request.


 If the processing is fast, why would you go to the complexity and overhead
 of switching to another thread to process the request?

 You should also read the servlet spec, in particular SRV.2.3.3.3:

 Implementations of the request and response objects are not guaranteed to
 be thread
 safe. This means that they should only be used within the scope of the
 request handling
 thread.

 References to the request and response objects should not be given to
 objects
 executing in other threads as the resulting behavior may be
 nondeterministic. If
 the thread created by the application uses the container-managed objects,
 such as
 the request or response object, those objects must be accessed only within
 the
 servlet's service life cycle and such thread itself should have a life
 cycle within
 the life cycle of the servlet's service method because accessing those
 objects
 after the service method ends may cause undeterministic problems.

 The illogical behavior you're asking for simply isn't allowed.


 At least not until Tomcat 7 ;).  In there, you can do exactly what the user
 is asking for using the AsyncContext.  The details are in the Servlet 3.0
 spec, but basically the request is handed off to another thread and the
 original one returns to the pool.  This also allows for two-way
 communication instead of relying on polling.

 Of course Comet offers similar functionality in Tomcat 6 if you don't mind
 being bound to Tomcat.


  - Chuck


 THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
 MATERIAL and is thus for use only by the intended recipient. If you received
 this in error, please contact the sender and delete the e-mail and its
 attachments from all computers.




 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat threads

That would be the last go. Trying to figure out if there is techniques to
handle such scenarios. Looks like comet can asynchronously push data to the
user. If this is going to be seamless to the client, it might as well look
like a response for their request.

On Wed, Mar 3, 2010 at 9:25 AM, Mark Thomas ma...@apache.org wrote:

 On 03/03/2010 17:21, Bharath Vasudevan wrote:
  Hi Charles,
 
  Let me explain the scenario. When tomcat gets a request, it does a socket
  send to some other process to handle the request and then respond. This
  would happen fast. But assuming 20k client requests come in at the same
  time, the server would try to allocate 20k threads and handle it. Mostly
 the
  system would trash and go down.
 
  But if it was to be handled gracefully, we can have the threads pick up
 the
  request throw it in a queue and let the worker threads work and respond.
 Let
  me know if I am missing something here.

 Like the fact that Tomncat is open source and you could just look at the
 source code for the various connectors and figure this all out for
 yourself? You don't even have to read the source. The connectors
 documentation covers this pretty well.

 Mark



 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat threads

2010-03-03 Thread Caldarale, Charles R

 From: Bharath Vasudevan [mailto:bharath@gmail.com]
 Subject: Re: Tomcat threads

 When tomcat gets a request, it does a socket send to some 
 other process to handle the request and then respond.

Tomcat doesn't do that - your webapp does.

You have now introduced a previously unmentioned dependency - that the response 
depends on the behavior of an external resource, unrelated to Tomcat.  That 
changes the picture considerably.  The neat thing the servlet 3.0 AsyncContext 
does is handle this case by letting you decouple the original Request and 
Response objects from the original processing thread.

 But assuming 20k client requests come in at the same time,
 the server would try to allocate 20k threads and handle it.

No, the server will allocate maxThreads request handlers; the other requests 
would sit in the TCP stack's queue (not in the JVM), up to the configured 
acceptCount value - which you can set as high as your OS allows.

 Mostly the system would trash and go down.

Which is my point about why you don't want to set maxThreads to arbitrarily 
large (or small) values.

 But if it was to be handled gracefully, we can have the 
 threads pick up the request throw it in a queue and let
 the worker threads work and respond.

Which you can do in your webapp by sending an interim response to the client 
and having the client poll for the final one.  You'd have to disable 
keep-alives as well.  The 3.0 spec allows the container to perform this task, 
rather than requiring the webapp to do so.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat threads

2010-03-03 Thread Caldarale, Charles R

 From: Bharath Vasudevan [mailto:bharath@gmail.com]
 Subject: Re: Tomcat threads

 Looks like comet can asynchronously push data to the user.

You do understand that there really is no such thing as push (at least in 
HTTP)?  The client has to have a receive up, so you can't use just a plain 
browser - the client has to be Comet-compatible.

With Tomcat 7, you will be able to use the trick Bill B mentioned: send 
incomplete chunked responses to keep the client from timing out until the full 
response is ready, without dedicating a thread to the task.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat threads

2010-03-03 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Bharath,

On 3/2/2010 7:42 PM, Bharath Vasudevan wrote:
 Why is it illlogical? Fast is a relative term. If the number of requests
 increases, the number of threads that can be handled by the system goes down
 . The context switches and the pain to handle the switches makes handling of
 the requests in lesser threads which is scalable.

If you can afford the memory and CPU to create a second thread pool that
does the real work of your webapp, why not just allocate those same
threads in the request processing thread pool?

Seems like a wash in terms of thread availability with a significant
reduction in overhead of managing multiple thread pools, passing data
between them, etc. and trying to work around a problem that really
doesn't exist.

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkuOpC4ACgkQ9CaO5/Lv0PCGXACgkVGoBYzmZwMUPGT9ZE5G8vlD
LqEAn1IX6EuxlfY4hPZGO5uJOzbb2gI8
=pvO1
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat threads

2010-03-03 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Bharath,

On 3/3/2010 12:21 PM, Bharath Vasudevan wrote:
 Let me explain the scenario. When tomcat gets a request, it does a socket
 send to some other process to handle the request and then respond. This
 would happen fast. But assuming 20k client requests come in at the same
 time, the server would try to allocate 20k threads and handle it. Mostly the
 system would trash and go down.
 
 But if it was to be handled gracefully, we can have the threads pick up the
 request throw it in a queue and let the worker threads work and respond. Let
 me know if I am missing something here.

Wouldn't you have to limit that queue, otherwise you might never catch
up with the requests and run out of memory? Limiting the work queue size
versus the thread pool size are equivalent in terms of user experience:
at some point, the remote requests are refused.

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkuOpKMACgkQ9CaO5/Lv0PB/GQCgk+dKujEGRKF69ZVjmn7uWcVA
rbAAniTg+2nvvOygxACU1tNCmRauitsS
=wO/z
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat threads

Hmmm...

No, the server will allocate maxThreads request handlers; the other
requests would sit in the TCP stack's queue (not in the JVM), up to the
configured acceptCount value - which you can set as high as your OS allows.

I was assuming that the tomcat main thread is going to pick it up from the
TCP stack and let the web server thread handle the request. Wont tomcat
reject the request when max threads have been reached (since it would have
already picked up tcp layer)?

On Wed, Mar 3, 2010 at 9:45 AM, Caldarale, Charles R 
chuck.caldar...@unisys.com wrote:

  From: Bharath Vasudevan [mailto:bharath@gmail.com]
  Subject: Re: Tomcat threads
 
  When tomcat gets a request, it does a socket send to some
  other process to handle the request and then respond.

 Tomcat doesn't do that - your webapp does.

 You have now introduced a previously unmentioned dependency - that the
 response depends on the behavior of an external resource, unrelated to
 Tomcat.  That changes the picture considerably.  The neat thing the servlet
 3.0 AsyncContext does is handle this case by letting you decouple the
 original Request and Response objects from the original processing thread.

  But assuming 20k client requests come in at the same time,
  the server would try to allocate 20k threads and handle it.

 No, the server will allocate maxThreads request handlers; the other
 requests would sit in the TCP stack's queue (not in the JVM), up to the
 configured acceptCount value - which you can set as high as your OS allows.

  Mostly the system would trash and go down.

 Which is my point about why you don't want to set maxThreads to arbitrarily
 large (or small) values.

  But if it was to be handled gracefully, we can have the
  threads pick up the request throw it in a queue and let
  the worker threads work and respond.

 Which you can do in your webapp by sending an interim response to the
 client and having the client poll for the final one.  You'd have to disable
 keep-alives as well.  The 3.0 spec allows the container to perform this
 task, rather than requiring the webapp to do so.

  - Chuck


 THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
 MATERIAL and is thus for use only by the intended recipient. If you received
 this in error, please contact the sender and delete the e-mail and its
 attachments from all computers.


 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat threads

2010-03-03 Thread Peter Crowther

On 3 March 2010 18:24, Bharath Vasudevan bharath@gmail.com wrote:

 Hmmm...

 No, the server will allocate maxThreads request handlers; the other
 requests would sit in the TCP stack's queue (not in the JVM), up to the
 configured acceptCount value - which you can set as high as your OS
 allows.

 I was assuming that the tomcat main thread is going to pick it up from the
 TCP stack and let the web server thread handle the request. Wont tomcat
 reject the request when max threads have been reached (since it would have
 already picked up tcp layer)?

 No, because the container doesn't know when a thread might become
available.  Serving content after a delay is considered better than not
serving content at all, so the requests are simply queued in the TCP stack.

- Peter

Re: Tomcat threads

Thanks Folks.

On Wed, Mar 3, 2010 at 1:03 PM, Peter Crowther
peter.crowt...@melandra.comwrote:

 On 3 March 2010 18:24, Bharath Vasudevan bharath@gmail.com wrote:

  Hmmm...
 
  No, the server will allocate maxThreads request handlers; the other
  requests would sit in the TCP stack's queue (not in the JVM), up to the
  configured acceptCount value - which you can set as high as your OS
  allows.
 
  I was assuming that the tomcat main thread is going to pick it up from
 the
  TCP stack and let the web server thread handle the request. Wont tomcat
  reject the request when max threads have been reached (since it would
 have
  already picked up tcp layer)?
 
  No, because the container doesn't know when a thread might become
 available.  Serving content after a delay is considered better than not
 serving content at all, so the requests are simply queued in the TCP stack.

 - Peter

Tomcat threads

2010-03-02 Thread Bharath Vasudevan

Hi,

I would like to know more on the threads created in tomcat. Are they user
level threads or kernel threads?

I see that the requests are kicked off through the threads from a pool
(which has a configured size), how can this be scalable?

Also, from my understanding in servlets, the response for a request in has
to be done then and there (as in I cant respond for a request later in
future from some random thread) .

If there is contention for a resource which might take time (causing the
response to take time), wont tomcat run out of threads?

Regards,
Bharath

RE: Tomcat threads

 From: Bharath Vasudevan [mailto:bharath@gmail.com]
 Subject: Tomcat threads

 Are they user level threads or kernel threads?

Depends on the particular JVM being used, not on Tomcat.  Pretty much all 
modern JVMs use a kernel thread for each java.lang.Thread instance on which 
start() has been called.

 I see that the requests are kicked off through the threads from a pool
 (which has a configured size), how can this be scalable?

Because you can make the limit as large as you want.  Making it too large 
leaves you open to DOS attacks and running out of resources on the server.

 Also, from my understanding in servlets, the response for a 
 request in has to be done then and there (as in I cant respond
 for a request later in future from some random thread) .

That's the nature of HTTP, not servlets in particular.  A properly designed 
application facing a long-duration situation will usually send an interim 
status back to the client, and kick off an auxiliary thread to do whatever is 
going to run for a while.  The auxiliary thread places the final response in 
some location associated with the client when it's ready.  The client polls for 
status and the final response using JavaScript or some other mechanism.

 If there is contention for a resource which might take time
 (causing the response to take time), wont tomcat run out of
 threads?

Yes; Tomcat won't care, but your users might.  Don't use poorly designed 
applications.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat threads

2010-03-02 Thread Bharath Vasudevan

Thanks Chuck.

One thing I am not able to get is why would the user care if the response
was done from a different thread? Is it not going to be opaque to the user
as long as its fast?

If we get a request on a thread, let some other thread do the work for it
and store the response info. The thread which does the work writes the
response on that request.

Regards,
Bharath

On Tue, Mar 2, 2010 at 3:57 PM, Caldarale, Charles R 
chuck.caldar...@unisys.com wrote:

  From: Bharath Vasudevan [mailto:bharath@gmail.com]
  Subject: Tomcat threads
 
  Are they user level threads or kernel threads?

 Depends on the particular JVM being used, not on Tomcat.  Pretty much all
 modern JVMs use a kernel thread for each java.lang.Thread instance on which
 start() has been called.

  I see that the requests are kicked off through the threads from a pool
  (which has a configured size), how can this be scalable?

 Because you can make the limit as large as you want.  Making it too large
 leaves you open to DOS attacks and running out of resources on the server.

  Also, from my understanding in servlets, the response for a
  request in has to be done then and there (as in I cant respond
  for a request later in future from some random thread) .

 That's the nature of HTTP, not servlets in particular.  A properly designed
 application facing a long-duration situation will usually send an interim
 status back to the client, and kick off an auxiliary thread to do whatever
 is going to run for a while.  The auxiliary thread places the final response
 in some location associated with the client when it's ready.  The client
 polls for status and the final response using JavaScript or some other
 mechanism.

  If there is contention for a resource which might take time
  (causing the response to take time), wont tomcat run out of
  threads?

 Yes; Tomcat won't care, but your users might.  Don't use poorly designed
 applications.

  - Chuck


 THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
 MATERIAL and is thus for use only by the intended recipient. If you received
 this in error, please contact the sender and delete the e-mail and its
 attachments from all computers.


 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat threads

 From: Bharath Vasudevan [mailto:bharath@gmail.com]
 Subject: Re: Tomcat threads

 If we get a request on a thread, let some other thread do 
 the work for it and store the response info. The thread 
 which does the work writes the response on that request.

If the processing is fast, why would you go to the complexity and overhead of 
switching to another thread to process the request?

You should also read the servlet spec, in particular SRV.2.3.3.3:

Implementations of the request and response objects are not guaranteed to be 
thread
safe. This means that they should only be used within the scope of the request 
handling
thread.

References to the request and response objects should not be given to objects
executing in other threads as the resulting behavior may be nondeterministic. If
the thread created by the application uses the container-managed objects, such 
as
the request or response object, those objects must be accessed only within the
servlet's service life cycle and such thread itself should have a life cycle 
within
the life cycle of the servlet's service method because accessing those objects
after the service method ends may cause undeterministic problems.

The illogical behavior you're asking for simply isn't allowed.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat threads

2010-03-02 Thread Bharath Vasudevan

Why is it illlogical? Fast is a relative term. If the number of requests
increases, the number of threads that can be handled by the system goes down
. The context switches and the pain to handle the switches makes handling of
the requests in lesser threads which is scalable.

On Tue, Mar 2, 2010 at 4:34 PM, Caldarale, Charles R 
chuck.caldar...@unisys.com wrote:

  From: Bharath Vasudevan [mailto:bharath@gmail.com]
  Subject: Re: Tomcat threads

  If we get a request on a thread, let some other thread do
  the work for it and store the response info. The thread
  which does the work writes the response on that request.

 If the processing is fast, why would you go to the complexity and overhead
 of switching to another thread to process the request?

 You should also read the servlet spec, in particular SRV.2.3.3.3:

 Implementations of the request and response objects are not guaranteed to
 be thread
 safe. This means that they should only be used within the scope of the
 request handling
 thread.

 References to the request and response objects should not be given to
 objects
 executing in other threads as the resulting behavior may be
 nondeterministic. If
 the thread created by the application uses the container-managed objects,
 such as
 the request or response object, those objects must be accessed only within
 the
 servlet's service life cycle and such thread itself should have a life
 cycle within
 the life cycle of the servlet's service method because accessing those
 objects
 after the service method ends may cause undeterministic problems.

 The illogical behavior you're asking for simply isn't allowed.

  - Chuck

 THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY
 MATERIAL and is thus for use only by the intended recipient. If you received
 this in error, please contact the sender and delete the e-mail and its
 attachments from all computers.

 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat threads

2010-03-02 Thread Bill Barker

Caldarale, Charles R chuck.caldar...@unisys.com wrote in message 
news:99c8b2929b39c24493377ac7a121e21f96cb817...@usea-exch8.na.uis.unisys.com...

From: Bharath Vasudevan [mailto:bharath@gmail.com]
Subject: Re: Tomcat threads

If we get a request on a thread, let some other thread do
the work for it and store the response info. The thread
which does the work writes the response on that request.

If the processing is fast, why would you go to the complexity and overhead 
of switching to another thread to process the request?

You should also read the servlet spec, in particular SRV.2.3.3.3:

Implementations of the request and response objects are not guaranteed to 
be thread
safe. This means that they should only be used within the scope of the 
request handling

thread.

References to the request and response objects should not be given to 
objects
executing in other threads as the resulting behavior may be 
nondeterministic. If
the thread created by the application uses the container-managed objects, 
such as
the request or response object, those objects must be accessed only within 
the
servlet's service life cycle and such thread itself should have a life 
cycle within
the life cycle of the servlet's service method because accessing those 
objects

after the service method ends may cause undeterministic problems.

The illogical behavior you're asking for simply isn't allowed.

At least not until Tomcat 7 ;).  In there, you can do exactly what the user 
is asking for using the AsyncContext.  The details are in the Servlet 3.0 
spec, but basically the request is handed off to another thread and the 
original one returns to the pool.  This also allows for two-way 
communication instead of relying on polling.

Of course Comet offers similar functionality in Tomcat 6 if you don't mind 
being bound to Tomcat.

- Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you 
received this in error, please contact the sender and delete the e-mail 
and its attachments from all computers. 

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat threads

 From: Bharath Vasudevan [mailto:bharath@gmail.com]
 Subject: Re: Tomcat threads

 Why is it illlogical?

40+ years of system architecture experience.

 If the number of requests increases, the number of threads 
 that can be handled by the system goes down.

You'll have to explain that one, since it doesn't make sense as written.

 The context switches and the pain to handle the switches makes
 handling of the requests in lesser threads which is scalable.

Nor does that.  What are you categorizing as a lesser thread?  What makes 
those any more scalable than any other kind of thread?  You're free to set the 
thread pool limit to any value you want, including infinity.  Setting the limit 
to larger than what the logical and physical resources of the system you're 
running on can handle will induce performance problems (e.g., page thrashing, 
excessive GCs) or outright failures as soon as someone decides to start 
pounding on your server.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat threads

 From: Bill Barker [mailto:billwbar...@verizon.net]
 Subject: Re: Tomcat threads

 basically the request is handed off to another thread and the
 original one returns to the pool.  This also allows for two-way
 communication instead of relying on polling.

What's required on the client end?  You can't simply push HTML or other content 
to a browser.  Is a specialized (or at least scripted) client required?

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Tomcat threads

2010-03-02 Thread Bill Barker

Caldarale, Charles R chuck.caldar...@unisys.com wrote in message 
news:99c8b2929b39c24493377ac7a121e21f96cb817...@usea-exch8.na.uis.unisys.com...

From: Bill Barker [mailto:billwbar...@verizon.net]
Subject: Re: Tomcat threads

basically the request is handed off to another thread and the
original one returns to the pool.  This also allows for two-way
communication instead of relying on polling.

What's required on the client end?  You can't simply push HTML or other 
content to a browser.  Is a specialized (or at least scripted) client 
required?

Mostly useful for custom clients channeling under HTTP, but you could 
probably write a scripted client as well that works with it (haven't tried). 
The examples in Tomcat 7 are mostly server-push (e.g. fake stock ticker, 
chat).  These are done by not completing the connection, and just pumping 
more data to a chunked response (so you get the spinning wheel the entire 
time the apps is running).   They work in most browsers (only most, because 
I've only tested them in IE and FireFox).

But as I said before, would probably help the OP, assuming that traffic 
isn't too great.

- Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you 
received this in error, please contact the sender and delete the e-mail 
and its attachments from all computers. 

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Tomcat threads

 From: Bill Barker [mailto:billwbar...@verizon.net]
 Subject: Re: Tomcat threads

 The examples in Tomcat 7 are mostly server-push (e.g. fake stock
 ticker, chat).  These are done by not completing the connection,
 and just pumping more data to a chunked response (so you get the
 spinning wheel the entire time the apps is running).

An interesting trick; thanks for the info.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-23 Thread André Warnier


Pid wrote:


I am not a number!


That betrays your age too, you know that ?

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-22 Thread Rainer Jung

On 22.05.2009 03:54, Pantvaidya, Vishwajit wrote:
 -Original Message-
 From: Rainer Jung [mailto:rainer.j...@kippdata.de]
 Sent: Thursday, May 21, 2009 3:37 PM
 To: Tomcat Users List
 Subject: Re: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 On 22.05.2009 00:19, Pantvaidya, Vishwajit wrote:
 [Pantvaidya, Vishwajit] I will set
 - cachesize=1 (doc says jk will autoset this value only for worker-mpm
 and we use httpd 2.0 prefork)

 You don't have to: JK will discover this number for the Apache web
 server automatically and set the pool size to this value.

 [Pantvaidya, Vishwajit] Does what you say hold true for jk 1.2.15 also? 
 Because I saw that for the 1.2.15 cachesize directive, 
 http://tomcat.apache.org/connectors-doc/reference/workers.html#Deprecated%20Worker%20Directives
  says that JK will discover the number of threads per child process on 
 Apache 2 web server with worker-mpm and set its default value to match the 
 ThreadsPerChild Apache directive.. Since we use pre-fork MPM, I assumed we 
 need to set cachesize.

I would say yes, but now your Ops people who resist upgrading try to
play my time against their time. I'm not going to look it up for them,
they should upgrade ;)

 -  remove cache and recycle timeouts
 Chris and me are not having the same opinion here. You can choose :)

 [Pantvaidya, Vishwajit] I think that may be only because my adding the 
 connectionTimeout led you to believe that I wanted nonpersistent conn's. Now 
 that I know persistent connections are better, I am trying to rollback 
 connectionTimeout - and then I guess you will agree with Chris that I need to 
 rollback the recycletimeouts, etc in workers file on httpd side also?

My point is: persistent connections are good, but connections which are
idle for a long time are not as good, so close them after some idle
time, like e.g. 10 minutes. Of course this means you need to create new
ones once your load goes up again, but that's not a big problem.

Regards,

Rainer

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-22 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Chetan,

On 5/21/2009 2:08 PM, Chetan Chheda wrote:
 I am following this thread with great interest. I have a similar
 issue as Vishwajit and have resorted to adding the connectionTimeout
 to get rid of a large number of RUNNABLE threads.

Why? Are you just offended by the number of threads, or do you have a
legitimate resource problem?

 But mod_jk does not like tomcat threads timing out and logs the
 message increase the backend idle connection timeout or the
 connection_pool_minsize in the mod_jk logs
 
 which leads me to believe that its apache thats not letting go of the
 threads in my case.

Again, you need to set the Connector's connectionTimeout /and/ your
workers' connection_pool_timeout settings to the same time interval
(note that they use different semantics... one is in seconds and the
other is in ms, so read the documentation carefully).

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkoWnKAACgkQ9CaO5/Lv0PBcTwCbBhuJ8/nwYLq/LAxCSVDer35t
jAIAn2oUL3on6ki/x9pZHn8n0tLuVS8H
=y10X
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-22 Thread Chetan Chheda

Chris, 

  As with Vishwajit, my tomcat ends up with all threads busy and not serving 
any new requests. After setting the connectionTimeout the threads are being 
recycled but apache is not liking .. as per the messsage in mod_jk.log 

Chetan





From: Christopher Schultz ch...@christopherschultz.net
To: Tomcat Users List users@tomcat.apache.org
Sent: Friday, May 22, 2009 8:37:52 AM
Subject: Re: Running out of tomcat threads - why many threads in RUNNABLEstage 
even with no activity

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Chetan,

On 5/21/2009 2:08 PM, Chetan Chheda wrote:
 I am following this thread with great interest. I have a similar
 issue as Vishwajit and have resorted to adding the connectionTimeout
 to get rid of a large number of RUNNABLE threads.

Why? Are you just offended by the number of threads, or do you have a
legitimate resource problem?

 But mod_jk does not like tomcat threads timing out and logs the
 message increase the backend idle connection timeout or the
 connection_pool_minsize in the mod_jk logs
 
 which leads me to believe that its apache thats not letting go of the
 threads in my case.

Again, you need to set the Connector's connectionTimeout /and/ your
workers' connection_pool_timeout settings to the same time interval
(note that they use different semantics... one is in seconds and the
other is in ms, so read the documentation carefully).

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkoWnKAACgkQ9CaO5/Lv0PBcTwCbBhuJ8/nwYLq/LAxCSVDer35t
jAIAn2oUL3on6ki/x9pZHn8n0tLuVS8H
=y10X
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-22 Thread Caldarale, Charles R

 From: Chetan Chheda [mailto:chetan_chh...@yahoo.com]
 Subject: Re: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 As with Vishwajit, my tomcat ends up with all threads busy and not
 serving any new requests. After setting the connectionTimeout the
 threads are being recycled but apache is not liking .. as per the
 messsage in mod_jk.log

Again, the most likely cause is something between httpd and Tomcat that is 
silently dropping the connections; a badly behaving firewall is a possible 
culprit.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-22 Thread Pantvaidya, Vishwajit

 -Original Message-
 From: Rainer Jung [mailto:rainer.j...@kippdata.de]
 Sent: Friday, May 22, 2009 2:53 AM
 To: Tomcat Users List
 Subject: Re: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 My point is: persistent connections are good, but connections which are
 idle for a long time are not as good, so close them after some idle
 time, like e.g. 10 minutes. Of course this means you need to create new
 ones once your load goes up again, but that's not a big problem.

[Pantvaidya, Vishwajit] Why are connections idle for a long time not good? I 
thought threads when idle take only a little memory and cpu. Are there any 
other reasons?

Thanks a lot Rainer, Chuck, Chris, Andre, Pid, Martin and everyone else I 
missed. I spent quite some time yesterday chewing on everything I gathered in 
the last few days' interactions and the conflicting behavior we are seeing in 
our systems - that led to following conclusions and action plan:

Behavior observed in diff production systems:
a. medium-to-large thread count whether firewall exists or not
b. % of runnable threads is much higher where firewall between httpd/tomcat
c. atleast 1 server where firewall exists has run out of threads
d. atleast 1 server where no firewall exists has run out of threads

Conclusions:
1. In general, runnable threads should not be a prob, unless they correspond to 
dropped connections. Since on our servers that have firewall between httpd and 
tomcat, runnable connections are not being used for new requests and tomcat 
keeps on creating new threads (leading to #b/c above), those threads could 
correspond to:
i. connections dropped by firewall or
ii. hanging tomcat threads as httpd recycle timeout disconnected the 
connection from that side (and there was no connectiontimeout in server.xml so 
that tomcat could do the same) or
iii. combination of these i and ii
2. Runnable threads on servers where no firewall exist (and we do not see 
server running out of threads) should not be a point of concerns as they do not 
correspond to dropped connections, as seen from netstat o/p at the end of this 
email. So #a above could be ignored.
3. Observation #d above is puzzling and currently I have no answers for that

Action:
- check both sides by using netstat -anop (Apache side and the Tomcat side 
without connectionTimeout, so you can see the problem in the original form). 
See whether the number of AJP connections in the various TCP states differs 
much between the netstat output on the Apache and on the Tomcat system.
- Bring workers.properties settings in line with Apache recommendations:
- Worker...cachesize=10 - set to 1
- Worker...cache_timeout=600 - remove
- Worker...recycle_timeout=300 - remove

Netstat o/p's: connector running on 21005, no firewall between httpd/tomcat

Httpd Side:

Proto Recv-Q Send-Q Local Address   Foreign Address 
State   PID/Program nameTimer
tcp0  0 129.41.29.241:53777 129.41.29.48:21005  
ESTABLISHED -   keepalive (2869.65/0/0)
tcp0  0 129.41.29.241:53943 129.41.29.48:21005  
ESTABLISHED -   keepalive (3341.39/0/0)
tcp0  0 129.41.29.241:49950 129.41.29.48:21005  
ESTABLISHED -   keepalive (6701.51/0/0)
tcp0  0 129.41.29.241:49927 129.41.29.48:21005  
ESTABLISHED -   keepalive (6240.25/0/0)
tcp0  0 129.41.29.241:49926 129.41.29.48:21005  
ESTABLISHED -   keepalive (6239.47/0/0)
tcp0  0 129.41.29.241:49971 129.41.29.48:21005  
ESTABLISHED -   keepalive (6931.40/0/0)
tcp0  0 129.41.29.241:49868 129.41.29.48:21005  
ESTABLISHED -   keepalive (5743.83/0/0)
tcp0  0 129.41.29.241:49865 129.41.29.48:21005  
ESTABLISHED -   keepalive (5741.65/0/0)
tcp0  0 129.41.29.241:49867 129.41.29.48:21005  
ESTABLISHED -   keepalive (5743.16/0/0)
tcp0  0 129.41.29.241:49901 129.41.29.48:21005  
ESTABLISHED -   keepalive (5906.92/0/0)
tcp0  0 129.41.29.241:49795 129.41.29.48:21005  
ESTABLISHED -   keepalive (4659.11/0/0)
tcp0  0 129.41.29.241:49558 129.41.29.48:21005  
ESTABLISHED -   keepalive (1705.06/0/0)
tcp0  0 129.41.29.241:50796 129.41.29.48:21005  
ESTABLISHED -   keepalive (4551.79/0/0)
tcp0  0 129.41.29.241:50784 129.41.29.48:21005  
ESTABLISHED -   keepalive (4539.53/0/0)
tcp0  0 129.41.29.241:50711 129.41.29.48:21005  
ESTABLISHED -   keepalive

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-22 Thread Caldarale, Charles R

 From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com]
 Subject: RE: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 [Pantvaidya, Vishwajit] Why are connections idle for a 
 long time not good?

He didn't say they weren't good, he said they weren't _as_ good - a relative, 
not absolute, assessment.  Things sitting around idle are not /as good/ as 
things in use.  It doesn't make them bad.

 1. In general, runnable threads should not be a prob, unless they
 correspond to dropped connections.

Correct; need to focus on why the connections are becoming unusable.

 3. Observation #d above is puzzling and currently I have 
 no answers for that

That is strange; is there anything else between httpd and Tomcat in that 
situation?  (E.g., an overly clever router?)

 Netstat o/p's: connector running on 21005, no firewall between
 httpd/tomcat

That all looks reasonable to me - the port numbers match.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-22 Thread Rainer Jung

On 22.05.2009 21:09, Pantvaidya, Vishwajit wrote:
 -Original Message-
 From: Rainer Jung [mailto:rainer.j...@kippdata.de]
 Sent: Friday, May 22, 2009 2:53 AM
 To: Tomcat Users List
 Subject: Re: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 My point is: persistent connections are good, but connections which are
 idle for a long time are not as good, so close them after some idle
 time, like e.g. 10 minutes. Of course this means you need to create new
 ones once your load goes up again, but that's not a big problem.

 [Pantvaidya, Vishwajit] Why are connections idle for a long time not good? I 
 thought threads when idle take only a little memory and cpu. Are there any 
 other reasons?

Because you might want to monitor connections in order to learn how many
threads you need for your load and how things grow or shrink over time.
If you keep connections open an infinite number of time, you'll only
monitor the biggest need since restart, which is often not very
interesting, because it often is artificial (triggered by some
performance slowness you might have a very big connection number created
during a short time).

Second: because they are making trouble so often in combination with
firewalls. So in general I like persistent connections as long as they
are closed when idle for a longer time. So usually I set the pool min
size to 0 and the idle connection timeout to 10 minutes.

 Thanks a lot Rainer, Chuck, Chris, Andre, Pid, Martin and everyone else I 
 missed. I spent quite some time yesterday chewing on everything I gathered in 
 the last few days' interactions and the conflicting behavior we are seeing in 
 our systems - that led to following conclusions and action plan:

 Behavior observed in diff production systems:
 a. medium-to-large thread count whether firewall exists or not
 b. % of runnable threads is much higher where firewall between httpd/tomcat
 c. atleast 1 server where firewall exists has run out of threads
 d. atleast 1 server where no firewall exists has run out of threads

Concurrency = Load * ResponseTime

Concurrency: number of requests being processed in parallel
Load: Number of Requests per Second being handled
ResponseTime: Average Response time in seconds.

So in case you have a performance problem and for a given load your
response time goes up by a factor of ten, the number of connections will
also go up by a factpr of 10. That's most often the reason for d) and
was the reason, why we asked for thread dumps.

 Conclusions:
 1. In general, runnable threads should not be a prob, unless they correspond 
 to dropped connections. Since on our servers that have firewall between httpd 
 and tomcat, runnable connections are not being used for new requests and 
 tomcat keeps on creating new threads (leading to #b/c above), those threads 
 could correspond to:
   i. connections dropped by firewall or
   ii. hanging tomcat threads as httpd recycle timeout disconnected the 
 connection from that side (and there was no connectiontimeout in server.xml 
 so that tomcat could do the same) or
   iii. combination of these i and ii
 2. Runnable threads on servers where no firewall exist (and we do not see 
 server running out of threads) should not be a point of concerns as they do 
 not correspond to dropped connections, as seen from netstat o/p at the end of 
 this email. So #a above could be ignored.
 3. Observation #d above is puzzling and currently I have no answers for that

If d) happens again, do some thread dumps.

 Action:
 - check both sides by using netstat -anop (Apache side and the Tomcat side 
 without connectionTimeout, so you can see the problem in the original form). 
 See whether the number of AJP connections in the various TCP states differs 
 much between the netstat output on the Apache and on the Tomcat system.
 - Bring workers.properties settings in line with Apache recommendations:
   - Worker...cachesize=10 - set to 1

respectively when using Apache, remove this. Rely on the defaults for
that one.

   - Worker...cache_timeout=600 - remove
   - Worker...recycle_timeout=300 - remove

Hmmm.

 Netstat o/p's: connector running on 21005, no firewall between httpd/tomcat

 Httpd Side:

 Proto Recv-Q Send-Q Local Address   Foreign Address 
 State   PID/Program nameTimer
 tcp0  0 129.41.29.241:53777 129.41.29.48:21005  
 ESTABLISHED -   keepalive (2869.65/0/0)
 tcp0  0 129.41.29.241:53943 129.41.29.48:21005  
 ESTABLISHED -   keepalive (3341.39/0/0)
 tcp0  0 129.41.29.241:49950 129.41.29.48:21005  
 ESTABLISHED -   keepalive (6701.51/0/0)
 tcp0  0 129.41.29.241:49927 129.41.29.48:21005  
 ESTABLISHED -   keepalive (6240.25/0/0)
 tcp0  0 129.41.29.241:49926 129.41.29.48:21005

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-22 Thread Pantvaidya, Vishwajit

 -Original Message-
 From: Rainer Jung [mailto:rainer.j...@kippdata.de]
 Sent: Friday, May 22, 2009 12:39 PM
 To: Tomcat Users List
 Subject: Re: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity
  [Pantvaidya, Vishwajit] Why are connections idle for a long time not
 good? I thought threads when idle take only a little memory and cpu. Are
 there any other reasons?

 Because you might want to monitor connections in order to learn how many
 threads you need for your load and how things grow or shrink over time.
 If you keep connections open an infinite number of time, you'll only
 monitor the biggest need since restart, which is often not very
 interesting, because it often is artificial (triggered by some
 performance slowness you might have a very big connection number created
 during a short time).

[Pantvaidya, Vishwajit] Good reason - I think ultimately after some immediate 
testing to diagnose the outofthread issues, I will use timeouts.

  d. atleast 1 server where no firewall exists has run out of threads

 Concurrency = Load * ResponseTime

 Concurrency: number of requests being processed in parallel
 Load: Number of Requests per Second being handled
 ResponseTime: Average Response time in seconds.

 So in case you have a performance problem and for a given load your
 response time goes up by a factor of ten, the number of connections will
 also go up by a factpr of 10. That's most often the reason for d) and
 was the reason, why we asked for thread dumps.

[Pantvaidya, Vishwajit] Again good explanation and makes a lot of sense - I do 
seem to remember we had performance problems on that machine. Will keep this in 
mind and monitor threads and take dumps if outofthreads reoccurs on that server.

  - Bring workers.properties settings in line with Apache recommendations:
  - Worker...cachesize=10 - set to 1

 respectively when using Apache, remove this. Rely on the defaults for
 that one.

[Pantvaidya, Vishwajit] Sure will do - once we migrate to jk 1.2.28.

  - Worker...cache_timeout=600 - remove
  - Worker...recycle_timeout=300 - remove

 Hmmm.

[Pantvaidya, Vishwajit] Considering the excellent reasons you have given above 
- ultimately I will retain timeouts. But for testing firewall issues, I need to 
rollback connTimeout in server.xml and to make sure that my settings are 
consistent I need to rollback the above timeouts also.

Again thanks - I think I have reasonable explanations for most of the issues / 
conflicting observations. This thread may be quiet for some time as I do more 
testing as per the actions I mentioned - will get back with results and final 
conclusions later.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-21 Thread Pid

Caldarale, Charles R wrote:
 From: André Warnier [mailto:a...@ice-sa.com]
 Subject: Re: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 if you add -p to netstat (at least under Linux), it will also 
 show the program that corresponds to that line.

 Or at least -o to show the pid number.

I am not a number!

p

 - Chuck

 THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
 MATERIAL and is thus for use only by the intended recipient. If you received 
 this in error, please contact the sender and delete the e-mail and its 
 attachments from all computers.

 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-21 Thread Caldarale, Charles R

 From: Pid [mailto:p...@pidster.com]
 Subject: Re: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 I am not a number!

But are you a free man?

We want information, by hook or by crook...

(Apologies to the late Patrick McGoohan.)

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-21 Thread Rainer Jung

Trying to add some info below.

On 21.05.2009 05:09, Caldarale, Charles R wrote:
From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com]
Subject: RE: Running out of tomcat threads - why many threads in
RUNNABLEstage even with no activity

So socket_keepalive is already 1. So does this mean that firewall is
dropping connections in spite of it.

The doc does not mention using 1 here, just true (although other variables
allow either). Would be best to get Rainer's opinion when the sun comes up
in Europe.

My netstat o/p had only 2 tomcat connections active in FIN_WAIT2 and
about 11 in keepalive on httpd side - I guess this does not indicate
any hanging connections?

It's not what one would like to see, especially since none of the ports match
- all of the connections are broken.

Could that be because currently connectionTimeout is
active in my server.xml?

I think so; it appears that setting the connectionTimeout on the Tomcat side
will effectively disable the expected persistence; it's an expensive
workaround for the problem.

1) If you want to analyze your original problem, you need to get back to
the original situation, i.e. without connectionTimeout. It doesn't make
much sense to guess about the original problem by looking at something
very different.

2) The output of netstat and the content of a thread dump change in
time. If you want to understand the exact relations between the two
netstat outputs and a thread dump, you need to ensure to produce those
three things as close in time as possible. I'm talking about seconds not
milliseconds.

3) I think I already indicated that you do not want to look at entries
in TIME_WAIT state. This state is special and not related to any threads
in Apache or in Tomcat. A connection in TIME_WAIT state is in no way
longer associated with a process and will no longer handle any data.
It's a placeholder to prevent new connections using the same ports and
possibly getting confused by old packets for the previous connections
coming in late. The only reason to care about TIME_WAIT connections is,
when the total number of all connections on the system (all ports and
including TIME_WAIT) gets into more than 1. Some systems can cope
with more, like 6, but if you go above 1, then you need to start
thinking about it. I assume this is in no way the case here. So let's
for the moment always forget about the TIME_WAIT connections.

4) Firewall idle connection drop: First read

http://tomcat.apache.org/connectors-doc/generic_howto/timeouts.html#Firewall%20Connection%20Dropping

carefully and try to understand.

Any mod_jk attribute that takes a booelan value will accept 1, true,
True, t or T as true, 0, false, False, f or F as false (and maybe even
more).

5) Matching port numbers

Usually the port numbers should match. The non matching of the port
numbers could indicate, that there is a firewall in between, although
most firewall systems will be transparent to the ports (yes, I know
there are many variations). Since the port numbers are very close I
would guess, that the reason for not matching is that netstat was done a
couple of seconds or more apart, and your connections are only used for
a very short time, so we have lots of new connections activity.

6) TCP states

LISTEN on the Tomcat side corresponds to the one TP thread, that does a
socket accept.

ESTABLISHED: both sides still want to use this connection. On the Tomcat
side shows up as socketRead0()

CLOSE_WAIT: the other side has closed the connection, the local side
hasn't yet. E.g. if Tomcat closes the connection because of
connectionTimeout, but Apache doesn't have a configured idle timeout and
didn't yet try to reuse the half-closed connection, the connection will
be shown as CLOSE_WAIT on the httpd side. If Apache closed the
connection, but Tomcat hasn't noticed yet, it will be CLOSE_WAIT at the
Tomcat end. In this case it could be also socketRead0() in the thread dump.

FIN_WAIT2: most likely the other end of CLOSE_WAIT.

7) mod_jk update

Before you start to fix your mod_jk configuration, go to your ops people
and tell them that they are using a very bad mod_jk version and they
have to update. The right version to update to is 1.2.28. It does make
no sense at all to try to fix this with your old version. Solve your
problem in the right way, by setting much more attributes on the JK side
than simply the connectionTimeout on the Tomcat side.

Most important: read the above timeouts page fully.

Regards,

Rainer

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-21 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Vishwajit,

On 5/19/2009 6:08 PM, Pantvaidya, Vishwajit wrote:
 No, Tomcat uses precisely 1 thread to handle each incoming HTTP
 request. If keepalives are used, multiple requests may be handled
 by the same thread before it is returned to the pool, or different
 threads may be used to serve different requests from the single
 connection, but in either case, no more than 1 thread will be used
 to service a single HTTP request.
 
 [Pantvaidya, Vishwajit] Could this happen if upon my http browser
 request, the app could be spawning multiple redirects in quick
 succession leading tomcat to create multiple threads. Any other
 thoughts why I could be seeing tomcat spawn threads in multiples of
 4?

No, the only thing your browser can do in order to activate a thread is
to make a request. If a single page contains many items to download
(images, CSS files, scripts, etc.) then the browser may make several
requests to the server simultaneously. Most web browsers limit their
simultaneous requests to a single server to 4, so maybe that's what
you're observing.

The weird thing is that you're observing 4 threads doing something
/without/ any requests being made.

 [Pantvaidya, Vishwajit] Ok thanks. httpd -l showed perfork.c. I
 guess that means we are using prefork MPM. So our cachesize should be
 1? Our mod_jk version is 1.2.15 - will that also auto-detect the
 cache-size?

Yes and yes.

 [Pantvaidya, Vishwajit] I agree - but again as I mentioned above
 because the admin will be conservative about any changes, I need to
 have a strong reason.

How's this for a compelling reason: your server keeps locking-up in its
current configuration.

 Also when you say let the connection stay
 alive, does that mean let the TP-Processor thread remain in Waiting
 state / Runnable state?

I mean that you shouldn't set any connection_pool_timeout on any of your
workers. There really isn't a need to set this unless you are having
some kind of problem IMO.


- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkoVe7oACgkQ9CaO5/Lv0PAbdQCeK088qxz+9/TNBnJgfHBg6Gyz
7+4AoI6fbXdn4Sh6kR9QaQhvK/jmHWpF
=frxU
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-21 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Vishwajit,

On 5/20/2009 3:01 PM, Pantvaidya, Vishwajit wrote:
 [Pantvaidya, Vishwajit] Ok so RUNNABLE i.e. persistent threads should
 not be an issue. The only reason why I thought that was an issue was
 that I was observing that the none of the RUNNABLE connections were
 not being used to serve new requests, only the WAITING ones were -
 and I do know for sure that the RUNNABLE threads were not servicing
 any existing requests as I was the only one using the system then.

It seems pretty clear that this is what your problem is. See if you can
follow the order of events described below:

1. Tomcat and Apache httpd are started. httpd makes one or more
   (persistent) AJP connections to Tomcat and holds them open (duh).
   Each connection from httpd-Tomcat puts a Java thread in the RUNNABLE
   state (though actually blocked on socket read, it's not really
   runnable)

2. Some requests are received by httpd and sent over the AJP connections
   to Tomcat (or not ... it really doesn't matter)

3. Time passes, your recycle_timeout (300s) or cache_timeout (600s)
   expires

4. A new request comes in to httpd destined for Tomcat. mod_jk dutifully
   follows your instructions for closing the connections expired in #3
   above (note that Tomcat has no idea that the connection has been
   closed, and so those threads remain in the RUNNABLE state, not
   connected to anything, lost forever)

5. A new connection (or multiple new connections... not sure exactly
   how mod_jks connection expiration-and-reconnect logic is done)
   is made to Tomcat which allocates a new thread (or threads)
   which is/are in the RUNNABLE state

Rinse, repeat, your server chokes to death when it runs out of threads.

The above description accounts for your loss of 4 threads at a time:
your web browser requests the initial page followed by 3 other assets
(image, css, whatever). Each one of them hits step #4 above, causing a
new AJP connection to be created, with the old one still hanging around
on the Tomcat side just wasting a thread and memory.

By setting connectionTimeout on the AJP Connector, you are /doing what
you should have done in the first place, which is match mod_jk
cache_timeout with Connector connectionTimeout/. This allows the threads
on the Tomcat side to expire just like those on the httpd side. They
should expire at (virtually) the same time and everything works as expected.

This problem is compounded by your initial configuration which created
10 connections from httpd-Tomcat for every (prefork) httpd process,
resulting in 9 useless AJP connections for every httpd process. I
suspect that you were expiring 10 connections at a time instead of just
one, meaning that you were running out of threads 10 times faster than
you otherwise would.

Suggestions:
1. Tell your ops guys we know what we're talking about
2. Upgrade mod_jk
3. Set connection_pool_size=1, or, better yet, remove the config
   altogether and let mod_jk determine its own value
4. Remove all timeouts unless you know that you have a misbehaving
   firewall. If you do, enable cping/cpong (the preferred strategy
   by at least one author or mod_jk)

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEUEARECAAYFAkoVidEACgkQ9CaO5/Lv0PCzwACYsrhskrNVgJFk6hI1gU+Kkmbe
WQCfTNbSLTgNtHcTbTAu5kw5igicNMw=
=0EWv
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-21 Thread Chetan Chheda

I am following this thread with great interest. I have a similar issue as 
Vishwajit and have resorted to adding the connectionTimeout to get rid of a 
large number of RUNNABLE threads. 

But mod_jk does not like tomcat threads timing out and logs the message 
increase the backend idle connection timeout or
 the connection_pool_minsize in the mod_jk logs

which leads me to believe that its apache thats not letting go of the threads 
in my case. 



From: Christopher Schultz ch...@christopherschultz.net
To: Tomcat Users List users@tomcat.apache.org
Sent: Thursday, May 21, 2009 1:05:21 PM
Subject: Re: Running out of tomcat threads - why many threads in RUNNABLEstage 
even with no activity

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Vishwajit,

On 5/20/2009 3:01 PM, Pantvaidya, Vishwajit wrote:
 [Pantvaidya, Vishwajit] Ok so RUNNABLE i.e. persistent threads should
 not be an issue. The only reason why I thought that was an issue was
 that I was observing that the none of the RUNNABLE connections were
 not being used to serve new requests, only the WAITING ones were -
 and I do know for sure that the RUNNABLE threads were not servicing
 any existing requests as I was the only one using the system then.

It seems pretty clear that this is what your problem is. See if you can
follow the order of events described below:

1. Tomcat and Apache httpd are started. httpd makes one or more
  (persistent) AJP connections to Tomcat and holds them open (duh).
  Each connection from httpd-Tomcat puts a Java thread in the RUNNABLE
  state (though actually blocked on socket read, it's not really
  runnable)

2. Some requests are received by httpd and sent over the AJP connections
  to Tomcat (or not ... it really doesn't matter)

3. Time passes, your recycle_timeout (300s) or cache_timeout (600s)
  expires

4. A new request comes in to httpd destined for Tomcat. mod_jk dutifully
  follows your instructions for closing the connections expired in #3
  above (note that Tomcat has no idea that the connection has been
  closed, and so those threads remain in the RUNNABLE state, not
  connected to anything, lost forever)

5. A new connection (or multiple new connections... not sure exactly
  how mod_jks connection expiration-and-reconnect logic is done)
  is made to Tomcat which allocates a new thread (or threads)
  which is/are in the RUNNABLE state

Rinse, repeat, your server chokes to death when it runs out of threads.

The above description accounts for your loss of 4 threads at a time:
your web browser requests the initial page followed by 3 other assets
(image, css, whatever). Each one of them hits step #4 above, causing a
new AJP connection to be created, with the old one still hanging around
on the Tomcat side just wasting a thread and memory.

By setting connectionTimeout on the AJP Connector, you are /doing what
you should have done in the first place, which is match mod_jk
cache_timeout with Connector connectionTimeout/. This allows the threads
on the Tomcat side to expire just like those on the httpd side. They
should expire at (virtually) the same time and everything works as expected.

This problem is compounded by your initial configuration which created
10 connections from httpd-Tomcat for every (prefork) httpd process,
resulting in 9 useless AJP connections for every httpd process. I
suspect that you were expiring 10 connections at a time instead of just
one, meaning that you were running out of threads 10 times faster than
you otherwise would.

Suggestions:
1. Tell your ops guys we know what we're talking about
2. Upgrade mod_jk
3. Set connection_pool_size=1, or, better yet, remove the config
  altogether and let mod_jk determine its own value
4. Remove all timeouts unless you know that you have a misbehaving
  firewall. If you do, enable cping/cpong (the preferred strategy
  by at least one author or mod_jk)

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEUEARECAAYFAkoVidEACgkQ9CaO5/Lv0PCzwACYsrhskrNVgJFk6hI1gU+Kkmbe
WQCfTNbSLTgNtHcTbTAu5kw5igicNMw=
=0EWv
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-21 Thread Pantvaidya, Vishwajit


 
 1) If you want to analyze your original problem, you need to get back to
 the original situation, i.e. without connectionTimeout. It doesn't make
 much sense to guess about the original problem by looking at something
 very different.

[Pantvaidya, Vishwajit] Yes - I have already initiated that. Expect to be able 
to test the system without connectionTimeout tonight.


 2) The output of netstat and the content of a thread dump change in
 time. If you want to understand the exact relations between the two
 netstat outputs and a thread dump, you need to ensure to produce those
 three things as close in time as possible. I'm talking about seconds not
 milliseconds.

[Pantvaidya, Vishwajit] Yes, I had taken the netstats and the threads dumps 
pretty close together. But I will try and append a timestamp next time I take 
the output.


 3) I think I already indicated that you do not want to look at entries
 in TIME_WAIT state. This state is special and not related to any threads

[Pantvaidya, Vishwajit] My netstat o/p had FIN_WAIT and CLOSE_WAIT, but not 
TIMED_WAIT. Did some reading on TCP states, and seems to me that next time I do 
a netstat (w/o the connTimeout in server.xml), I can ignore processes with all 
these wait states as they seem to just indicate connections in different stages 
of closing.


 4) Firewall idle connection drop: First read
 
 http://tomcat.apache.org/connectors-
 doc/generic_howto/timeouts.html#Firewall%20Connection%20Dropping
 
 carefully and try to understand.
 
 Any mod_jk attribute that takes a booelan value will accept 1, true,
 True, t or T as true, 0, false, False, f or F as false (and maybe even
 more).

[Pantvaidya, Vishwajit] Our workers.props file has most recommended settings:
Worker...type=ajp13
Worker...cachesize=10
Worker...cache_timeout=600
Worker...socket_keepalive=1
Worker...recycle_timeout=300

We are not setting connectionpoolsize and minsize - but from the timeouts doc 
that should be okay as JK auto adjusts to httpd poolsize. So I think only thing 
left is to remove connectionTimeout and test.

Your link recommends connectionTimeouts and JKoption +DisableReuse as a final 
option - I think that will remove persistent connections on httpd and tomcat 
side. For us, the connectionTimeout alone worked. And my netstat o/p showing 11 
conns on http side and only 2 on tomcat side means our http conn's are 
persistent while tomcat ones are not, right?. So I am thinking the perf 
downside should be better than if I had set +DisableReuse also?


 5) Matching port numbers
 
 Usually the port numbers should match. The non matching of the port
 numbers could indicate, that there is a firewall in between, although
 most firewall systems will be transparent to the ports (yes, I know
 there are many variations). Since the port numbers are very close I
 would guess, that the reason for not matching is that netstat was done a
 couple of seconds or more apart, and your connections are only used for
 a very short time, so we have lots of new connections activity.

[Pantvaidya, Vishwajit] Yes - I confirmed from admins that there is a firewall 
and I will work with them to understand that side more. Our connection timeout 
are in the order of 10 mins - so I am not sure why the port#s don't match - 
will try and find if the firewall is having different port# ranges configured 
for httpd  tomcat side.


 6) TCP states
 
 LISTEN on the Tomcat side corresponds to the one TP thread, that does a
 socket accept.
 
 ESTABLISHED: both sides still want to use this connection. On the Tomcat
 side shows up as socketRead0()
 
 CLOSE_WAIT: the other side has closed the connection, the local side
 hasn't yet. E.g. if Tomcat closes the connection because of
 connectionTimeout, but Apache doesn't have a configured idle timeout and
 didn't yet try to reuse the half-closed connection, the connection will
 be shown as CLOSE_WAIT on the httpd side. If Apache closed the
 connection, but Tomcat hasn't noticed yet, it will be CLOSE_WAIT at the
 Tomcat end. In this case it could be also socketRead0() in the thread
 dump.
 
 FIN_WAIT2: most likely the other end of CLOSE_WAIT.

[Pantvaidya, Vishwajit] This is interesting because all the conn's in the 
netstat o/p on httpd and tomcat sides which involved the connector port 21065 
(either in local/foreign addr) were in WAIT states. But I was seeing one 
RUNNABLE thread in socketAccept in the thread console. But anyway I will redo 
this whole thing with the conntimeouts removed and make sure I take the 
netstats and thread dumps in close succession.

 
 7) mod_jk update
 
 Before you start to fix your mod_jk configuration, go to your ops people
 and tell them that they are using a very bad mod_jk version and they
 have to update. The right version to update to is 1.2.28. It does make
 no sense at all to try to fix this with your old version. Solve your
 problem in the right way, by setting much more attributes on the JK side
 than simply the connectionTimeout

Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-21 Thread Rainer Jung

On 21.05.2009 20:59, Pantvaidya, Vishwajit wrote:
 3) I think I already indicated that you do not want to look at entries
 in TIME_WAIT state. This state is special and not related to any threads
 
 [Pantvaidya, Vishwajit] My netstat o/p had FIN_WAIT and CLOSE_WAIT, but not 
 TIMED_WAIT. Did some reading on TCP states, and seems to me that next time I 
 do a netstat (w/o the connTimeout in server.xml), I can ignore processes with 
 all these wait states as they seem to just indicate connections in different 
 stages of closing.

No, you can not ignore CLOSE_WAIT, FIN_WAIT, FIN_WAIT2 and maybe others.

 4) Firewall idle connection drop: First read

 http://tomcat.apache.org/connectors-
 doc/generic_howto/timeouts.html#Firewall%20Connection%20Dropping

 carefully and try to understand.

 Any mod_jk attribute that takes a booelan value will accept 1, true,
 True, t or T as true, 0, false, False, f or F as false (and maybe even
 more).
 
 [Pantvaidya, Vishwajit] Our workers.props file has most recommended settings:
 Worker...type=ajp13
 Worker...cachesize=10
 Worker...cache_timeout=600
 Worker...socket_keepalive=1
 Worker...recycle_timeout=300

It doesn't make sense for me checking values for attributes that do no
longer exist since a long time.

 We are not setting connectionpoolsize and minsize - but from the timeouts doc 
 that should be okay as JK auto adjusts to httpd poolsize. So I think only 
 thing left is to remove connectionTimeout and test.

No. First cachesize is equivalent to connection pool size. If you want
to get all idle connections closed you will also need the min pool size.
Also important are the cping/cpong features. All that applies to recent
versions.

 Your link recommends connectionTimeouts and JKoption +DisableReuse as a final 
 option - I think that will remove persistent connections on httpd and tomcat 
 side. For us, the connectionTimeout alone worked. And my netstat o/p showing 
 11 conns on http side and only 2 on tomcat side means our http conn's are 
 persistent while tomcat ones are not, right?. So I am thinking the perf 
 downside should be better than if I had set +DisableReuse also?

You would only use DisableReuse if the other configuration options do
not help. Usually they do help, so you won't use DisableReuse.

Regards,

Rainer

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-21 Thread Pantvaidya, Vishwajit

-Original Message-
From: Christopher Schultz [mailto:ch...@christopherschultz.net]
Sent: Thursday, May 21, 2009 10:05 AM
To: Tomcat Users List
Subject: Re: Running out of tomcat threads - why many threads in
RUNNABLEstage even with no activity

Vishwajit,

On 5/20/2009 3:01 PM, Pantvaidya, Vishwajit wrote:
[Pantvaidya, Vishwajit] Ok so RUNNABLE i.e. persistent threads should
not be an issue. The only reason why I thought that was an issue was
that I was observing that the none of the RUNNABLE connections were
not being used to serve new requests, only the WAITING ones were -
and I do know for sure that the RUNNABLE threads were not servicing
any existing requests as I was the only one using the system then.

It seems pretty clear that this is what your problem is. See if you can
follow the order of events described below:

1. Tomcat and Apache httpd are started. httpd makes one or more
(persistent) AJP connections to Tomcat and holds them open (duh).
Each connection from httpd-Tomcat puts a Java thread in the RUNNABLE
state (though actually blocked on socket read, it's not really
runnable)

2. Some requests are received by httpd and sent over the AJP connections
to Tomcat (or not ... it really doesn't matter)

3. Time passes, your recycle_timeout (300s) or cache_timeout (600s)
expires

4. A new request comes in to httpd destined for Tomcat. mod_jk dutifully
follows your instructions for closing the connections expired in #3
above (note that Tomcat has no idea that the connection has been
closed, and so those threads remain in the RUNNABLE state, not
connected to anything, lost forever)

5. A new connection (or multiple new connections... not sure exactly
how mod_jks connection expiration-and-reconnect logic is done)
is made to Tomcat which allocates a new thread (or threads)
which is/are in the RUNNABLE state

Rinse, repeat, your server chokes to death when it runs out of threads.

The above description accounts for your loss of 4 threads at a time:
your web browser requests the initial page followed by 3 other assets
(image, css, whatever). Each one of them hits step #4 above, causing a
new AJP connection to be created, with the old one still hanging around
on the Tomcat side just wasting a thread and memory.

By setting connectionTimeout on the AJP Connector, you are /doing what
you should have done in the first place, which is match mod_jk
cache_timeout with Connector connectionTimeout/. This allows the threads
on the Tomcat side to expire just like those on the httpd side. They
should expire at (virtually) the same time and everything works as
expected.

[Pantvaidya, Vishwajit] Thanks Chris - all this makes a lot of sense. However I
am not seeing same problem (tomcat running out of threads) on other servers
which are running exactly same configuration except that in those cases is no
firewall separating websvr and tomcat. Here are the figures of RUNNABLE on 3
different tomcat server running same config:

1. Firewall between httpd and tomcat - 120 threads, 112 runnable (93%)
2. No firewall between httpd and tomcat - 40 threads, 11 runnable (27%)
3. No firewall between httpd and tomcat - 48 threads, 2 runnable (4%)

Leads me to believe there is some firewall related mischief happening with #1.

This problem is compounded by your initial configuration which created
10 connections from httpd-Tomcat for every (prefork) httpd process,
resulting in 9 useless AJP connections for every httpd process. I
suspect that you were expiring 10 connections at a time instead of just
one, meaning that you were running out of threads 10 times faster than
you otherwise would.

[Pantvaidya, Vishwajit] I did not note connections expiring in multiple of 10.
But I will keep an eye out for this. However from the cachesize explanation at
http://tomcat.apache.org/connectors-doc/reference/workers.html#Deprecated%20Worker%20Directives
I get the impression that this value imposes an upper limit - meaning it may
not necessarily create 10 tomcat/jk connections for an httpd child process

Suggestions:
1. Tell your ops guys we know what we're talking about
2. Upgrade mod_jk
3. Set connection_pool_size=1, or, better yet, remove the config
altogether and let mod_jk determine its own value
4. Remove all timeouts unless you know that you have a misbehaving
firewall. If you do, enable cping/cpong (the preferred strategy
by at least one author or mod_jk)

- -chris

[Pantvaidya, Vishwajit] I will set
- cachesize=1 (doc says jk will autoset this value only for worker-mpm and we
use httpd 2.0 prefork)
- remove cache and recycle timeouts

But before all this, I will retest after removing connectionTimeout in
server.xml - just to test if there are firewall caused issues as mentioned
above.

-
To unsubscribe, e-mail: users

Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-21 Thread Rainer Jung

On 22.05.2009 00:19, Pantvaidya, Vishwajit wrote:
 [Pantvaidya, Vishwajit] I will set
 - cachesize=1 (doc says jk will autoset this value only for worker-mpm and we 
 use httpd 2.0 prefork)

You don't have to: JK will discover this number for the Apache web
server automatically and set the pool size to this value.

 -  remove cache and recycle timeouts

Chris and me are not having the same opinion here. You can choose :)

 But before all this, I will retest after removing connectionTimeout in 
 server.xml - just to test if there are firewall caused issues as mentioned 
 above.

Regards,

Rainer

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-21 Thread Pantvaidya, Vishwajit

 -Original Message-
 From: Rainer Jung [mailto:rainer.j...@kippdata.de]
 Sent: Thursday, May 21, 2009 3:37 PM
 To: Tomcat Users List
 Subject: Re: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 On 22.05.2009 00:19, Pantvaidya, Vishwajit wrote:
  [Pantvaidya, Vishwajit] I will set
  - cachesize=1 (doc says jk will autoset this value only for worker-mpm
 and we use httpd 2.0 prefork)

 You don't have to: JK will discover this number for the Apache web
 server automatically and set the pool size to this value.

[Pantvaidya, Vishwajit] Does what you say hold true for jk 1.2.15 also? Because 
I saw that for the 1.2.15 cachesize directive, 
http://tomcat.apache.org/connectors-doc/reference/workers.html#Deprecated%20Worker%20Directives
 says that JK will discover the number of threads per child process on Apache 
2 web server with worker-mpm and set its default value to match the 
ThreadsPerChild Apache directive.. Since we use pre-fork MPM, I assumed we 
need to set cachesize.

  -  remove cache and recycle timeouts

 Chris and me are not having the same opinion here. You can choose :)

[Pantvaidya, Vishwajit] I think that may be only because my adding the 
connectionTimeout led you to believe that I wanted nonpersistent conn's. Now 
that I know persistent connections are better, I am trying to rollback 
connectionTimeout - and then I guess you will agree with Chris that I need to 
rollback the recycletimeouts, etc in workers file on httpd side also?

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

 From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com]
 Subject: RE: Running out of tomcat threads - why many threads
 inRUNNABLEstage even with no activity

   - Setting connectionTimeout in server.xml seems to have resolved
   the issue

 Only because you're throwing away what appears to be a usable
 connection that's designed to be persistent.

Do you have something between Tomcat and httpd that could be silently closing 
connections?  (Some badly behaved firewalls are known to do this.)  That would 
make the existing AJP connections useless, without notifying the Tomcat threads 
that the connection is no longer there.  Setting the timeout would allow those 
connections to be discarded and new ones created.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity


  RUNNABLE and WAITING are thread states in the JVM. They don't relate in
  general to states inside Tomcat. In this special situation they do.
 
  The states you observe are both completely normal in themselves. One
  (the stack you abbreviate with RUNNABLE) is handling a persistant
  connection between web server and Tomcat which could send more requests,
  but at the moment no request is being processed, the other (you
  abbreviate with WAITING) is available to be associated with a new
  connection that might come in some time in the future.
 
 
 [Pantvaidya, Vishwajit] Thanks Rainer. The RUNNABLE thread - is it a
 connection between Tomcat and webserver, or between Tomcat and AJP? Is it
 still RUNNABLE and not WAITING because the servlet has not explicitly
 closed the connection yet (something like
 HttpServletResponse.getOutputStresm.close)
 
 
 [Pantvaidya, Vishwajit] My problem is that tomcat is running out of
 threads (maxthreadcount=200). My analysis of the issue is:
 - threads count is exceeded because of a slow buildup of RUNNABLE threads
 (and not because number of simultaneous http requests at some point
 exceeded max thread count)
 - most/all newly created TP-Processor threads are in RUNNABLE state and
 remain RUNNABLE - never go back to WAITING state (waiting for thread pool)
 - in such case, I find that tomcat spawns new threads when a new request
 comes in
 - this continues and finally tomcat runs out of threads
 - Setting connectionTimeout in server.xml seems to have resolved the issue
 - but I am wondering if that was just a workaround i.e. whether so many
 threads remaining RUNNABLE indicate a flaw in our webapp i.e. it not doing
 whatever's necessary to close them and return them to WAITING state.
 

[Pantvaidya, Vishwajit] After setting connectionTimeout in tomcat server.xml, 
the number of open threads is now consistently under 10 and most of them are 
now in WAITING stage. So looks like connectionTimeout also destroys idle 
threads. But I am still wondering - why should I have to set connectionTimeout 
to prevent tomcat running out of threads. I certainly don't mind if the 
TP-Processor threads continue to hang around as long as they are in WAITING 
state.
1. Is it expected behavior that most tomcat threads are in RUNNABLE state?
2. If not, does it indicate a problem in the app or in tomcat configuration?
My thinking is that the answer to #1 is no, and that to #2 is that it is an app 
problem. But just wanted to confirm and find out what people out there think.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads inRUNNABLEstage even with no activity

 From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com]
 Subject: RE: Running out of tomcat threads - why many threads
 inRUNNABLEstage even with no activity

  [Pantvaidya, Vishwajit] Thanks Rainer. The RUNNABLE thread - 
  is it a connection between Tomcat and webserver, or between 
  Tomcat and AJP?

It's not at all clear what you mean by that.  AJP is a protocol used by Tomcat 
and a front-end webserver, such as httpd.

  Is it still RUNNABLE and not WAITING because the servlet has 
  not explicitly closed the connection yet (something like
  HttpServletResponse.getOutputStresm.close)

No, the RUNNABLE is the normal state when the thread is not processing a 
request.  The connections between Tomcat and httpd are persistent - they're 
never meant to close.

  [Pantvaidya, Vishwajit] My problem is that tomcat is running out of
  threads (maxthreadcount=200).

Perhaps you provided some evidence earlier, but I missed it.  Having threads in 
a RUNNABLE state waiting for requests to arrive from httpd is *normal* and 
expected; it's *not* a problem.  You need to stop treating it like it's some 
terrible condition - it's how things should be.

  - Setting connectionTimeout in server.xml seems to have resolved 
  the issue

Only because you're throwing away what appears to be a usable connection that's 
designed to be persistent.

 1. Is it expected behavior that most tomcat threads are in 
 RUNNABLE state?

As we keep telling you, yes.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-20 Thread Rainer Jung

On 20.05.2009 19:47, Caldarale, Charles R wrote:
 From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com]
 Subject: RE: Running out of tomcat threads - why many threads
 inRUNNABLEstage even with no activity

 - Setting connectionTimeout in server.xml seems to have resolved
 the issue
 Only because you're throwing away what appears to be a usable
 connection that's designed to be persistent.

 Do you have something between Tomcat and httpd that could be silently closing 
 connections?  (Some badly behaved firewalls are known to do this.)  That 
 would make the existing AJP connections useless, without notifying the Tomcat 
 threads that the connection is no longer there.  Setting the timeout would 
 allow those connections to be discarded and new ones created.

That's a good point. You should check both sides by using netstat -an.
The Apache side and the Tomcat side (without connectionTimeout, so you
can see the problem in the original form). See whether the number of AJP
connections in the various TCP states differs much between the netstat
output on the Apache and on the Tomcat system.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

 
  [Pantvaidya, Vishwajit] My problem is that tomcat is running out of
 threads (maxthreadcount=200). My analysis of the issue is:
  - threads count is exceeded because of a slow buildup of RUNNABLE
 threads (and not because number of simultaneous http requests at some
 point exceeded max thread count)
 
 I don't belibve this reason. I would say thread count is exceeded,
 because you allow a much higher concurrency on the web server layer.
 
[Pantvaidya, Vishwajit] Is there a tool you can recommend for me to monitor/log 
the http requests so that I have figures to back up my analysis.


  - most/all newly created TP-Processor threads are in RUNNABLE state and
 remain RUNNABLE - never go back to WAITING state (waiting for thread pool)
 
 So you are using persistent connections. There's no *problem* with that
 per se. If you ae uncomfortable with it configure the timeouts in the
 Tomcat connector *and* mod_jk.
 
[Pantvaidya, Vishwajit] Ok so RUNNABLE i.e. persistent threads should not be an 
issue. The only reason why I thought that was an issue was that I was observing 
that the none of the RUNNABLE connections were not being used to serve new 
requests, only the WAITING ones were - and I do know for sure that the RUNNABLE 
threads were not servicing any existing requests as I was the only one using 
the system then.

  - in such case, I find that tomcat spawns new threads when a new request
 comes in
 
 request - connection
 
  - this continues and finally tomcat runs out of threads
 
 That's to simple, usually the new requests should be handled by
 existing Apache processes that already have a connection to Tomcat and
 will not create a new one.
 
[Pantvaidya, Vishwajit] In my case the existing persistent connections are not 
servicing any new requests.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

 -Original Message-
 From: Rainer Jung [mailto:rainer.j...@kippdata.de]
 Sent: Wednesday, May 20, 2009 11:53 AM
 To: Tomcat Users List
 Subject: Re: Running out of tomcat threads - why many threads in RUNNABLE
 stage even with no activity

 On 20.05.2009 19:47, Caldarale, Charles R wrote:
  From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com]
  Subject: RE: Running out of tomcat threads - why many threads
  inRUNNABLEstage even with no activity

  - Setting connectionTimeout in server.xml seems to have resolved
  the issue
  Only because you're throwing away what appears to be a usable
  connection that's designed to be persistent.

  Do you have something between Tomcat and httpd that could be silently
 closing connections?  (Some badly behaved firewalls are known to do this.)
 That would make the existing AJP connections useless, without notifying
 the Tomcat threads that the connection is no longer there.  Setting the
 timeout would allow those connections to be discarded and new ones
 created.

 That's a good point. You should check both sides by using netstat -an.
 The Apache side and the Tomcat side (without connectionTimeout, so you
 can see the problem in the original form). See whether the number of AJP
 connections in the various TCP states differs much between the netstat
 output on the Apache and on the Tomcat system.

[Pantvaidya, Vishwajit] Ok will do this.
To complicate things - we are seeing these outofthread problems only in one of 
our production servers - so I need to figure out if there are any differences 
in firewall settings between the 2 servers.

Finally, is it possible that some bad code in the app could be hanging onto 
those RUNNABLE connections which is why tomcat is not releasing them? Or if 
that was the case, would the stack trace of that thread show the code that was 
hanging onto it? In my case, all RUNNABLE connections show the stacktrace 

TP-Processor4 - Thread t...@29
   java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)...
at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:293)...
at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:595)

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-20 Thread Rainer Jung

On 20.05.2009 00:53, Pantvaidya, Vishwajit wrote:
 -Original Message-
 From: Rainer Jung [mailto:rainer.j...@kippdata.de]
 Sent: Monday, May 18, 2009 11:10 PM
 To: Tomcat Users List
 Subject: Re: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 On 19.05.2009 02:54, Caldarale, Charles R wrote:
 From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com]
 Subject: RE: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 Ok - so then the question is when does tomcat transition the thread
 from Running to Waiting? Does that happen after AJP drops that
 connection?
 RUNNABLE and WAITING are thread states in the JVM. They don't relate in
 general to states inside Tomcat. In this special situation they do.

 The states you observe are both completely normal in themselves. One
 (the stack you abbreviate with RUNNABLE) is handling a persistant
 connection between web server and Tomcat which could send more requests,
 but at the moment no request is being processed, the other (you
 abbreviate with WAITING) is available to be associated with a new
 connection that might come in some time in the future.

 [Pantvaidya, Vishwajit] Thanks Rainer. The RUNNABLE thread - is it a 
 connection between Tomcat and webserver, or between Tomcat and AJP? Is it 
 still RUNNABLE and not WAITING because the servlet has not explicitly closed 
 the connection yet (something like HttpServletResponse.getOutputStresm.close)

The thread handles a connection betwen the web server and Tomcat. AJP is
the protocol used on that connection.

It is runnable, because a socket read from inside the JVM puts a thread
into runnable state. The socket read is used to read the next request
and will block until data arrives over the established connection.

 So could the problem be occurring here because AJP is holding on to
 connections?
 Sorry, I haven't been following the thread that closely.  Not sure
 what the problem you're referring to actually is, but having a Tomcat
 thread reading input from the AJP connector is pretty normal.
 The same to me. What's the problem? AJP is designed to reuse connections
 (use persistent connections). If you do not want them to be used for a
 very long time or like those connections to be closed when being idle,
 you have to configure the appropriate timeouts. Look at the timeouts
 documentation page of mod_jk.

 In general your max thread numbers in the web server layer and in the
 Tomcat AJP pool need to be set consistently.

 [Pantvaidya, Vishwajit] My problem is that tomcat is running out of threads 
 (maxthreadcount=200). My analysis of the issue is:
 - threads count is exceeded because of a slow buildup of RUNNABLE threads 
 (and not because number of simultaneous http requests at some point exceeded 
 max thread count)

I don't belibve this reason. I would say thread count is exceeded,
because you allow a much higher concurrency on the web server layer.

 - most/all newly created TP-Processor threads are in RUNNABLE state and 
 remain RUNNABLE - never go back to WAITING state (waiting for thread pool)

So you are using persistent connections. There's no *problem* with that
per se. If you ae uncomfortable with it configure the timeouts in the
Tomcat connector *and* mod_jk.

 - in such case, I find that tomcat spawns new threads when a new request 
 comes in

request - connection

 - this continues and finally tomcat runs out of threads

That's to simple, usually the new requests should be handled by
existing Apache processes that already have a connection to Tomcat and
will not create a new one.

 - Setting connectionTimeout in server.xml seems to have resolved the issue - 
 but I am wondering if that was just a workaround i.e. whether so many threads 
 remaining RUNNABLE indicate a flaw in our webapp i.e. it not doing whatever's 
 necessary to close them and return them to WAITING state.

No it is a misconifguration of your web server, mod_jk and Tomcat. The
use of persistent AJP connections is opaque to the web application.

Regards,

Rainer

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

 From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com]
 Subject: RE: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 Finally, is it possible that some bad code in the app could 
 be hanging onto those RUNNABLE connections which is why tomcat
 is not releasing them?

Once more: NO, NO, NO!  The threads you see in a RUNNABLE state are perfectly 
normal and expected.  Go do the netstat that Rainer suggested and let us know 
what you see.  Stop fixating on the thread state.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

 
  Finally, is it possible that some bad code in the app could
  be hanging onto those RUNNABLE connections which is why tomcat
  is not releasing them?
 
 Once more: NO, NO, NO!  The threads you see in a RUNNABLE state are
 perfectly normal and expected.  Go do the netstat that Rainer suggested
 and let us know what you see.  Stop fixating on the thread state.
 
  - Chuck
 
 

[Pantvaidya, Vishwajit] Ok will do Chuck - thanks a lot for persisting with me 
through this issue.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

 -Original Message-
 From: Rainer Jung [mailto:rainer.j...@kippdata.de]
 Sent: Wednesday, May 20, 2009 11:53 AM
 To: Tomcat Users List
 Subject: Re: Running out of tomcat threads - why many threads in RUNNABLE
 stage even with no activity
 
 On 20.05.2009 19:47, Caldarale, Charles R wrote:
  From: Caldarale, Charles R [mailto:chuck.caldar...@unisys.com]
  Subject: RE: Running out of tomcat threads - why many threads
  inRUNNABLEstage even with no activity
 
  - Setting connectionTimeout in server.xml seems to have resolved
  the issue
  Only because you're throwing away what appears to be a usable
  connection that's designed to be persistent.
 
  Do you have something between Tomcat and httpd that could be silently
 closing connections?  (Some badly behaved firewalls are known to do this.)
 That would make the existing AJP connections useless, without notifying
 the Tomcat threads that the connection is no longer there.  Setting the
 timeout would allow those connections to be discarded and new ones
 created.
 
 That's a good point. You should check both sides by using netstat -an.
 The Apache side and the Tomcat side (without connectionTimeout, so you
 can see the problem in the original form). See whether the number of AJP
 connections in the various TCP states differs much between the netstat
 output on the Apache and on the Tomcat system.
 
[Pantvaidya, Vishwajit] My tomcat connector port is 21065. Netstat shows 
following output under Active Internet Connections:

On httpd machine

Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address   Foreign Address 
State
tcp0  0 0.0.0.0:25  0.0.0.0:*   
LISTEN
tcp1  0 129.41.29.243:43237 172.27.127.201:21065
CLOSE_WAIT
tcp1  0 129.41.29.243:43244 172.27.127.201:21065
CLOSE_WAIT
tcp1  0 129.41.29.243:43245 172.27.127.201:21065
CLOSE_WAIT
tcp1  0 129.41.29.243:43225 172.27.127.201:21065
CLOSE_WAIT
tcp1  0 129.41.29.243:43227 172.27.127.201:21065
CLOSE_WAIT
tcp0  0 129.41.29.243:43239 172.27.127.202:21069
ESTABLISHED
tcp0  0 129.41.29.243:43238 172.27.127.202:21069
ESTABLISHED
tcp0  0 129.41.29.243:43243 172.27.127.202:21069
ESTABLISHED
tcp0  0 129.41.29.243:43242 172.27.127.202:21069
ESTABLISHED
tcp0  0 129.41.29.243:43241 172.27.127.202:21069
ESTABLISHED
tcp0  0 129.41.29.243:43240 172.27.127.202:21069
ESTABLISHED
tcp0  0 129.41.29.243:43247 172.27.127.202:21069
ESTABLISHED
tcp0  0 129.41.29.243:43246 172.27.127.202:21069
ESTABLISHED
tcp0  0 129.41.29.243:43219 172.27.127.202:21069
TIME_WAIT
tcp0  0 129.41.29.243:43209 172.27.127.202:21069
ESTABLISHED
tcp0  0 129.41.29.243:43208 172.27.127.202:21069
TIME_WAIT


On tomcat machine

Proto Recv-Q Send-Q Local Address   Foreign Address 
State
tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43216  
FIN_WAIT2
tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43217  
FIN_WAIT2
tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43218  
FIN_WAIT2
tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43211  
TIME_WAIT
tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43212  
FIN_WAIT2
tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43213  
FIN_WAIT2
tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43214  
FIN_WAIT2
tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43215  
TIME_WAIT
tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43204  
TIME_WAIT
tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43205  
TIME_WAIT
tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43206  
TIME_WAIT


- My thread dump shows 8 TP-Processor threads - but this output has 11 threads.
- why do the 11 threads in the httpd o/p show port 21069 in foreign addr. Or 
are those not the correct threads I should be looking at?
- currently I do have connectionTimeout set in server.xml. I will need to wait 
until night to reset that.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-20 Thread André Warnier


Pantvaidya, Vishwajit wrote:

- why do the 11 threads in the httpd o/p show port 21069 in foreign addr. Or are those not the correct threads I should be looking at?a A bit off-topic maybe, but if you add -p to netstat (at least under 

Linux), it will also show the program that corresponds to that line.
That may help checking if you are looking at the correct ones.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

 From: André Warnier [mailto:a...@ice-sa.com]
 Subject: Re: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 if you add -p to netstat (at least under Linux), it will also 
 show the program that corresponds to that line.

Or at least -o to show the pid number.

- Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

 From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com]
 Subject: RE: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity
 
 On httpd machine
 Proto Recv-Q Send-Q Local Address   Foreign Address 
 State
 tcp1  0 129.41.29.243:43225 172.27.127.201:21065
 CLOSE_WAIT
 tcp1  0 129.41.29.243:43227 172.27.127.201:21065
 CLOSE_WAIT
 tcp1  0 129.41.29.243:43237 172.27.127.201:21065
 CLOSE_WAIT
 tcp1  0 129.41.29.243:43244 172.27.127.201:21065
 CLOSE_WAIT
 tcp1  0 129.41.29.243:43245 172.27.127.201:21065
 CLOSE_WAIT
 
 On tomcat machine
 Proto Recv-Q Send-Q Local Address   Foreign Address 
 State
 tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43204  
 TIME_WAIT
 tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43205  
 TIME_WAIT
 tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43206  
 TIME_WAIT
 tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43211  
 TIME_WAIT
 tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43212  
 FIN_WAIT2
 tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43213  
 FIN_WAIT2
 tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43214  
 FIN_WAIT2
 tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43215  
 TIME_WAIT
 tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43216  
 FIN_WAIT2
 tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43217  
 FIN_WAIT2
 tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43218  
 FIN_WAIT2

(The above was edited to remove irrelevant IP addresses and sort by port.)

The fact that *none* of the ports match would suggest (but not prove) that 
someone in the middle is closing the connections, and not telling either end 
about it.

 - why do the 11 threads in the httpd o/p show port 
 21069 in foreign addr.

They're for a different IP address.

 - currently I do have connectionTimeout set in server.xml.
 I will need to wait until night to reset that.

Do the netstat -anop again; it should be more interesting.

 - Chuck


THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

 The fact that *none* of the ports match would suggest (but not prove) that
 someone in the middle is closing the connections, and not telling either
 end about it.
 
 Do the netstat -anop again; it should be more interesting.
 
  - Chuck
 

[Pantvaidya, Vishwajit] Tomcat server port 11065, connector port 21065

On Httpd Side:

Proto Recv-Q Send-Q Local Address   Foreign Address 
State   PID/Program nameTimer
...
tcp0  0 0.0.0.0:25  0.0.0.0:*   
LISTEN  -   off (0.00/0/0)
tcp1  0 129.41.29.243:44003 172.27.127.201:21065
CLOSE_WAIT  -   keepalive (7194.80/0/0)
tcp1  0 129.41.29.243:44002 172.27.127.201:21065
CLOSE_WAIT  -   keepalive (7194.43/0/0)
tcp1  0 129.41.29.243:44001 172.27.127.201:21065
CLOSE_WAIT  -   keepalive (7192.26/0/0)
tcp1  0 129.41.29.243:44000 172.27.127.201:21065
CLOSE_WAIT  -   keepalive (7189.64/0/0)
tcp1  0 129.41.29.243:43990 172.27.127.201:21065
CLOSE_WAIT  -   keepalive (7016.23/0/0)
tcp1  0 129.41.29.243:43999 172.27.127.201:21065
CLOSE_WAIT  -   keepalive (7189.30/0/0)
tcp1  0 129.41.29.243:43998 172.27.127.201:21065
CLOSE_WAIT  -   keepalive (7186.76/0/0)
tcp1  0 129.41.29.243:43996 172.27.127.201:21065
CLOSE_WAIT  -   keepalive (7183.86/0/0)
tcp1  0 129.41.29.243:43994 172.27.127.201:21065
CLOSE_WAIT  -   keepalive (7174.09/0/0)
tcp1  0 129.41.29.243:43993 172.27.127.201:21065
CLOSE_WAIT  -   keepalive (7164.63/0/0)
...


On Tomcat side:

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address   Foreign Address 
State   PID/Program nameTimer
...
tcp0  0 :::21065:::*
LISTEN  6988/java   off (0.00/0/0)
tcp0  0 :::127.0.0.1:11065  :::*
LISTEN  6988/java   off (0.00/0/0)
tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43992  
FIN_WAIT2   -   timewait (56.71/0/0)
tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43991  
FIN_WAIT2   -   timewait (56.24/0/0)
...



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

  The fact that *none* of the ports match would suggest (but not prove)
 that
  someone in the middle is closing the connections, and not telling either
  end about it.
 
  Do the netstat -anop again; it should be more interesting.
 
   - Chuck
 
 
 [Pantvaidya, Vishwajit] Tomcat server port 11065, connector port 21065
 
 On Httpd Side:
 
 Proto Recv-Q Send-Q Local Address   Foreign Address
 State   PID/Program nameTimer
 ...
 tcp0  0 0.0.0.0:25  0.0.0.0:*
 LISTEN  -   off (0.00/0/0)
 tcp1  0 129.41.29.243:44003 172.27.127.201:21065
 CLOSE_WAIT  -   keepalive (7194.80/0/0)
 tcp1  0 129.41.29.243:44002 172.27.127.201:21065
 CLOSE_WAIT  -   keepalive (7194.43/0/0)
 tcp1  0 129.41.29.243:44001 172.27.127.201:21065
 CLOSE_WAIT  -   keepalive (7192.26/0/0)
 tcp1  0 129.41.29.243:44000 172.27.127.201:21065
 CLOSE_WAIT  -   keepalive (7189.64/0/0)
 tcp1  0 129.41.29.243:43990 172.27.127.201:21065
 CLOSE_WAIT  -   keepalive (7016.23/0/0)
 tcp1  0 129.41.29.243:43999 172.27.127.201:21065
 CLOSE_WAIT  -   keepalive (7189.30/0/0)
 tcp1  0 129.41.29.243:43998 172.27.127.201:21065
 CLOSE_WAIT  -   keepalive (7186.76/0/0)
 tcp1  0 129.41.29.243:43996 172.27.127.201:21065
 CLOSE_WAIT  -   keepalive (7183.86/0/0)
 tcp1  0 129.41.29.243:43994 172.27.127.201:21065
 CLOSE_WAIT  -   keepalive (7174.09/0/0)
 tcp1  0 129.41.29.243:43993 172.27.127.201:21065
 CLOSE_WAIT  -   keepalive (7164.63/0/0)
 ...
 
 
 On Tomcat side:
 
 (Not all processes could be identified, non-owned process info
  will not be shown, you would have to be root to see it all.)
 Active Internet connections (servers and established)
 Proto Recv-Q Send-Q Local Address   Foreign Address
 State   PID/Program nameTimer
 ...
 tcp0  0 :::21065:::*
 LISTEN  6988/java   off (0.00/0/0)
 tcp0  0 :::127.0.0.1:11065  :::*
 LISTEN  6988/java   off (0.00/0/0)
 tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43992
 FIN_WAIT2   -   timewait (56.71/0/0)
 tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43991
 FIN_WAIT2   -   timewait (56.24/0/0)
 ...
 
 
 
[Pantvaidya, Vishwajit] By the way, in the thread console, I see 8 TP-Processor 
threads (2 RUNNABLE, 6 WAITING). But above netstat output on tomcat side shows 
only 2 connections on port 21065. Shouldn't there be 1 thread / connection? 

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-20 Thread Martin Gainty


definitely not TC ..the problem is with your WebServer 
please read this tutorial on diagnosing CLOSE_WAIT sockets from your WebServer

http://publib.boulder.ibm.com/infocenter/wasinfo/v4r0/index.jsp?topic=/com.ibm.support.was40.doc/html/Plug_in/swg21163659.html

Martin
__ 
Verzicht und Vertraulichkeitanmerkung/Note de déni et de confidentialité
 
Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene Empfaenger 
sein, so bitten wir hoeflich um eine Mitteilung. Jede unbefugte Weiterleitung 
oder Fertigung einer Kopie ist unzulaessig. Diese Nachricht dient lediglich dem 
Austausch von Informationen und entfaltet keine rechtliche Bindungswirkung. 
Aufgrund der leichten Manipulierbarkeit von E-Mails koennen wir keine Haftung 
fuer den Inhalt uebernehmen.
Ce message est confidentiel et peut être privilégié. Si vous n'êtes pas le 
destinataire prévu, nous te demandons avec bonté que pour satisfaire informez 
l'expéditeur. N'importe quelle diffusion non autorisée ou la copie de ceci est 
interdite. Ce message sert à l'information seulement et n'aura pas n'importe 
quel effet légalement obligatoire. Étant donné que les email peuvent facilement 
être sujets à la manipulation, nous ne pouvons accepter aucune responsabilité 
pour le contenu fourni.




 From: vpant...@selectica.com
 To: users@tomcat.apache.org
 Date: Wed, 20 May 2009 15:17:15 -0700
 Subject: RE: Running out of tomcat threads - why many threads in  
 RUNNABLEstage even with no activity
 
   The fact that *none* of the ports match would suggest (but not prove)
  that
   someone in the middle is closing the connections, and not telling either
   end about it.
  
   Do the netstat -anop again; it should be more interesting.
  
- Chuck
  
  
  [Pantvaidya, Vishwajit] Tomcat server port 11065, connector port 21065
  
  On Httpd Side:
  
  Proto Recv-Q Send-Q Local Address   Foreign Address
  State   PID/Program nameTimer
  ...
  tcp0  0 0.0.0.0:25  0.0.0.0:*
  LISTEN  -   off (0.00/0/0)
  tcp1  0 129.41.29.243:44003 172.27.127.201:21065
  CLOSE_WAIT  -   keepalive (7194.80/0/0)
  tcp1  0 129.41.29.243:44002 172.27.127.201:21065
  CLOSE_WAIT  -   keepalive (7194.43/0/0)
  tcp1  0 129.41.29.243:44001 172.27.127.201:21065
  CLOSE_WAIT  -   keepalive (7192.26/0/0)
  tcp1  0 129.41.29.243:44000 172.27.127.201:21065
  CLOSE_WAIT  -   keepalive (7189.64/0/0)
  tcp1  0 129.41.29.243:43990 172.27.127.201:21065
  CLOSE_WAIT  -   keepalive (7016.23/0/0)
  tcp1  0 129.41.29.243:43999 172.27.127.201:21065
  CLOSE_WAIT  -   keepalive (7189.30/0/0)
  tcp1  0 129.41.29.243:43998 172.27.127.201:21065
  CLOSE_WAIT  -   keepalive (7186.76/0/0)
  tcp1  0 129.41.29.243:43996 172.27.127.201:21065
  CLOSE_WAIT  -   keepalive (7183.86/0/0)
  tcp1  0 129.41.29.243:43994 172.27.127.201:21065
  CLOSE_WAIT  -   keepalive (7174.09/0/0)
  tcp1  0 129.41.29.243:43993 172.27.127.201:21065
  CLOSE_WAIT  -   keepalive (7164.63/0/0)
  ...
  
  
  On Tomcat side:
  
  (Not all processes could be identified, non-owned process info
   will not be shown, you would have to be root to see it all.)
  Active Internet connections (servers and established)
  Proto Recv-Q Send-Q Local Address   Foreign Address
  State   PID/Program nameTimer
  ...
  tcp0  0 :::21065:::*
  LISTEN  6988/java   off (0.00/0/0)
  tcp0  0 :::127.0.0.1:11065  :::*
  LISTEN  6988/java   off (0.00/0/0)
  tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43992
  FIN_WAIT2   -   timewait (56.71/0/0)
  tcp0  0 :::172.27.127.201:21065 :::129.41.29.243:43991
  FIN_WAIT2   -   timewait (56.24/0/0)
  ...
  
  
  
 [Pantvaidya, Vishwajit] By the way, in the thread console, I see 8 
 TP-Processor threads (2 RUNNABLE, 6 WAITING). But above netstat output on 
 tomcat side shows only 2 connections on port 21065. Shouldn't there be 1 
 thread / connection? 
 
 -
 To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
 For additional commands, e-mail: users-h...@tomcat.apache.org
 

_
Hotmail® has ever-growing storage! Don’t worry about storage limits.
http://windowslive.com/Tutorial/Hotmail/Storage?ocid=TXT_TAGLM_WL_HM_Tutorial_Storage1_052009

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

 From: Martin Gainty [mailto:mgai...@hotmail.com]
 Subject: RE: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 http://publib.boulder.ibm.com/infocenter/wasinfo/v4r0/index.jsp?topic=/
 com.ibm.support.was40.doc/html/Plug_in/swg21163659.html

Vaguely interesting, but not really applicable.  The protocol in question here 
is AJP, not HTTP, and the connections are expected to be persistent, with no 
keep-alives needed.  Unlike in the article cited, neither end will close the 
connection (unless an error occurs, of course).  However, when there's an 
ill-behaved firewall in between Tomcat and the http server (as we appear to 
have here), keep-alives may be required.

This FAQ entry looks promising:
http://tomcat.apache.org/connectors-doc/miscellaneous/faq.html

Look at the entry entitled I've got a firewall between my web server and 
Tomcat which drops ajp13 connections after some time.

Configuring keep-alives is a fairly low-overhead workaround, but it would be 
better to fix the firewall so it doesn't silently drop connections.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

 This FAQ entry looks promising:
 http://tomcat.apache.org/connectors-doc/miscellaneous/faq.html
 
 Look at the entry entitled I've got a firewall between my web server and
 Tomcat which drops ajp13 connections after some time.
 
 Configuring keep-alives is a fairly low-overhead workaround, but it would
 be better to fix the firewall so it doesn't silently drop connections.
 
  - Chuck
 
[Pantvaidya, Vishwajit] Thanks Chuck. My workers.properties already has 
following settings:

port=21065
cachesize=10
cache_timeout=600
socket_keepalive=1
recycle_timeout=300

So socket_keepalive is already 1. So does this mean that firewall is dropping 
connections in spite of it.

About the netstat output I sent earlier - I guess an indicator of a firewall 
dropping connections, would be if the output showed many more active 
connections on the tomcat side than on the httpd side - is that accurate?
My netstat o/p had only 2 tomcat connections active in FIN_WAIT2 and about 11 
in keepalive on httpd side - I guess this does not indicate any hanging 
connections? Could that be because currently connectionTimeout is active in my 
server.xml?


-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

 From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com]
 Subject: RE: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 So socket_keepalive is already 1. So does this mean that firewall is
 dropping connections in spite of it.

The doc does not mention using 1 here, just true (although other variables 
allow either).  Would be best to get Rainer's opinion when the sun comes up in 
Europe.

 My netstat o/p had only 2 tomcat connections active in FIN_WAIT2 and
 about 11 in keepalive on httpd side - I guess this does not indicate
 any hanging connections?

It's not what one would like to see, especially since none of the ports match - 
all of the connections are broken.

 Could that be because currently connectionTimeout is 
 active in my server.xml?

I think so; it appears that setting the connectionTimeout on the Tomcat side 
will effectively disable the expected persistence; it's an expensive workaround 
for the problem.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-19 Thread Rainer Jung

On 19.05.2009 02:54, Caldarale, Charles R wrote:
 From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] 
 Subject: RE: Running out of tomcat threads - why many threads in 
 RUNNABLEstage even with no activity

 Ok - so then the question is when does tomcat transition the thread
 from Running to Waiting? Does that happen after AJP drops that
 connection?

RUNNABLE and WAITING are thread states in the JVM. They don't relate in
general to states inside Tomcat. In this special situation they do.

The states you observe are both completely normal in themselves. One
(the stack you abbreviate with RUNNABLE) is handling a persistant
connection between web server and Tomcat which could send more requests,
but at the moment no request is being processed, the other (you
abbreviate with WAITING) is available to be associated with a new
connection that might come in some time in the future.

 That's my understanding; I would presume some from of keep-alive is
 in play.  However, others know the AJP characteristics better than I

HTTP Keep-Alive does not change the picture. It's transparent to Tomcat
and mod_jk. Those Keep-Alice packets do not count as requests.

 do.  Rainer is the ultimate resource, but I suspect he's asleep right
 now.

 So could the problem be occurring here because AJP is holding on to
 connections?

 Sorry, I haven't been following the thread that closely.  Not sure
 what the problem you're referring to actually is, but having a Tomcat
 thread reading input from the AJP connector is pretty normal.

The same to me. What's the problem? AJP is designed to reuse connections
(use persistent connections). If you do not want them to be used for a
very long time or like those connections to be closed when being idle,
you have to configure the appropriate timeouts. Look at the timeouts
documentation page of mod_jk.

In general your max thread numbers in the web server layer and in the
Tomcat AJP pool need to be set consistently.

Regards,

Rainer

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-19 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Chuck,

On 5/18/2009 8:54 PM, Caldarale, Charles R wrote:
 From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com] 
 Subject: RE: Running out of tomcat threads - why many threads in 
 RUNNABLEstage even with no activity

 Ok - so then the question is when does tomcat transition the thread
 from Running to Waiting? Does that happen after AJP drops that
 connection?

 That's my understanding; I would presume some from of keep-alive is
 in play.  However, others know the AJP characteristics better than I
 do.  Rainer is the ultimate resource, but I suspect he's asleep right
 now.

My expectation would be that an AJP connection waiting for the next
request in a set of keepalive requests would be WAITING: blocked on a
socket read, rather than RUNNABLE. Or, maybe Java's thread states don't
differentiate between actually runnable and runnable but blocked (as
opposed to WAITING which means waiting on a synchronization monitor).

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkoSpdcACgkQ9CaO5/Lv0PAcnwCgq33fogBqYaYD5INtQk8D/x7d
RewAn23Ft0nSsgSQeupKhuanWdlwsIsS
=jHcT
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstageeven with no activity

2009-05-19 Thread Caldarale, Charles R

 From: Christopher Schultz [mailto:ch...@christopherschultz.net]
 Subject: Re: Running out of tomcat threads - why many threads in
 RUNNABLEstageeven with no activity

 Or, maybe Java's thread states don't differentiate between 
 actually runnable and runnable but blocked (as opposed to
 WAITING which means waiting on a synchronization monitor).

Correct; there is no BLOCKED state from the point of view of a Java thread.  
When in native code, such as reading from a socket or any other JNI method, the 
thread is still considered RUNNABLE, since the VM has no control over when it 
might be executing.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-19 Thread Pantvaidya, Vishwajit

  Yes, sure - we will upgrade at some point of
  time. But since upgrading all our servers will be some work, that may
  not happen right away.
 
 Upgrading mod_jk is the least painful of all of these, and the most
 likely to affect you.
 
[Pantvaidya, Vishwajit] I understand and agree and will push for this - but 
most admins are conservative and so I am not harboring high hopes for success 
in this. 

  Here are figures from 3 of the servers which
  have not yet run out of threads (so the thread count does not add up
  to 200). I have taken these late at night when no users are present,
  so I was expecting all threads to be Waiting for tomcat thread-pool.
 
  1. Total TP-Processor threads 48, Waiting 46, Runnable 2
  2. Total TP-Processor threads 40, Waiting 29, Runnable 11
  3. Total TP-Processor threads 120, Waiting 7, Runnable 113
 
 Are you sure you aren't seeing any traffic, even that late at night?
 What if you watch the access logs? Are there requests actively being
 serviced?
 
[Pantvaidya, Vishwajit] I was tailing the logs - there were no accesses.

  Do you think this could be because of the
  application? I was under the impression that there is some tomcat
  config parameter that controls this - which was set to 4.
 
 No, Tomcat uses precisely 1 thread to handle each incoming HTTP request.
 If keepalives are used, multiple requests may be handled by the same
 thread before it is returned to the pool, or different threads may be
 used to serve different requests from the single connection, but in
 either case, no more than 1 thread will be used to service a single HTTP
 request.
 

[Pantvaidya, Vishwajit] Could this happen if upon my http browser request, the 
app could be spawning multiple redirects in quick succession leading tomcat to 
create multiple threads. Any other thoughts why I could be seeing tomcat spawn 
threads in multiples of 4?

  My workers config is:
 
  Worker...type=ajp13
  Worker...cachesize=10
  Are you using the prefork MPM? If so, cachesize should be /1/.
 
  [Pantvaidya, Vishwajit] Could you please elaborate. What is the
  prefork MPM?
 
 The MPM is the concurrency strategy employed by Apache httpd. Either you
 are using the prefork MPM which starts multiple httpd processes to
 handle requests, or you are using the worker MPM which starts multiple
 threads to handle requests. Actually, mod_jk should be able to
 auto-detect the appropriate cachesize (called connection_pool_size,
 now), so you shouldn't have to set this.
 
[Pantvaidya, Vishwajit] Ok thanks. httpd -l showed perfork.c. I guess that 
means we are using prefork MPM. So our cachesize should be 1? Our mod_jk 
version is 1.2.15 - will that also auto-detect the cache-size?


  Worker...cache_timeout=600 Worker...socket_keepalive=1
  Worker...recycle_timeout=300
  Are these timeouts necessary? Why not simply let the connections
  stay alive all the time?
 
  [Pantvaidya, Vishwajit] Sure we could. But for any production change,
  I would have to offer a good enough reason.
 
 What was the good enough reason to set those timeouts in the first
 place?
 
[Pantvaidya, Vishwajit] I agree - but again as I mentioned above because the 
admin will be conservative about any changes, I need to have a strong reason.
Also when you say let the connection stay alive, does that mean let the 
TP-Processor thread remain in Waiting state / Runnable state?

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-19 Thread Pantvaidya, Vishwajit

 -Original Message-
 From: Rainer Jung [mailto:rainer.j...@kippdata.de]
 Sent: Monday, May 18, 2009 11:10 PM
 To: Tomcat Users List
 Subject: Re: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 On 19.05.2009 02:54, Caldarale, Charles R wrote:
  From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com]
  Subject: RE: Running out of tomcat threads - why many threads in
  RUNNABLEstage even with no activity

  Ok - so then the question is when does tomcat transition the thread
  from Running to Waiting? Does that happen after AJP drops that
  connection?

 RUNNABLE and WAITING are thread states in the JVM. They don't relate in
 general to states inside Tomcat. In this special situation they do.

 The states you observe are both completely normal in themselves. One
 (the stack you abbreviate with RUNNABLE) is handling a persistant
 connection between web server and Tomcat which could send more requests,
 but at the moment no request is being processed, the other (you
 abbreviate with WAITING) is available to be associated with a new
 connection that might come in some time in the future.

[Pantvaidya, Vishwajit] Thanks Rainer. The RUNNABLE thread - is it a connection 
between Tomcat and webserver, or between Tomcat and AJP? Is it still RUNNABLE 
and not WAITING because the servlet has not explicitly closed the connection 
yet (something like HttpServletResponse.getOutputStresm.close)

  So could the problem be occurring here because AJP is holding on to
  connections?

  Sorry, I haven't been following the thread that closely.  Not sure
  what the problem you're referring to actually is, but having a Tomcat
  thread reading input from the AJP connector is pretty normal.

 The same to me. What's the problem? AJP is designed to reuse connections
 (use persistent connections). If you do not want them to be used for a
 very long time or like those connections to be closed when being idle,
 you have to configure the appropriate timeouts. Look at the timeouts
 documentation page of mod_jk.

 In general your max thread numbers in the web server layer and in the
 Tomcat AJP pool need to be set consistently.

[Pantvaidya, Vishwajit] My problem is that tomcat is running out of threads 
(maxthreadcount=200). My analysis of the issue is:
- threads count is exceeded because of a slow buildup of RUNNABLE threads (and 
not because number of simultaneous http requests at some point exceeded max 
thread count)
- most/all newly created TP-Processor threads are in RUNNABLE state and remain 
RUNNABLE - never go back to WAITING state (waiting for thread pool)
- in such case, I find that tomcat spawns new threads when a new request comes 
in
- this continues and finally tomcat runs out of threads
- Setting connectionTimeout in server.xml seems to have resolved the issue - 
but I am wondering if that was just a workaround i.e. whether so many threads 
remaining RUNNABLE indicate a flaw in our webapp i.e. it not doing whatever's 
necessary to close them and return them to WAITING state.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

Hi Chris,

Thanks for your reply.

 
 On 5/13/2009 5:28 PM, Pantvaidya, Vishwajit wrote:
  My setup is tomcat 5.5.17 + mod_jk 1.2.15 + httpd 2.2.2. I am using
  AJP1.3.
 
 Old versions of everything. Consider upgrading?
 
[Pantvaidya, Vishwajit] Yes, sure - we will upgrade at some point of time. But 
since upgrading all our servers will be some work, that may not happen right 
away.


  Every 2-3 days with no major load, tomcat throws the error: SEVERE:
  All threads (200) are currently busy, waiting...
 
  I have been monitoring my tomcat TP-Processor thread behavior over
  extended time intervals and observe that: - even when there is no
  activity on the server, several TP-Processor threads are in RUNNABLE
  state while few are in WAITING state
 
 It appears that you have 200 threads available. How many (on average)
 are RUNNABLE versus WAITING? (The two counts should add to 200, unless
 there's some other state (BLOCKED?) that the threads can be in, but you
 didn't mention any other states).

[Pantvaidya, Vishwajit] Here are figures from 3 of the servers which have not 
yet run out of threads (so the thread count does not add up to 200). I have 
taken these late at night when no users are present, so I was expecting all 
threads to be Waiting for tomcat thread-pool.

1. Total TP-Processor threads 48, Waiting 46, Runnable 2
2. Total TP-Processor threads 40, Waiting 29, Runnable 11
3. Total TP-Processor threads 120, Waiting 7, Runnable 113


 
  - RUNNABLE threads stack trace shows java.lang.Thread.State:
  RUNNABLE at java.net.SocketInputStream.socketRead0(Native
  Method)...
 
 This indicates that the client has not yet disconnected, and is probably
 still sending data. If there were not any data waiting, the state should
 be BLOCKED.
 
  - WAITING thread stack trace shows java.lang.Thread.State: WAITING
  on
  org.apache.tomcat.util.threads.threadpool$controlrunna...@53533c55
 
 These are idle threads.
 
  - tomcat adds 4 new TP-Processor threads when a request comes in and
  it can find no WAITING threads
 
 Wow, 4 new threads? That seems like 3 too many...

[Pantvaidya, Vishwajit] Do you think this could be because of the application? 
I was under the impression that there is some tomcat config parameter that 
controls this - which was set to 4.

 
  So I conclude that my tomcat is running out of threads due to many
  threads being in RUNNABLE state when actually they should be in
  WAITING state. Is that happening because of the socket_keepalive in
  my workers.properties shown below?
 
 worker.socket_keepalive just keeps the connection between Apache httpd
 and Tomcat alive in case you have an overzealous firewall that closes
 inactive connections. The request processor shouldn't be affected by
 this setting.
 
  Why are threads added in bunches of 4 - is there any way to configure
  this?
 
  My workers config is:
 
  Worker...type=ajp13 Worker...cachesize=10
 
 Are you using the prefork MPM? If so, cachesize should be /1/.
 

[Pantvaidya, Vishwajit] Could you please elaborate. What is the prefork MPM?


  Worker...cache_timeout=600 Worker...socket_keepalive=1
  Worker...recycle_timeout=300
 
 Are these timeouts necessary? Why not simply let the connections stay
 alive all the time?
 
[Pantvaidya, Vishwajit] Sure we could. But for any production change, I would 
have to offer a good enough reason.

  Earlier posts related to this issue on the list seem to recommend
  tweaking: - several timeouts - JkOptions +DisableReuse
 
 This will require that every incoming HTTP connection opens up a new
 ajp13 connection to Tomcat. Your performance will totally suck if you
 enable this. But if it's the only way for you to get your application
 working properly, then I guess you'll have to do it. I suspect you /will
 not/ have to enable +DisableReuse.
 
[Pantvaidya, Vishwajit] I was seeing earlier posts on this list mention some 
disagreement on the performance impact of setting +DisableReuse. Otherwise I 
would not even think of this.

By the way, the above 3 figures I provided are without connectiontimeout being 
set for Connector element in server.xml.

- Vish.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-18 Thread Rainer Jung

On 13.05.2009 23:28, Pantvaidya, Vishwajit wrote:
 My setup is tomcat 5.5.17 + mod_jk 1.2.15 + httpd 2.2.2. I am using 
 AJP1.3. Every 2-3 days with no major load, tomcat throws the error: 
 SEVERE: All threads (200) are currently busy, waiting...
 
 I have been monitoring my tomcat TP-Processor thread behavior over 
 extended time intervals and observe that: - even when there is no 
 activity on the server, several TP-Processor threads are in RUNNABLE 
 state while few are in WAITING state - RUNNABLE threads stack trace 
 shows java.lang.Thread.State: RUNNABLE at 
 java.net.SocketInputStream.socketRead0(Native Method)...

We would need to see more of the stack. It's likely that those are
connected to the web server, waiting for the next request.

 - WAITING thread stack trace shows java.lang.Thread.State: WAITING 
 on 
 org.apache.tomcat.util.threads.threadpool$controlrunna...@53533c55

Likely idle in the pool, availale to handle new connections.

 - tomcat adds 4 new TP-Processor threads when a request comes in and 
 it can find no WAITING threads
 
 So I conclude that my tomcat is running out of threads due to many 
 threads being in RUNNABLE state when actually they should be in 
 WAITING state. Is that happening because of the socket_keepalive in 
 my workers.properties shown below? Why are threads added in bunches 
 of 4 - is there any way to configure this?

Those socketRead0 threads (Disclaimer: I already said we need more of
the stack to be sure) are connected to the web server, waiting for new
requests. As long as the new requests come from one of theose web server
processes, no new thread is needed to handle them.

socket_keepalive is not directly related to that. It tries to workaround
a problem, where some component (e.g. firewall) between web server and
Tomcat cuts an idle connection without letting the web server and Tomcat
know.

If you want to free the thread handling the persistent connections, you
caqn use the connection pool timeout on the jk side and also the
connection pool minimum size (e.g. setting it to 0).

On the Tomcat side use connetionTimeout. Be warned, that jk and Tomcat
do not use the same time unit for those parameters. Have a look at the
timeouts documentation of mod_jk.

 My workers config is:
 
 Worker...type=ajp13 Worker...cachesize=10 Worker...cache_timeout=600
  Worker...socket_keepalive=1 Worker...recycle_timeout=300
 
 Earlier posts related to this issue on the list seem to recommend 
 tweaking: - several timeouts - JkOptions +DisableReuse

Very last resort. Should not be needed and might obscure some other problem.

 I am planning to do the following to resolve our problem: - upgrade 
 jk to latest version - e.g. 1.2.28 - replace recycle_timeout with 
 connection_pool_timeout - add connectionTimeout in server.xml - add 
 JkOptions +DisableReuse
 
 Please let me know if this is okay or suggestions if any.

I suspect, that during the time of the all threads are currently busy
message, something in or behind your app was slow and so requests got
stuck in front of Tomcat, the web server pool was growing until it
connected 200 web server processes/threads trying to send requests to
Tomcat. To find out what the root cause was, you'll need to make the
thread dumps during the problem time.

Also note, that the maximum concurrency in your web server layer should
be a good fit to the maximum concurrency (thread pool size) in the
Tomcat layer.

Regards,

Rainer

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-18 Thread Rainer Jung

On 14.05.2009 20:29, Pantvaidya, Vishwajit wrote:
 I set connectionTimeout in server.xml to 60 and now the RUNNABLE
 threads go back to WAITING stage after that time.
 
 But our other servers which are running the same configuration, same
 webapp and do not have connectionTimeout set in server.xml, do not
 show so many RUNNABLE threads, but more WAITING threads. So looks
 like the threads are getting recycled properly there.
 
 Any idea why could this be? Could it be the OS (all servers run Linux
 but I do not know which flavors/versions)?

Different web server configuration? Different load specifics?

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-18 Thread Christopher Schultz

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Vishwajit,

On 5/18/2009 2:01 PM, Pantvaidya, Vishwajit wrote:
 Yes, sure - we will upgrade at some point of
 time. But since upgrading all our servers will be some work, that may
 not happen right away.

Upgrading mod_jk is the least painful of all of these, and the most
likely to affect you.

 Here are figures from 3 of the servers which
 have not yet run out of threads (so the thread count does not add up
 to 200). I have taken these late at night when no users are present,
 so I was expecting all threads to be Waiting for tomcat thread-pool.
 
 1. Total TP-Processor threads 48, Waiting 46, Runnable 2
 2. Total TP-Processor threads 40, Waiting 29, Runnable 11
 3. Total TP-Processor threads 120, Waiting 7, Runnable 113

Are you sure you aren't seeing any traffic, even that late at night?
What if you watch the access logs? Are there requests actively being
serviced?

 Do you think this could be because of the
 application? I was under the impression that there is some tomcat
 config parameter that controls this - which was set to 4.

No, Tomcat uses precisely 1 thread to handle each incoming HTTP request.
If keepalives are used, multiple requests may be handled by the same
thread before it is returned to the pool, or different threads may be
used to serve different requests from the single connection, but in
either case, no more than 1 thread will be used to service a single HTTP
request.

 My workers config is:
 
 Worker...type=ajp13
 Worker...cachesize=10
 Are you using the prefork MPM? If so, cachesize should be /1/.
 
 [Pantvaidya, Vishwajit] Could you please elaborate. What is the
 prefork MPM?

The MPM is the concurrency strategy employed by Apache httpd. Either you
are using the prefork MPM which starts multiple httpd processes to
handle requests, or you are using the worker MPM which starts multiple
threads to handle requests. Actually, mod_jk should be able to
auto-detect the appropriate cachesize (called connection_pool_size,
now), so you shouldn't have to set this.

 Worker...cache_timeout=600 Worker...socket_keepalive=1 
 Worker...recycle_timeout=300
 Are these timeouts necessary? Why not simply let the connections
 stay alive all the time?
 
 [Pantvaidya, Vishwajit] Sure we could. But for any production change,
 I would have to offer a good enough reason.

What was the good enough reason to set those timeouts in the first place?

 Earlier posts related to this issue on the list seem to
 recommend tweaking: - several timeouts - JkOptions +DisableReuse
 This will require that every incoming HTTP connection opens up a
 new ajp13 connection to Tomcat. Your performance will totally suck
 if you enable this. But if it's the only way for you to get your
 application working properly, then I guess you'll have to do it. I
 suspect you /will not/ have to enable +DisableReuse.
 
 [Pantvaidya, Vishwajit] I was seeing earlier posts on this list
 mention some disagreement on the performance impact of setting
 +DisableReuse. Otherwise I would not even think of this.

+DisableReuse will, without a doubt, decrease your performance.

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAkoRufgACgkQ9CaO5/Lv0PBGHwCfaOGjjv8A1wOAO0CrBKiMbVhM
9MEAoKD8QUbxfqd4h/8YcppkKAt2J1qM
=+kJC
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity


 [Pantvaidya, Vishwajit] Here are figures from 3 of the servers which have
 not yet run out of threads (so the thread count does not add up to 200). I
 have taken these late at night when no users are present, so I was
 expecting all threads to be Waiting for tomcat thread-pool.
 
 1. Total TP-Processor threads 48, Waiting 46, Runnable 2
 2. Total TP-Processor threads 40, Waiting 29, Runnable 11
 3. Total TP-Processor threads 120, Waiting 7, Runnable 113
 
 
 

[Pantvaidya, Vishwajit] Posting the thread dumps for the above 3 cases, since 
Rainer mentioned that he would like to see more of the stack trace.



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-18 Thread Caldarale, Charles R

 From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com]
 Subject: RE: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 [Pantvaidya, Vishwajit] Posting the thread dumps for the above 3 cases,

The list usually filters out attachments, as it has done with yours.  Either 
put them on a publicly accessible web site, or right in the text of the e-mail.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

  [Pantvaidya, Vishwajit] Here are figures from 3 of the servers which

 have

  not yet run out of threads (so the thread count does not add up to 200).

 I

  have taken these late at night when no users are present, so I was

  expecting all threads to be Waiting for tomcat thread-pool.

 

  1. Total TP-Processor threads 48, Waiting 46, Runnable 2

  2. Total TP-Processor threads 40, Waiting 29, Runnable 11

  3. Total TP-Processor threads 120, Waiting 7, Runnable 113

 

 

[Pantvaidya, Vishwajit] Since Rainer mentioned that he would like to see more 
of the stack trace, here are the complete stack traces for a Runnable and 
Waiting thread from #3 above. All Runnable/Waiting threads from all the above 
cases have same stack trace as below:



TP-Processor119 - Thread t...@2294

   java.lang.Thread.State: RUNNABLE

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(SocketInputStream.java:129)

at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)

at java.io.BufferedInputStream.read(BufferedInputStream.java:313)

at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:607)

at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:545)

at 
org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:672)

at 
org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)

at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)

at java.lang.Thread.run(Thread.java:595)



TP-Processor118 - Thread t...@2293

   java.lang.Thread.State: WAITING on 
org.apache.tomcat.util.threads.threadpool$controlrunna...@3579cafe

at java.lang.Object.wait(Native Method)

at java.lang.Object.wait(Object.java:474)

at 
org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)

at java.lang.Thread.run(Thread.java:595)

Re: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

2009-05-18 Thread Rainer Jung

Yes, those two look like waiting for next request on an existing
connection from the web server to Tomcat and sitting idle in the pool,
waiting for a new connection to handle.

On 18.05.2009 22:44, Pantvaidya, Vishwajit wrote:
 [Pantvaidya, Vishwajit] Here are figures from 3 of the servers which
 
 have
 
 not yet run out of threads (so the thread count does not add up to 200).
 
 I
 
 have taken these late at night when no users are present, so I was
 
 expecting all threads to be Waiting for tomcat thread-pool.
 
 
 1. Total TP-Processor threads 48, Waiting 46, Runnable 2
 
 2. Total TP-Processor threads 40, Waiting 29, Runnable 11
 
 3. Total TP-Processor threads 120, Waiting 7, Runnable 113
 
 
 
 [Pantvaidya, Vishwajit] Since Rainer mentioned that he would like to see more 
 of the stack trace, here are the complete stack traces for a Runnable and 
 Waiting thread from #3 above. All Runnable/Waiting threads from all the above 
 cases have same stack trace as below:
 
 
 
 TP-Processor119 - Thread t...@2294
 
java.lang.Thread.State: RUNNABLE
 
 at java.net.SocketInputStream.socketRead0(Native Method)
 
 at java.net.SocketInputStream.read(SocketInputStream.java:129)
 
 at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
 
 at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
 
 at java.io.BufferedInputStream.read(BufferedInputStream.java:313)
 
 at org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:607)
 
 at org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:545)
 
 at 
 org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:672)
 
 at 
 org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:876)
 
 at 
 org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
 
 at java.lang.Thread.run(Thread.java:595)
 
 
 
 TP-Processor118 - Thread t...@2293
 
java.lang.Thread.State: WAITING on 
 org.apache.tomcat.util.threads.threadpool$controlrunna...@3579cafe
 
 at java.lang.Object.wait(Native Method)
 
 at java.lang.Object.wait(Object.java:474)
 
 at 
 org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
 
 at java.lang.Thread.run(Thread.java:595)
 

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLE stage even with no activity

 -Original Message-
 From: Rainer Jung [mailto:rainer.j...@kippdata.de]
 Sent: Monday, May 18, 2009 2:43 PM
 To: Tomcat Users List
 Subject: Re: Running out of tomcat threads - why many threads in RUNNABLE
 stage even with no activity

 Yes, those two look like waiting for next request on an existing
 connection from the web server to Tomcat and sitting idle in the pool,
 waiting for a new connection to handle.

[Pantvaidya, Vishwajit] Thanks Rainier. Any idea why threads would be sitting 
around in Runnable state even when nobody has been using application for a long 
time. From whatever I have read on this, it seems to me that this could happen 
if a servlet writes something to a response stream, closes the response stream, 
but after that keeps on doing some processing (e.g. running an infinite loop). 
I am reasonably sure that our app is not doing something like that. Unless 
there was something like an infinite loop running in a servlet, I would assume 
that the servlet would eventually return and the tomcat TP-Processor thread 
would be released back to the connection pool (go into Waiting state).

 On 18.05.2009 22:44, Pantvaidya, Vishwajit wrote:
  [Pantvaidya, Vishwajit] Here are figures from 3 of the servers which

  have

  not yet run out of threads (so the thread count does not add up to
 200).

  I

  have taken these late at night when no users are present, so I was

  expecting all threads to be Waiting for tomcat thread-pool.

  1. Total TP-Processor threads 48, Waiting 46, Runnable 2

  2. Total TP-Processor threads 40, Waiting 29, Runnable 11

  3. Total TP-Processor threads 120, Waiting 7, Runnable 113

  [Pantvaidya, Vishwajit] Since Rainer mentioned that he would like to see
 more of the stack trace, here are the complete stack traces for a Runnable
 and Waiting thread from #3 above. All Runnable/Waiting threads from all
 the above cases have same stack trace as below:

  TP-Processor119 - Thread t...@2294

 java.lang.Thread.State: RUNNABLE

  at java.net.SocketInputStream.socketRead0(Native Method)

  at java.net.SocketInputStream.read(SocketInputStream.java:129)

  at
 java.io.BufferedInputStream.fill(BufferedInputStream.java:218)

  at
 java.io.BufferedInputStream.read1(BufferedInputStream.java:256)

  at
 java.io.BufferedInputStream.read(BufferedInputStream.java:313)

  at
 org.apache.jk.common.ChannelSocket.read(ChannelSocket.java:607)

  at
 org.apache.jk.common.ChannelSocket.receive(ChannelSocket.java:545)

  at
 org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:67
 2)

  at
 org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.ja
 va:876)

  at
 org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.j
 ava:684)

  at java.lang.Thread.run(Thread.java:595)

  TP-Processor118 - Thread t...@2293

 java.lang.Thread.State: WAITING on
 org.apache.tomcat.util.threads.threadpool$controlrunna...@3579cafe

  at java.lang.Object.wait(Native Method)

  at java.lang.Object.wait(Object.java:474)

  at
 org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.j
 ava:656)

  at java.lang.Thread.run(Thread.java:595)

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity

2009-05-18 Thread Caldarale, Charles R

 From: Pantvaidya, Vishwajit [mailto:vpant...@selectica.com]
 Subject: RE: Running out of tomcat threads - why many threads in
 RUNNABLEstage even with no activity

 From whatever I have read on this, it seems to me that this could 
 happen if a servlet writes something to a response stream, closes
 the response stream, but after that keeps on doing some processing
 (e.g. running an infinite loop).

No - the thread would be inside the servlet in that case.  The thread here in 
the RUNNABLE state is waiting for a *new* request to come in over an active AJP 
connection; a thread in the WAITING state would be assigned to a new connection 
when one is accepted.

 - Chuck

THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY 
MATERIAL and is thus for use only by the intended recipient. If you received 
this in error, please contact the sender and delete the e-mail and its 
attachments from all computers.

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org

RE: Running out of tomcat threads - why many threads in RUNNABLEstage even with no activity