POST request hangs hup
Hi, I'm running tomcat 6.0.18 with java 1.6.0_13 on a linux box. The only webapp deployed is Sun OpenSSO (http://www.opensso.org) built from source (20090930). Sometimes a tomcat thread hangs hup while serving a POST request (/UI/Login): monitoring tomcat status I see the processing time grows and the thread never ends. When this happens I am not able to shutdown tomcat with shutdown.sh and I need to kill -9 the java process. I think it is not a problem of memory exausted because if I run top the memory usage by java process is ok (a few hundreds MB despite CATALINA_OPTS=-Xmx6048M) but the cpu usage is 100%. Is there a way to kill a single thread in tomcat whitout restarting it? Is there a way to limit the max execution time for a single thread/request? thanks, ivan -- ciao, ivan
Re: POST request hangs hup
On Thu, 2009-07-02 at 23:40 +0200, Ivan Longhi wrote: Hi, Hi, Ivan. I'm running tomcat 6.0.18 with java 1.6.0_13 on a linux box. The only webapp deployed is Sun OpenSSO (http://www.opensso.org) built from source (20090930). Sometimes a tomcat thread hangs hup while serving a POST request (/UI/Login): monitoring tomcat status I see the processing time grows and the thread never ends. When this happens I am not able to shutdown tomcat with shutdown.sh and I need to kill -9 the java process. I think it is not a problem of memory exausted because if I run top the memory usage by java process is ok (a few hundreds MB despite CATALINA_OPTS=-Xmx6048M) but the cpu usage is 100%. I think could be anything: An infinite loop in your app, a database (or other resource) deadlock, any intensive db query (with a really long transaction timeout). For instance, last month we've adopted a new Transaction Manager that keeps some threads (non daemons) running and prevents Tomcat of doing a clean stop. Now i'm forcing Tomcat to stop after 5 minutes waiting. My (little) experience tells me that in cases like this, your app is more guilt than your Application Server/Web Container. But it is just my 2 cents. []'s -- If there must be trouble, let it be in my day, that my child may have peace. Thomas Paine - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: POST request hangs hup
is probably more guilt :-) On Thu, 2009-07-02 at 19:20 -0300, Daniel Henrique Alves Lima wrote: On Thu, 2009-07-02 at 23:40 +0200, Ivan Longhi wrote: Hi, Hi, Ivan. I'm running tomcat 6.0.18 with java 1.6.0_13 on a linux box. The only webapp deployed is Sun OpenSSO (http://www.opensso.org) built from source (20090930). Sometimes a tomcat thread hangs hup while serving a POST request (/UI/Login): monitoring tomcat status I see the processing time grows and the thread never ends. When this happens I am not able to shutdown tomcat with shutdown.sh and I need to kill -9 the java process. I think it is not a problem of memory exausted because if I run top the memory usage by java process is ok (a few hundreds MB despite CATALINA_OPTS=-Xmx6048M) but the cpu usage is 100%. I think could be anything: An infinite loop in your app, a database (or other resource) deadlock, any intensive db query (with a really long transaction timeout). For instance, last month we've adopted a new Transaction Manager that keeps some threads (non daemons) running and prevents Tomcat of doing a clean stop. Now i'm forcing Tomcat to stop after 5 minutes waiting. My (little) experience tells me that in cases like this, your app is more guilt than your Application Server/Web Container. But it is just my 2 cents. []'s -- If there must be trouble, let it be in my day, that my child may have peace. Thomas Paine - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: POST request hangs hup
From: Ivan Longhi [mailto:ivan.lon...@gmail.com] Subject: POST request hangs hup Is there a way to kill a single thread in tomcat whitout restarting it? No. Is there a way to limit the max execution time for a single thread/request? No. You need to fix your webapp, since it has a serious bug resulting in an infinite loop. Take some thread dumps a few seconds apart to find out where the thread is spending its time. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: POST request hangs hup
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Daniel, On 7/2/2009 6:20 PM, Daniel Henrique Alves Lima wrote: I think could be anything: An infinite loop in your app, a database (or other resource) deadlock, any intensive db query (with a really long transaction timeout). +1 Take a thread dump of your application to find out where the thread is stuck (see http://wiki.apache.org/tomcat/HowTo#head-a3243c1d640297a9f2aa9446cbed434efbd4d1d9 if you aren't sure how to take a thread dump). Several thread dumps a few seconds apart is usually enough to give you a good picture of what's happening. Post them to the list if you are having a hard time determining what's going on. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkpNbt0ACgkQ9CaO5/Lv0PArpQCgqIKN+mkCc+W8QCtru1S0Kjag HrEAoJmO1k3SlqnuXBOpmRZwt8xFEJz5 =W5UC -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: request hangs
if you are on TC 6, you can always use the NIO connector as an alternative.
There is a mem leak in 6.0.10, but fixed in SVN, new release around the
corner
take a look at
http://tomcat.apache.org/tomcat-6.0-doc/config/http.html
the protocol attribute tells you how to configure the different connectors
Filip
Chris Eldredge wrote:
Well, as I've stated I'm not aware of any resource contention. The
UDP socket is being used in a daemon thread that is executing
asynchronously with the standard Tomcat request processing threads and
I don't see any blocks waiting for monitors etc.
Even worse, I tried removing tc-native from $TOMCAT_HOME/bin so APR is
not being used anymore, and poof, the problem went away. That doesn't
make me comfortable, but I don't have time to dig into the bowels of APR.
Martin Gainty wrote:
Hi Chris-
Possible if the invoker 1)is executing the thread in a synchronized
fashion ..but.. synchronization produces contention (the analogy is 2
boys reach for the same piece of bread at the dinner table at the
same time where neither one wants to give the other his prize..it's
best to avoid synchronization contention scenarios)
2)'Classic Thread' objects although in most scenarios these thread
objects when associated with a key are not necessarily short-lived
and may never be GCed so eventually you may see 'permgen space
errors' happening as the objects which are classic Thread stay in
heap forever..
3)The best solution is to implement your class using ThreadLocal to
quote
A thread-local variable effectively provides a separate copy of its
value for each thread that uses it. Each thread can see only the
value associated with that thread
The classic example is acquiring DBConnection objects where you want
a specific DBConnection alloced and init'ed on a per thread basis an
example
public class ConnectionFactory {
private static class ThreadLocalConnection extends ThreadLocal
public Object initialValue() {
return
DriverManager.getConnection(ConfigurationSingleton.getDbUrl());
}
} //ThreadLocalConnection
private static ThreadLocalConnection conn = new
ThreadLocalConnection(); //this will acquire a per-thread singleton
object only for your thread }//ConnectionFactory
This example comes from IBM site located at
http://www-128.ibm.com/developerworks/java/library/j-threads3.html
Does this make sense?
HTH,
Martin--
---
This e-mail message (including attachments, if any) is intended for
the use of the individual or entity to which it is addressed and may
contain information that is privileged, proprietary , confidential
and exempt from disclosure. If you are not the intended recipient,
you are notified that any dissemination, distribution or copying of
this communication is strictly prohibited.
---
Le présent message électronique (y compris les pièces qui y sont
annexées, le cas échéant) s'adresse au destinataire indiqué et peut
contenir des renseignements de caractère privé ou confidentiel. Si
vous n'êtes pas le destinataire de ce document, nous vous signalons
qu'il est strictement interdit de le diffuser, de le distribuer ou de
le reproduire.
- Original Message - From: Chris Eldredge
[EMAIL PROTECTED]
To: users@tomcat.apache.org
Sent: Thursday, March 22, 2007 10:10 AM
Subject: Re: request hangs
Martin,
Thanks for the response. The thread accepting UDP packets has a
timeout of 100ms after which it waits again for a packet. Anyway,
this is happening in its own thread, executing asynchronously from
Tomcat's http request processing threads. I'm not aware of any
limitations where accepting UDP packets should prevent another
thread from accepting TCP connections... are you?
Thanks again,
Chris
Martin Gainty wrote:
Hi Chris-
what happens when you log these events?
start of UDP loop
Accepting UDP packets on the loopback address.
log the buffer from UDP accept goto start of UDP loop
start of loop to write to temp file
Reading standard out from a child process log the buffer which is
read from standard out
writing it to a temp file.
go start of loop to write to temp file
Im guessing the UDP packet accept logic *may possibly* be blocking
as it waits for the socket to read
(and hanging the thread)
Martin --
---
This e-mail message (including attachments, if any) is intended for
the use of the individual or entity to which it is addressed and
may contain information that is privileged, proprietary ,
confidential and exempt from disclosure. If you are not the
intended recipient, you are notified that any dissemination,
distribution or copying of this communication is strictly prohibited.
---
Le présent
Re: request hangs
Well, as I've stated I'm not aware of any resource contention. The UDP
socket is being used in a daemon thread that is executing asynchronously
with the standard Tomcat request processing threads and I don't see any
blocks waiting for monitors etc.
Even worse, I tried removing tc-native from $TOMCAT_HOME/bin so APR is
not being used anymore, and poof, the problem went away. That doesn't
make me comfortable, but I don't have time to dig into the bowels of APR.
Martin Gainty wrote:
Hi Chris-
Possible if the invoker
1)is executing the thread in a synchronized fashion ..but.. synchronization produces contention (the analogy is 2 boys reach for the same piece of bread at the dinner table at the same time where neither one wants to give the other his prize..it's best to avoid synchronization contention scenarios)
2)'Classic Thread' objects although in most scenarios these thread objects when
associated with a key are not necessarily short-lived and may never be GCed so
eventually you may see 'permgen space errors' happening as the objects which
are classic Thread stay in heap forever..
3)The best solution is to implement your class using ThreadLocal to quote
A thread-local variable effectively provides a separate copy of its value for each
thread that uses it. Each thread can see only the value associated with that thread
The classic example is acquiring DBConnection objects where you want a specific
DBConnection alloced and init'ed on a per thread basis an example
public class ConnectionFactory
{
private static class ThreadLocalConnection extends ThreadLocal
public Object initialValue() {
return
DriverManager.getConnection(ConfigurationSingleton.getDbUrl());
}
} //ThreadLocalConnection
private static ThreadLocalConnection conn = new ThreadLocalConnection(); //this will acquire a per-thread singleton object only for your thread
}//ConnectionFactory
This example comes from IBM site located at
http://www-128.ibm.com/developerworks/java/library/j-threads3.html
Does this make sense?
HTH,
Martin--
---
This e-mail message (including attachments, if any) is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, proprietary , confidential and exempt from disclosure. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this communication is strictly prohibited.
---
Le présent message électronique (y compris les pièces qui y sont annexées, le cas échéant) s'adresse au destinataire indiqué et peut contenir des renseignements de caractère privé ou confidentiel. Si vous n'êtes pas le destinataire de ce document, nous vous signalons qu'il est strictement interdit de le diffuser, de le distribuer ou de le reproduire.
- Original Message -
From: Chris Eldredge [EMAIL PROTECTED]
To: users@tomcat.apache.org
Sent: Thursday, March 22, 2007 10:10 AM
Subject: Re: request hangs
Martin,
Thanks for the response. The thread accepting UDP packets has a timeout
of 100ms after which it waits again for a packet. Anyway, this is
happening in its own thread, executing asynchronously from Tomcat's http
request processing threads. I'm not aware of any limitations where
accepting UDP packets should prevent another thread from accepting TCP
connections... are you?
Thanks again,
Chris
Martin Gainty wrote:
Hi Chris-
what happens when you log these events?
start of UDP loop
Accepting UDP packets on the loopback address.
log the buffer from UDP accept
goto start of UDP loop
start of loop to write to temp file
Reading standard out from a child process
log the buffer which is read from standard out
writing it to a temp file.
go start of loop to write to temp file
Im guessing the UDP packet accept logic *may possibly* be blocking as it waits
for the socket to read
(and hanging the thread)
Martin --
---
This e-mail message (including attachments, if any) is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, proprietary , confidential and exempt from disclosure. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this communication is strictly prohibited.
---
Le présent message électronique (y compris les pièces qui y sont annexées, le cas échéant) s'adresse au destinataire indiqué et peut contenir des renseignements de caractère privé ou confidentiel. Si vous n'êtes pas le destinataire de ce document, nous vous signalons qu'il est strictement interdit de le diffuser, de le distribuer ou de le reproduire.
- Original
Re: request hangs
Martin, Thanks for the response. The thread accepting UDP packets has a timeout of 100ms after which it waits again for a packet. Anyway, this is happening in its own thread, executing asynchronously from Tomcat's http request processing threads. I'm not aware of any limitations where accepting UDP packets should prevent another thread from accepting TCP connections... are you? Thanks again, Chris Martin Gainty wrote: Hi Chris- what happens when you log these events? start of UDP loop Accepting UDP packets on the loopback address. log the buffer from UDP accept goto start of UDP loop start of loop to write to temp file Reading standard out from a child process log the buffer which is read from standard out writing it to a temp file. go start of loop to write to temp file Im guessing the UDP packet accept logic *may possibly* be blocking as it waits for the socket to read (and hanging the thread) Martin -- --- This e-mail message (including attachments, if any) is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, proprietary , confidential and exempt from disclosure. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this communication is strictly prohibited. --- Le présent message électronique (y compris les pièces qui y sont annexées, le cas échéant) s'adresse au destinataire indiqué et peut contenir des renseignements de caractère privé ou confidentiel. Si vous n'êtes pas le destinataire de ce document, nous vous signalons qu'il est strictement interdit de le diffuser, de le distribuer ou de le reproduire. - Original Message - From: Chris Eldredge [EMAIL PROTECTED] To: users@tomcat.apache.org Sent: Wednesday, March 21, 2007 6:30 PM Subject: request hangs I'm working on a web application which sometimes has several daemon threads doing I/O processing in the background. The application seems to be fine except when several tasks are running, sometimes Tomcat gets a request and doesn't seem to process it. The request seems to time out without ever being passed into my application for processing. My index page has an auto-refresh meta tag so I see this problem frequently. The really strange thing is if I click reload once, the next request also hangs, but if I click reload a 2nd time, this request is processed very quickly. This behavior is very consistent, and doesn't seem to have anything to do with the state of the background tasks (they are still running). I mention the background tasks because I only see this hanging behavior when the background tasks are active. When my application is idle, I never see the behavior. Beyond that, I can't figure out what the background tasks might be doing which would prevent Tomcat from processing incoming requests: * Accepting UDP packets on the loopback address. * Reading standard out from a child process and writing it to a temp file. Neither of these activities seem like they should interfere with Tomcat request processing. I have placed some logging calls on a filter I configured in my application and for the hung requests, the filter never logs a request. This seems to indicate that the request is getting stuck before my application gets a chance to process it. Has anybody seen anything like this before? Any advice for troubleshooting? - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: request hangs
Hi Chris-
Possible if the invoker
1)is executing the thread in a synchronized fashion ..but.. synchronization
produces contention (the analogy is 2 boys reach for the same piece of bread at
the dinner table at the same time where neither one wants to give the other his
prize..it's best to avoid synchronization contention scenarios)
2)'Classic Thread' objects although in most scenarios these thread objects when
associated with a key are not necessarily short-lived and may never be GCed so
eventually you may see 'permgen space errors' happening as the objects which
are classic Thread stay in heap forever..
3)The best solution is to implement your class using ThreadLocal to quote
A thread-local variable effectively provides a separate copy of its value for
each thread that uses it. Each thread can see only the value associated with
that thread
The classic example is acquiring DBConnection objects where you want a specific
DBConnection alloced and init'ed on a per thread basis an example
public class ConnectionFactory
{
private static class ThreadLocalConnection extends ThreadLocal
public Object initialValue() {
return
DriverManager.getConnection(ConfigurationSingleton.getDbUrl());
}
} //ThreadLocalConnection
private static ThreadLocalConnection conn = new ThreadLocalConnection();
//this will acquire a per-thread singleton object only for your thread
}//ConnectionFactory
This example comes from IBM site located at
http://www-128.ibm.com/developerworks/java/library/j-threads3.html
Does this make sense?
HTH,
Martin--
---
This e-mail message (including attachments, if any) is intended for the use of
the individual or entity to which it is addressed and may contain information
that is privileged, proprietary , confidential and exempt from disclosure. If
you are not the intended recipient, you are notified that any dissemination,
distribution or copying of this communication is strictly prohibited.
---
Le présent message électronique (y compris les pièces qui y sont annexées, le
cas échéant) s'adresse au destinataire indiqué et peut contenir des
renseignements de caractère privé ou confidentiel. Si vous n'êtes pas le
destinataire de ce document, nous vous signalons qu'il est strictement interdit
de le diffuser, de le distribuer ou de le reproduire.
- Original Message -
From: Chris Eldredge [EMAIL PROTECTED]
To: users@tomcat.apache.org
Sent: Thursday, March 22, 2007 10:10 AM
Subject: Re: request hangs
Martin,
Thanks for the response. The thread accepting UDP packets has a timeout
of 100ms after which it waits again for a packet. Anyway, this is
happening in its own thread, executing asynchronously from Tomcat's http
request processing threads. I'm not aware of any limitations where
accepting UDP packets should prevent another thread from accepting TCP
connections... are you?
Thanks again,
Chris
Martin Gainty wrote:
Hi Chris-
what happens when you log these events?
start of UDP loop
Accepting UDP packets on the loopback address.
log the buffer from UDP accept
goto start of UDP loop
start of loop to write to temp file
Reading standard out from a child process
log the buffer which is read from standard out
writing it to a temp file.
go start of loop to write to temp file
Im guessing the UDP packet accept logic *may possibly* be blocking as it
waits for the socket to read
(and hanging the thread)
Martin --
---
This e-mail message (including attachments, if any) is intended for the use
of the individual or entity to which it is addressed and may contain
information that is privileged, proprietary , confidential and exempt from
disclosure. If you are not the intended recipient, you are notified that any
dissemination, distribution or copying of this communication is strictly
prohibited.
---
Le présent message électronique (y compris les pièces qui y sont annexées,
le cas échéant) s'adresse au destinataire indiqué et peut contenir des
renseignements de caractère privé ou confidentiel. Si vous n'êtes pas le
destinataire de ce document, nous vous signalons qu'il est strictement
interdit de le diffuser, de le distribuer ou de le reproduire.
- Original Message -
From: Chris Eldredge [EMAIL PROTECTED]
To: users@tomcat.apache.org
Sent: Wednesday, March 21, 2007 6:30 PM
Subject: request hangs
I'm working on a web application which sometimes has several daemon
threads doing I/O processing in the background. The application seems
to be fine except when several tasks are running, sometimes Tomcat gets
a request and doesn't seem to process it. The request seems to time
request hangs
I'm working on a web application which sometimes has several daemon threads doing I/O processing in the background. The application seems to be fine except when several tasks are running, sometimes Tomcat gets a request and doesn't seem to process it. The request seems to time out without ever being passed into my application for processing. My index page has an auto-refresh meta tag so I see this problem frequently. The really strange thing is if I click reload once, the next request also hangs, but if I click reload a 2nd time, this request is processed very quickly. This behavior is very consistent, and doesn't seem to have anything to do with the state of the background tasks (they are still running). I mention the background tasks because I only see this hanging behavior when the background tasks are active. When my application is idle, I never see the behavior. Beyond that, I can't figure out what the background tasks might be doing which would prevent Tomcat from processing incoming requests: * Accepting UDP packets on the loopback address. * Reading standard out from a child process and writing it to a temp file. Neither of these activities seem like they should interfere with Tomcat request processing. I have placed some logging calls on a filter I configured in my application and for the hung requests, the filter never logs a request. This seems to indicate that the request is getting stuck before my application gets a chance to process it. Has anybody seen anything like this before? Any advice for troubleshooting? - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: request hangs
Hi Chris- what happens when you log these events? start of UDP loop Accepting UDP packets on the loopback address. log the buffer from UDP accept goto start of UDP loop start of loop to write to temp file Reading standard out from a child process log the buffer which is read from standard out writing it to a temp file. go start of loop to write to temp file Im guessing the UDP packet accept logic *may possibly* be blocking as it waits for the socket to read (and hanging the thread) Martin -- --- This e-mail message (including attachments, if any) is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, proprietary , confidential and exempt from disclosure. If you are not the intended recipient, you are notified that any dissemination, distribution or copying of this communication is strictly prohibited. --- Le présent message électronique (y compris les pièces qui y sont annexées, le cas échéant) s'adresse au destinataire indiqué et peut contenir des renseignements de caractère privé ou confidentiel. Si vous n'êtes pas le destinataire de ce document, nous vous signalons qu'il est strictement interdit de le diffuser, de le distribuer ou de le reproduire. - Original Message - From: Chris Eldredge [EMAIL PROTECTED] To: users@tomcat.apache.org Sent: Wednesday, March 21, 2007 6:30 PM Subject: request hangs I'm working on a web application which sometimes has several daemon threads doing I/O processing in the background. The application seems to be fine except when several tasks are running, sometimes Tomcat gets a request and doesn't seem to process it. The request seems to time out without ever being passed into my application for processing. My index page has an auto-refresh meta tag so I see this problem frequently. The really strange thing is if I click reload once, the next request also hangs, but if I click reload a 2nd time, this request is processed very quickly. This behavior is very consistent, and doesn't seem to have anything to do with the state of the background tasks (they are still running). I mention the background tasks because I only see this hanging behavior when the background tasks are active. When my application is idle, I never see the behavior. Beyond that, I can't figure out what the background tasks might be doing which would prevent Tomcat from processing incoming requests: * Accepting UDP packets on the loopback address. * Reading standard out from a child process and writing it to a temp file. Neither of these activities seem like they should interfere with Tomcat request processing. I have placed some logging calls on a filter I configured in my application and for the hung requests, the filter never logs a request. This seems to indicate that the request is getting stuck before my application gets a chance to process it. Has anybody seen anything like this before? Any advice for troubleshooting? - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
