RE: Tomcat request processing gets stuck
I checked the stuck and i got to a few conclusions: after tomcat stops receiving messages it will receive some kinds and some not: 1. if i use http client and write a string directly the tomcat will receive the data 2. if i use http client and write a stream, the tomcat will not receive the data 3. if i use wget and send a message to tomcat it will receive data which means, that after tomcat works with a load of data it gets stuck to streams and not to strings (maybe related to the fact that i use nio connector??) Does any one have any idea on this problem with tomcat any help will be highly appreciated. Thanks, Michal Peter Crowther wrote: From: Christopher Schultz [mailto:[EMAIL PROTECTED] acceptCount=200 just means that the socket will accept 200 clients /in addition/ to those currently being served by RequestProcessor threads. The only way to see those waiting clients would be to query the socket itself (maybe only available at the C-library level?). Rarely even that - these connections are held in limbo in the kernel. There are sometimes ways of examining the next request in the queue (to see whether you want to accept it or reject it), but I'm not aware of a general way of examining *all* requests in an accept queue. I'd love to hear about any that exist, as in the past I've needed to monitor these! - Peter - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Tomcat-request-processing-gets-stuck-tp20526036p20835816.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat request processing gets stuck
Michal Singer wrote: Hi. I am not sure the stuck as any thing to do with the Request Processor accumulation. I work with nio connector. I use acceptCount=200 so maybe this is why i see the 200 Request Processors. This is the full configuratin i use for connector: Connector executor=AgentExecutor useExecutor=true port=8080 protocol=org.apache.coyote.http11.Http11NioProtocol SSLEnabled=false scheme=http secure=false redirectPort=8443 connectionTimeout=2 enableLookups=false disableUploadTimeout=true allowTrace=false maxSpareThreads=15 maxThreads=500 acceptCount=200/ Odd configuration. You appear to have mixed up using executor and non-executor configuration. After running a high load for about an hour, using jmeter as my simulator for load, the tomcat stops hanling the requests. jmeter stops sending messages since tomcat does not process them. I can't see any stuck on the application side. I see that all requests are handled. Maybe the configuration i use is not good for the connector, maybe there is a problem with nio connectors. Again, no. It is an application issue. Get the thread dumps and find out where the app has locked up. Mark - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat request processing gets stuck
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Michal, Michal Singer wrote: I am not sure the stuck as any thing to do with the Request Processor accumulation. I work with nio connector. I use acceptCount=200 so maybe this is why i see the 200 Request Processors. acceptCount=200 just means that the socket will accept 200 clients /in addition/ to those currently being served by RequestProcessor threads. The only way to see those waiting clients would be to query the socket itself (maybe only available at the C-library level?). This is the full configuration i use for connector: Connector executor=AgentExecutor useExecutor=true port=8080 protocol=org.apache.coyote.http11.Http11NioProtocol SSLEnabled=false scheme=http secure=false redirectPort=8443 connectionTimeout=2 enableLookups=false disableUploadTimeout=true allowTrace=false maxSpareThreads=15 maxThreads=500 acceptCount=200/ Your configuration allows for 500 request processors, so seeing only 200 doesn't seem too bad. I agree with Mark's confusion over your configuration: you seem to be trying to use both a standard connector as well as an executer. You should pick one and stick with it. After running a high load for about an hour, using jmeter as my simulator for load, the tomcat stops hanling the requests. jmeter stops sending messages since tomcat does not process them. I can't see any stuck on the application side. I see that all requests are handled. Thread dump, thread dump, thread dump! - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkkkIvoACgkQ9CaO5/Lv0PDjAQCgorj69K3azqmYCASIexi4cMCN qhcAoItqx00LrA4wjAPjLUXgOatJZwie =4GQT -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat request processing gets stuck
Hi, thanks for the response. I have A few questions regarding your comment: 1. request processors are equivalent to threads? 2. doesn't nio work with one thread? (I thougt that this thread configuration is irrelevant since nio works with one thread to receive requests.) 3. I shouldn't work with Executor while working with nio connector? 4. can you send me an example of nio good configuration? Thank you very much, Michal Christopher Schultz-2 wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Michal, Michal Singer wrote: I am not sure the stuck as any thing to do with the Request Processor accumulation. I work with nio connector. I use acceptCount=200 so maybe this is why i see the 200 Request Processors. acceptCount=200 just means that the socket will accept 200 clients /in addition/ to those currently being served by RequestProcessor threads. The only way to see those waiting clients would be to query the socket itself (maybe only available at the C-library level?). This is the full configuration i use for connector: Connector executor=AgentExecutor useExecutor=true port=8080 protocol=org.apache.coyote.http11.Http11NioProtocol SSLEnabled=false scheme=http secure=false redirectPort=8443 connectionTimeout=2 enableLookups=false disableUploadTimeout=true allowTrace=false maxSpareThreads=15 maxThreads=500 acceptCount=200/ Your configuration allows for 500 request processors, so seeing only 200 doesn't seem too bad. I agree with Mark's confusion over your configuration: you seem to be trying to use both a standard connector as well as an executer. You should pick one and stick with it. After running a high load for about an hour, using jmeter as my simulator for load, the tomcat stops hanling the requests. jmeter stops sending messages since tomcat does not process them. I can't see any stuck on the application side. I see that all requests are handled. Thread dump, thread dump, thread dump! - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkkkIvoACgkQ9CaO5/Lv0PDjAQCgorj69K3azqmYCASIexi4cMCN qhcAoItqx00LrA4wjAPjLUXgOatJZwie =4GQT -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Tomcat-request-processing-gets-stuck-tp20526036p20581747.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat request processing gets stuck
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Michal, Michal Singer wrote: 1. request processors are equivalent to threads? Yes. The only thing that can execute code is a thread. Given your configuration, it appears that you will have 500 maximum threads. I am a little unfamiliar with the NIO connector, but this is what the documentation says when using useExecutor=true: If set to true(default) then the max pool size is the maxThreads attribute and the core pool size is the minSpareThreads. This value is ignored if the executor attribute is present and points to a valid shared thread pool. 2. doesn't nio work with one thread? (I thougt that this thread configuration is irrelevant since nio works with one thread to receive requests.) Nope. The NIO connector is just a non-blocking one. A single thread could never supply suitable performance for a web application. You might be thinking of Comet, which is a completely different beast. http://tomcat.apache.org/tomcat-6.0-doc/aio.html 3. I shouldn't work with Executor while working with nio connector? Using the executor is perfectly find. I think Chuck and I were confused since the executor is typically specified separately (as Executor) but the documentation appears to suggest that an executor will be automatically created given your configuration. 4. can you send me an example of nio good configuration? Your configuration appears to be good, actually. The only problems are: 1. You are nervous about the number of request processors appearing (and you probably shouldn't be, unless you are hardly using your app at all and you are still seeing hundreds of request processor threads being created) 2. Your application hangs, which is, of course, a serious problem. For that last one, you'll need to take thread dumps to find out what your app is waiting on. You might want to scale back the number of request processors (maxThreads setting) so your thread dumps are small enough to read without losing your mind. My guess is that you are hitting your database thread pool limit and the application is simply waiting for db connections to become available. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkkkMOQACgkQ9CaO5/Lv0PBYWACgwsqRM2XG4N1SOMSt722XlY/Q MbkAnjjThQY/WEFv65JtmkdENyltkFCF =yIAC -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat request processing gets stuck
Thanks again for the great response. I am not worried about the request processors any more now that i understand what they mean. However, i see that when i use the nio, my application gets stuck. i think working without nio is better. I am not sure why, if as i know nio is supposed to improve performance. I am checking my connectors configuration as well as my application getting stuck. Christopher Schultz-2 wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Michal, Michal Singer wrote: 1. request processors are equivalent to threads? Yes. The only thing that can execute code is a thread. Given your configuration, it appears that you will have 500 maximum threads. I am a little unfamiliar with the NIO connector, but this is what the documentation says when using useExecutor=true: If set to true(default) then the max pool size is the maxThreads attribute and the core pool size is the minSpareThreads. This value is ignored if the executor attribute is present and points to a valid shared thread pool. 2. doesn't nio work with one thread? (I thougt that this thread configuration is irrelevant since nio works with one thread to receive requests.) Nope. The NIO connector is just a non-blocking one. A single thread could never supply suitable performance for a web application. You might be thinking of Comet, which is a completely different beast. http://tomcat.apache.org/tomcat-6.0-doc/aio.html 3. I shouldn't work with Executor while working with nio connector? Using the executor is perfectly find. I think Chuck and I were confused since the executor is typically specified separately (as Executor) but the documentation appears to suggest that an executor will be automatically created given your configuration. 4. can you send me an example of nio good configuration? Your configuration appears to be good, actually. The only problems are: 1. You are nervous about the number of request processors appearing (and you probably shouldn't be, unless you are hardly using your app at all and you are still seeing hundreds of request processor threads being created) 2. Your application hangs, which is, of course, a serious problem. For that last one, you'll need to take thread dumps to find out what your app is waiting on. You might want to scale back the number of request processors (maxThreads setting) so your thread dumps are small enough to read without losing your mind. My guess is that you are hitting your database thread pool limit and the application is simply waiting for db connections to become available. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkkkMOQACgkQ9CaO5/Lv0PBYWACgwsqRM2XG4N1SOMSt722XlY/Q MbkAnjjThQY/WEFv65JtmkdENyltkFCF =yIAC -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Tomcat-request-processing-gets-stuck-tp20526036p20582385.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat request processing gets stuck
From: Christopher Schultz [mailto:[EMAIL PROTECTED] Subject: Re: Tomcat request processing gets stuck 2. doesn't nio work with one thread? (I thougt that this thread configuration is irrelevant since nio works with one thread to receive requests.) Nope. The NIO connector is just a non-blocking one. To elaborate a bit: that means that HTTP connections in a keep-alive state do not consume requests. The NIO poller will dispatch incoming messages to threads on a request basis, rather than allocating a thread per connection as the basic HTTP connector does. 3. I shouldn't work with Executor while working with nio connector? Using the executor is perfectly find. I think Chuck and I were confused since the executor is typically specified separately (as Executor) When Tomcat 6 was originally released, Remy M cautioned against using an executor due to performance problems. A recent posting by Filip H indicates those problems have been resolved and that an executor is the proper thing to use for fine-grained control of thread pools. 4. can you send me an example of nio good configuration? Your configuration appears to be good, actually. Not really - a good one would not include attributes that are ignored. Can't really tell if you config is o.k. with seeing the corresponding Executor element. Since you're using an executor, you should remove the min/max threads attributes. My guess is that you are hitting your database thread pool limit and the application is simply waiting for db connections to become available. Or the app is failing to return connections to the DB pool. As many have suggested in other threads, try testing with a DB pool size of one - it will be slow, but any mishandling of DB connections will become painfully obvious. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat request processing gets stuck
From: Michal Singer [mailto:[EMAIL PROTECTED] Subject: Re: Tomcat request processing gets stuck I am not sure why, if as i know nio is supposed to improve performance. Not true; NIO improves *capacity*, but it will slightly degrade throughput due to more thread switching. I am checking my connectors configuration as well as my application getting stuck. Take thread dumps! Note that JConsole and JVisualVM (among other tools) will let you watch your threads on the fly so you can see where they're getting stuck. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat request processing gets stuck
From: Christopher Schultz [mailto:[EMAIL PROTECTED] acceptCount=200 just means that the socket will accept 200 clients /in addition/ to those currently being served by RequestProcessor threads. The only way to see those waiting clients would be to query the socket itself (maybe only available at the C-library level?). Rarely even that - these connections are held in limbo in the kernel. There are sometimes ways of examining the next request in the queue (to see whether you want to accept it or reject it), but I'm not aware of a general way of examining *all* requests in an accept queue. I'd love to hear about any that exist, as in the past I've needed to monitor these! - Peter - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat request processing gets stuck
Deear Michal, When i checked the tomcat using jconsole i see that the object RequestProcessor accumulates to many objects which i guess may show the cause of this problem. Uhm. Could you explain what you mean here? I don't understand what you are trying to say. You see too many request processors? How many? What is the maximum and the expected number? -- Kees Jan http://java-monitor.com/forum/ [EMAIL PROTECTED] 06-51838192 The secret of success lies in the stability of the goal. -- Benjamin Disraeli - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat request processing gets stuck
Hi. I am not sure the stuck as any thing to do with the Request Processor accumulation. I work with nio connector. I use acceptCount=200 so maybe this is why i see the 200 Request Processors. This is the full configuratin i use for connector: Connector executor=AgentExecutor useExecutor=true port=8080 protocol=org.apache.coyote.http11.Http11NioProtocol SSLEnabled=false scheme=http secure=false redirectPort=8443 connectionTimeout=2 enableLookups=false disableUploadTimeout=true allowTrace=false maxSpareThreads=15 maxThreads=500 acceptCount=200/ After running a high load for about an hour, using jmeter as my simulator for load, the tomcat stops hanling the requests. jmeter stops sending messages since tomcat does not process them. I can't see any stuck on the application side. I see that all requests are handled. Maybe the configuration i use is not good for the connector, maybe there is a problem with nio connectors. Kees Jan Koster wrote: Deear Michal, When i checked the tomcat using jconsole i see that the object RequestProcessor accumulates to many objects which i guess may show the cause of this problem. Uhm. Could you explain what you mean here? I don't understand what you are trying to say. You see too many request processors? How many? What is the maximum and the expected number? -- Kees Jan http://java-monitor.com/forum/ [EMAIL PROTECTED] 06-51838192 The secret of success lies in the stability of the goal. -- Benjamin Disraeli - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Tomcat-request-processing-gets-stuck-tp20526036p20574015.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat request processing gets stuck
Michal Singer wrote: Any ideas why the stuck, what can i do to check this? can there be a problem with my tomcat configuration? Almost certainly an application issue. Take a series of thread dumps (2 or 3) 15 seconds apart and look for threads that are consistently stuck between dumps. Mark - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat request processing gets stuck
Hi, 1. Please explain what do you mean by thread dumps? 2. why do you think this is an application problem? Thanks, Michal markt-2 wrote: Michal Singer wrote: Any ideas why the stuck, what can i do to check this? can there be a problem with my tomcat configuration? Almost certainly an application issue. Take a series of thread dumps (2 or 3) 15 seconds apart and look for threads that are consistently stuck between dumps. Mark - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Tomcat-request-processing-gets-stuck-tp20526036p20526268.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat request processing gets stuck
From: Michal Singer [mailto:[EMAIL PROTECTED] Subject: Re: Tomcat request processing gets stuck 1. Please explain what do you mean by thread dumps? You could use Google to find definitions and examples... The JConsole Threads tab shows the stack trace of individual threads; a thread dump is the stack trace of all threads. Use the jstack tool to get one. 2. why do you think this is an application problem? Experience; in situations like this, it *always* is. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat request processing gets stuck
Michal Singer wrote: Hi, 1. Please explain what do you mean by thread dumps? Google is your friend, as is http://tomcat.markmail.org 2. why do you think this is an application problem? Years of experience (and if Tomcat had an issue the users list would be full of posts reporting problems and it isn't). Mark Thanks, Michal markt-2 wrote: Michal Singer wrote: Any ideas why the stuck, what can i do to check this? can there be a problem with my tomcat configuration? Almost certainly an application issue. Take a series of thread dumps (2 or 3) 15 seconds apart and look for threads that are consistently stuck between dumps. Mark - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Tomcat request processing gets stuck
I didn't think it is a bug in tomcat. i thought of some problem in configuration I use for tomcat. maybe the default configuration is not good for load?? markt-2 wrote: Michal Singer wrote: Hi, 1. Please explain what do you mean by thread dumps? Google is your friend, as is http://tomcat.markmail.org 2. why do you think this is an application problem? Years of experience (and if Tomcat had an issue the users list would be full of posts reporting problems and it isn't). Mark Thanks, Michal markt-2 wrote: Michal Singer wrote: Any ideas why the stuck, what can i do to check this? can there be a problem with my tomcat configuration? Almost certainly an application issue. Take a series of thread dumps (2 or 3) 15 seconds apart and look for threads that are consistently stuck between dumps. Mark - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- View this message in context: http://www.nabble.com/Tomcat-request-processing-gets-stuck-tp20526036p20526467.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Tomcat request processing gets stuck
From: Michal Singer [mailto:[EMAIL PROTECTED] Subject: Re: Tomcat request processing gets stuck maybe the default configuration is not good for load?? No, it's your webapp. Get the thread dumps and tell us what you see in there. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
