RE: When does Tomcat add and remove threads?
Yes the standard JDK ThreadPoolExecutor behavior is bad. Here is a good thread describing how to fix ThreadPoolExecutor to behave how it should be. https://stackoverflow.com/questions/19528304/how-to-get-the-threadpoolexecutor-to-increase-threads-to-max-before-queueing Obviously Tomcat had to do similar thing. I wish that JDK would add some kind of easy option to ThreadPoolExecutor to add more threads instead of adding to queue, when all existing threads are busy, because the current behavior does not make sense. -Harri -Original Message- From: john.e.gr...@wellsfargo.com.INVALID Sent: tiistai 12. maaliskuuta 2024 18.54 To: users@tomcat.apache.org Subject: RE: When does Tomcat add and remove threads? [Et saa yleensä sähköpostia john.e.gr...@wellsfargo.com.invalid. Lisätietoja siitä, miksi tämä on tärkeää, on osoitteessa https://aka.ms/LearnAboutSenderIdentification ] All, > -Original Message- > From: Christopher Schultz > Sent: Tuesday, March 12, 2024 8:31 AM > To: users@tomcat.apache.org > Subject: Re: When does Tomcat add and remove threads? > > John, > > On 3/11/24 18:14, john.e.gr...@wellsfargo.com.INVALID wrote: > > From: Christopher Schultz > > Sent: Monday, March 11, 2024 5:09 PM > > > >> On 3/11/24 17:47, john.e.gr...@wellsfargo.com.INVALID wrote: > >>> I am using Tomcat 9.x. > >>> > >>> When does Tomcat add and remove threads from its internal thread > >>> pool? I'm talking about the threads with names like > >>> http-nio-8080-exec-1. It appears the thread pool is Tomcat's own > >>> ThreadPoolExecutor but I don't see the exact behavior documented. > >>> I'm familiar with how java.util.concurrent does it, but it looks > >>> like Tomcat's version is a little different. > >> > >> Are you looking for a technical explanation with code references, > >> or a plain-English description of when threads are created and > >> added? > > > Mostly plain English like the j.u.c. ThreadPoolExecutor Java doc has. > > What happens when all core threads are in use? When do tasks go on > > the queue? When does core thread + 1 get added? When do threads > > get removed? > Tomcat will create thread pools under two separate circumstances. They > are related, but behave somewhat differently. > > First, if you declare an in your server.xml, then a thread > pool will be created. You can control the number of threads and their > retention policy such as "keep X spare threads around" and "retire > threads after N seconds without being used." > > Second, if you declare a without specifying an "executor", > a thread pool will be configured for you but you don't really have > control over it because all those nice configuration options for an > are not available on the . If you want to > control those settings, use a linked with an . > To be clear, if you declare a without an "executor" > attribute, your thread pool will be of a fixed size and threads will > never be released. (I think the thread pool starts small and grows but > will never shrink.) > > An is implemented in the StandardThreadExecutor and > ThreadPoolExecutor classes, which I believe were adaptations of > classes from java.util.concurrent introduced into Tomcat before > java.util.concurrent was actually available -- which is why it wasn't > used directly in Tomcat. (NB: The ThreadPoolExecutor class in Tomcat > contains an "@author Doug Lea" tag. The Tomcat source is licensed > under AL2, the JDK source is licensed under GPL2, but the original was > released by Doug Lea into the public domain under a CC0 > 1.0 Deed license.) > > The re-sizing occurs in the ThreadPoolExecutor class if you'd like to > read it. It is not entirely straightforward. You could start by > reading the code for the runWorker(Worker w) method where, at the end, > processWorkerExit is called. > > But since Tomcat's ThreadPoolExecutor is basically Java's > ThreadPoolExecutor, they work the same. > > -chris > > ----- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org I took matters in to my own hands and wrote a test to see what was happening. Unlike java.util.concurrent.ThreadPoolExecutor, Tomcat's TPE adds threads when all core threads are busy. It only adds tasks to the queue when max threads are busy. I prefer this behavior to the JDK behavior. Threads are removed when idle for 60 seconds. This appears to be hard-coded in AbstreactEndpoint.createExecutor(). Thanks B CB [ X ܚX KK[XZ[ \ \ ][ X ܚX P X ] \X K ܙ B ܈Y][ۘ[ [X[ K[XZ[ \ \ Z[ X ] \X K ܙ B
Re: When does Tomcat add and remove threads?
On Tue, Mar 12, 2024 at 5:55 PM wrote: > > All, > > > > -Original Message- > > From: Christopher Schultz > > Sent: Tuesday, March 12, 2024 8:31 AM > > To: users@tomcat.apache.org > > Subject: Re: When does Tomcat add and remove threads? > > > > John, > > > > On 3/11/24 18:14, john.e.gr...@wellsfargo.com.INVALID wrote: > > > From: Christopher Schultz > > > Sent: Monday, March 11, 2024 5:09 PM > > > > > >> On 3/11/24 17:47, john.e.gr...@wellsfargo.com.INVALID wrote: > > >>> I am using Tomcat 9.x. > > >>> > > >>> When does Tomcat add and remove threads from its internal thread > > >>> pool? I'm talking about the threads with names like > > >>> http-nio-8080-exec-1. It appears the thread pool is Tomcat's own > > >>> ThreadPoolExecutor but I don't see the exact behavior documented. > > >>> I'm familiar with how java.util.concurrent does it, but it looks > > >>> like Tomcat's version is a little different. > > >> > > >> Are you looking for a technical explanation with code references, or > > >> a plain-English description of when threads are created and added? > > > > Mostly plain English like the j.u.c. ThreadPoolExecutor Java doc has. > > > What happens when all core threads are in use? When do tasks go on > > > the queue? When does core thread + 1 get added? When do threads get > > > removed? > > Tomcat will create thread pools under two separate circumstances. They are > > related, but behave somewhat differently. > > > > First, if you declare an in your server.xml, then a thread pool > > will be > > created. You can control the number of threads and their retention policy > > such > > as "keep X spare threads around" and "retire threads after N seconds without > > being used." > > > > Second, if you declare a without specifying an "executor", a > > thread pool will be configured for you but you don't really have control > > over it > > because all those nice configuration options for an are not > > available > > on the . If you want to control those settings, use a > > linked with an . To be clear, if you declare a > > without an > > "executor" attribute, your thread pool will be of a fixed size and threads > > will > > never be released. (I think the thread pool starts small and grows but will > > never shrink.) > > > > An is implemented in the StandardThreadExecutor and > > ThreadPoolExecutor classes, which I believe were adaptations of classes from > > java.util.concurrent introduced into Tomcat before java.util.concurrent was > > actually available -- which is why it wasn't used directly in Tomcat. (NB: > > The > > ThreadPoolExecutor class in Tomcat contains an "@author Doug Lea" tag. The > > Tomcat source is licensed under AL2, the JDK source is licensed under GPL2, > > but the original was released by Doug Lea into the public domain under a CC0 > > 1.0 Deed license.) > > > > The re-sizing occurs in the ThreadPoolExecutor class if you'd like to read > > it. It is > > not entirely straightforward. You could start by reading the code for the > > runWorker(Worker w) method where, at the end, processWorkerExit is called. > > > > But since Tomcat's ThreadPoolExecutor is basically Java's > > ThreadPoolExecutor, > > they work the same. > > > > -chris > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > I took matters in to my own hands and wrote a test to see what was happening. > > Unlike java.util.concurrent.ThreadPoolExecutor, Tomcat's TPE adds threads > when all core threads are busy. It only adds tasks to the queue when max > threads are busy. I prefer this behavior to the JDK behavior. > > Threads are removed when idle for 60 seconds. This appears to be hard-coded > in AbstreactEndpoint.createExecutor(). I have added configuration for that, since this was not right. Rémy - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: When does Tomcat add and remove threads?
All, > -Original Message- > From: Christopher Schultz > Sent: Tuesday, March 12, 2024 8:31 AM > To: users@tomcat.apache.org > Subject: Re: When does Tomcat add and remove threads? > > John, > > On 3/11/24 18:14, john.e.gr...@wellsfargo.com.INVALID wrote: > > From: Christopher Schultz > > Sent: Monday, March 11, 2024 5:09 PM > > > >> On 3/11/24 17:47, john.e.gr...@wellsfargo.com.INVALID wrote: > >>> I am using Tomcat 9.x. > >>> > >>> When does Tomcat add and remove threads from its internal thread > >>> pool? I'm talking about the threads with names like > >>> http-nio-8080-exec-1. It appears the thread pool is Tomcat's own > >>> ThreadPoolExecutor but I don't see the exact behavior documented. > >>> I'm familiar with how java.util.concurrent does it, but it looks > >>> like Tomcat's version is a little different. > >> > >> Are you looking for a technical explanation with code references, or > >> a plain-English description of when threads are created and added? > > > Mostly plain English like the j.u.c. ThreadPoolExecutor Java doc has. > > What happens when all core threads are in use? When do tasks go on > > the queue? When does core thread + 1 get added? When do threads get > > removed? > Tomcat will create thread pools under two separate circumstances. They are > related, but behave somewhat differently. > > First, if you declare an in your server.xml, then a thread pool > will be > created. You can control the number of threads and their retention policy such > as "keep X spare threads around" and "retire threads after N seconds without > being used." > > Second, if you declare a without specifying an "executor", a > thread pool will be configured for you but you don't really have control over > it > because all those nice configuration options for an are not > available > on the . If you want to control those settings, use a > linked with an . To be clear, if you declare a without > an > "executor" attribute, your thread pool will be of a fixed size and threads > will > never be released. (I think the thread pool starts small and grows but will > never shrink.) > > An is implemented in the StandardThreadExecutor and > ThreadPoolExecutor classes, which I believe were adaptations of classes from > java.util.concurrent introduced into Tomcat before java.util.concurrent was > actually available -- which is why it wasn't used directly in Tomcat. (NB: The > ThreadPoolExecutor class in Tomcat contains an "@author Doug Lea" tag. The > Tomcat source is licensed under AL2, the JDK source is licensed under GPL2, > but the original was released by Doug Lea into the public domain under a CC0 > 1.0 Deed license.) > > The re-sizing occurs in the ThreadPoolExecutor class if you'd like to read > it. It is > not entirely straightforward. You could start by reading the code for the > runWorker(Worker w) method where, at the end, processWorkerExit is called. > > But since Tomcat's ThreadPoolExecutor is basically Java's ThreadPoolExecutor, > they work the same. > > -chris > > ----- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org I took matters in to my own hands and wrote a test to see what was happening. Unlike java.util.concurrent.ThreadPoolExecutor, Tomcat's TPE adds threads when all core threads are busy. It only adds tasks to the queue when max threads are busy. I prefer this behavior to the JDK behavior. Threads are removed when idle for 60 seconds. This appears to be hard-coded in AbstreactEndpoint.createExecutor(). Thanks
Re: When does Tomcat add and remove threads?
John, On 3/11/24 18:14, john.e.gr...@wellsfargo.com.INVALID wrote: From: Christopher Schultz Sent: Monday, March 11, 2024 5:09 PM > On 3/11/24 17:47, john.e.gr...@wellsfargo.com.INVALID wrote: I am using Tomcat 9.x. When does Tomcat add and remove threads from its internal thread pool? I'm talking about the threads with names like http-nio-8080-exec-1. It appears the thread pool is Tomcat's own ThreadPoolExecutor but I don't see the exact behavior documented. I'm familiar with how java.util.concurrent does it, but it looks like Tomcat's version is a little different. >> Are you looking for a technical explanation with code references, or a plain-English description of when threads are created and added? > Mostly plain English like the j.u.c. ThreadPoolExecutor Java doc has. What happens when all core threads are in use? When do tasks go on the queue? When does core thread + 1 get added? When do threads get removed? Tomcat will create thread pools under two separate circumstances. They are related, but behave somewhat differently. First, if you declare an in your server.xml, then a thread pool will be created. You can control the number of threads and their retention policy such as "keep X spare threads around" and "retire threads after N seconds without being used." Second, if you declare a without specifying an "executor", a thread pool will be configured for you but you don't really have control over it because all those nice configuration options for an are not available on the . If you want to control those settings, use a linked with an . To be clear, if you declare a without an "executor" attribute, your thread pool will be of a fixed size and threads will never be released. (I think the thread pool starts small and grows but will never shrink.) An is implemented in the StandardThreadExecutor and ThreadPoolExecutor classes, which I believe were adaptations of classes from java.util.concurrent introduced into Tomcat before java.util.concurrent was actually available -- which is why it wasn't used directly in Tomcat. (NB: The ThreadPoolExecutor class in Tomcat contains an "@author Doug Lea" tag. The Tomcat source is licensed under AL2, the JDK source is licensed under GPL2, but the original was released by Doug Lea into the public domain under a CC0 1.0 Deed license.) The re-sizing occurs in the ThreadPoolExecutor class if you'd like to read it. It is not entirely straightforward. You could start by reading the code for the runWorker(Worker w) method where, at the end, processWorkerExit is called. But since Tomcat's ThreadPoolExecutor is basically Java's ThreadPoolExecutor, they work the same. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: When does Tomcat add and remove threads?
Chris, From: Christopher Schultz Sent: Monday, March 11, 2024 5:09 PM To: users@tomcat.apache.org Subject: Re: When does Tomcat add and remove threads? John, On 3/11/24 17:47, john.e.gr...@wellsfargo.com.INVALID wrote: > I am using Tomcat 9.x. > > When does Tomcat add and remove threads from its internal thread > pool? I'm talking about the threads with names like > http-nio-8080-exec-1. It appears the thread pool is Tomcat's own > ThreadPoolExecutor but I don't see the exact behavior documented. > I'm familiar with how java.util.concurrent does it, but it looks like > Tomcat's version is a little different. Are you looking for a technical explanation with code references, or a plain-English description of when threads are created and added? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Mostly plain English like the j.u.c. ThreadPoolExecutor Java doc has. What happens when all core threads are in use? When do tasks go on the queue? When does core thread + 1 get added? When do threads get removed? Thanks
Re: When does Tomcat add and remove threads?
John, On 3/11/24 17:47, john.e.gr...@wellsfargo.com.INVALID wrote: I am using Tomcat 9.x. When does Tomcat add and remove threads from its internal thread pool? I'm talking about the threads with names like http-nio-8080-exec-1. It appears the thread pool is Tomcat's own ThreadPoolExecutor but I don't see the exact behavior documented. I'm familiar with how java.util.concurrent does it, but it looks like Tomcat's version is a little different. Are you looking for a technical explanation with code references, or a plain-English description of when threads are created and added? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
When does Tomcat add and remove threads?
I am using Tomcat 9.x. When does Tomcat add and remove threads from its internal thread pool? I'm talking about the threads with names like http-nio-8080-exec-1. It appears the thread pool is Tomcat's own ThreadPoolExecutor but I don't see the exact behavior documented. I'm familiar with how java.util.concurrent does it, but it looks like Tomcat's version is a little different. Thanks
Re: Virtual Threads
On 07/09/2023 15:41, Christopher Schultz wrote: On 9/6/23 16:29, Mark Thomas wrote: There isn't much point using an executor with virtual threads. Okay then perche https://tomcat.apache.org/tomcat-11.0-doc/config/executor.html#Virtual_Thread_Implementation ? That is the internal executor we use to provide virtual threads to the connector. The same code exists in all current versions. Not sure why it was documented. It may have been while we were exploring what was possible. I'd lean towards removing that section from the Tomcat 11 docs. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Virtual Threads
Mark, On 9/6/23 16:29, Mark Thomas wrote: On 06/09/2023 21:24, Christopher Schultz wrote: On 9/6/23 03:29, Mark Thomas wrote: On 05/09/2023 22:02, Christopher Schultz wrote: Thanks for the correction. I just did a quick docs[1] search for "virtual" in Tomcat 10.x for example and I didn't see useVirtualThreads, so I assumed it wasn't in there. Maybe we need some documentation back-ports? Odd. It is there as an attribute on the Connector. And in 9.0.x and 8.5.x too. https://tomcat.apache.org/tomcat-10.0-doc/config/executor.html I don't find the word "virtual" anywhere on that page. It is a Connector attribute, not an Executor attribute. There isn't much point using an executor with virtual threads. Okay then perche https://tomcat.apache.org/tomcat-11.0-doc/config/executor.html#Virtual_Thread_Implementation ? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Virtual Threads
On 06/09/2023 21:24, Christopher Schultz wrote: On 9/6/23 03:29, Mark Thomas wrote: On 05/09/2023 22:02, Christopher Schultz wrote: Thanks for the correction. I just did a quick docs[1] search for "virtual" in Tomcat 10.x for example and I didn't see useVirtualThreads, so I assumed it wasn't in there. Maybe we need some documentation back-ports? Odd. It is there as an attribute on the Connector. And in 9.0.x and 8.5.x too. https://tomcat.apache.org/tomcat-10.0-doc/config/executor.html I don't find the word "virtual" anywhere on that page. It is a Connector attribute, not an Executor attribute. There isn't much point using an executor with virtual threads. For those of you that are finding this topic interesting, I'll have a talk on this at the ASF conference, Community Over Code in Halifax. https://communityovercode.org/schedule-list/#TH001 More broadly, there are usually a reasonable number of Tomcat committers present at ASF conferences so this is a great opportunity to speak face to face with the committers. +1 Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Virtual Threads
Mark, On 9/6/23 03:29, Mark Thomas wrote: On 05/09/2023 22:02, Christopher Schultz wrote: Mark, On 9/5/23 15:55, Mark Thomas wrote: On 05/09/2023 20:38, Christopher Schultz wrote: All, I have some questions about Virtual Threads and their use within Tomcat. Note that only Tomcat 11 currently has support for Virtual Threads when running on a version 19 or later JVM. Not quite. All current versions support virtual threads when running on Java 21 or later. Thanks for the correction. I just did a quick docs[1] search for "virtual" in Tomcat 10.x for example and I didn't see useVirtualThreads, so I assumed it wasn't in there. Maybe we need some documentation back-ports? Odd. It is there as an attribute on the Connector. And in 9.0.x and 8.5.x too. https://tomcat.apache.org/tomcat-10.0-doc/config/executor.html I don't find the word "virtual" anywhere on that page. The virtual thread scheduler uses a work stealing queue so it isn't quite FIFO. That sounds like nit-picking to me. It is, but it is nit-picking for a reason. Violetta did some testing which showed switching to virtual threads showed a small (about 1%) throughput improvement it also showed a much broader spread of latencies. That increase in spread is due to the work-stealing queue. For those users watch latency carefully, the change was viewed as a significant drawback of switching to virtual threads. That's a significant point, thanks for reiterating it. I think you have summed things up pretty well. I don't see a way with the current API to specify multiple virtual thread schedulers (which is what I think you would need to address this). Yeah, in all of the coverage I've seen online, there are no examples where Virtual Threads are used with an "executor" other than one that (a) accepts Runnable tasks and (b) just generates a Virtual Thread from that task which appears to be "bound" to the build-in default JVM-wide Virtual Thread executor. It would be potentially interesting to see an API which would allow different Executors to be used with Virtual Threads, even though the whole point of the entire thing is to avoid the (application-level) complexity of a thread pool/executor/etc. I think it does make sense, however, to be able to prioritize your threads a little bit. I haven't read anything about Virtual Threads and their priorities, so I assume it's just not part of anything user-facing at this point. I haven't seen anything that suggests that there will be any such API but to be fair, I haven't been following the Loom project that closely. For those of you that are finding this topic interesting, I'll have a talk on this at the ASF conference, Community Over Code in Halifax. https://communityovercode.org/schedule-list/#TH001 More broadly, there are usually a reasonable number of Tomcat committers present at ASF conferences so this is a great opportunity to speak face to face with the committers. +1 -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Virtual Threads
On 05/09/2023 22:02, Christopher Schultz wrote: Mark, On 9/5/23 15:55, Mark Thomas wrote: On 05/09/2023 20:38, Christopher Schultz wrote: All, I have some questions about Virtual Threads and their use within Tomcat. Note that only Tomcat 11 currently has support for Virtual Threads when running on a version 19 or later JVM. Not quite. All current versions support virtual threads when running on Java 21 or later. Thanks for the correction. I just did a quick docs[1] search for "virtual" in Tomcat 10.x for example and I didn't see useVirtualThreads, so I assumed it wasn't in there. Maybe we need some documentation back-ports? Odd. It is there as an attribute on the Connector. And in 9.0.x and 8.5.x too. The virtual thread scheduler uses a work stealing queue so it isn't quite FIFO. That sounds like nit-picking to me. It is, but it is nit-picking for a reason. Violetta did some testing which showed switching to virtual threads showed a small (about 1%) throughput improvement it also showed a much broader spread of latencies. That increase in spread is due to the work-stealing queue. For those users watch latency carefully, the change was viewed as a significant drawback of switching to virtual threads. I think you have summed things up pretty well. I don't see a way with the current API to specify multiple virtual thread schedulers (which is what I think you would need to address this). Yeah, in all of the coverage I've seen online, there are no examples where Virtual Threads are used with an "executor" other than one that (a) accepts Runnable tasks and (b) just generates a Virtual Thread from that task which appears to be "bound" to the build-in default JVM-wide Virtual Thread executor. It would be potentially interesting to see an API which would allow different Executors to be used with Virtual Threads, even though the whole point of the entire thing is to avoid the (application-level) complexity of a thread pool/executor/etc. I think it does make sense, however, to be able to prioritize your threads a little bit. I haven't read anything about Virtual Threads and their priorities, so I assume it's just not part of anything user-facing at this point. I haven't seen anything that suggests that there will be any such API but to be fair, I haven't been following the Loom project that closely. For those of you that are finding this topic interesting, I'll have a talk on this at the ASF conference, Community Over Code in Halifax. https://communityovercode.org/schedule-list/#TH001 More broadly, there are usually a reasonable number of Tomcat committers present at ASF conferences so this is a great opportunity to speak face to face with the committers. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Virtual Threads
Mark, On 9/5/23 15:55, Mark Thomas wrote: On 05/09/2023 20:38, Christopher Schultz wrote: All, I have some questions about Virtual Threads and their use within Tomcat. Note that only Tomcat 11 currently has support for Virtual Threads when running on a version 19 or later JVM. Not quite. All current versions support virtual threads when running on Java 21 or later. Thanks for the correction. I just did a quick docs[1] search for "virtual" in Tomcat 10.x for example and I didn't see useVirtualThreads, so I assumed it wasn't in there. Maybe we need some documentation back-ports? My (admittedly limited) understanding is that the use of Virtual Threads, specifically within Tomcat, will allow Tomcat to use fewer platform threads (e.g. native OS threads) to support a larger number of JVM/application threads for request-processing. The justification for this improvement is that request-processing threads spend a lot of time in an I/O-blocked state which, when using Virtual Threads, would no longer consume a platform thread. There appears to only be one setting to enable Virtual Threads in Tomcat: or In both cases, there is only one setting to affect the number of "threads" (by any description) which the Executor will ultimately use (Connectors without an explicit Executor will create a non-shared Executor to be used internally). That setting is "maxThreads" which limits the total number of "threads" that the Executor will create. The implementation of the VirtualThreadExecutor does not seem to have any upper-bound on the number of Virtual Threads that will be created at all. I believe this means that a large number of incoming requests (i.e. a burst) will result in the creation of a large number of Virtual Threads. Without an upper-bound, we are expecting that the JVM's (virtual) thread scheduler will schedule each application thread to be mounted to a platform thread in the order it was added to the queue (essentially in request-order). The virtual thread scheduler uses a work stealing queue so it isn't quite FIFO. That sounds like nit-picking to me. I suppose if each new Virtual Thread is assigned to a platform thread when it's initially queued, things can be executed in an not-strictly-FIFO-manner, but one has to assume work loads are equal (which they likely are not) and work is spread evenly across all the platform threads (which is likely) to have a general conversation about all this. I think the work-stealing ForkJoinPool is about as close to FIFO as you can get without introducing more expensive contention in order to enforce strict FIFO while not gaining much in the way of meaningful "fairness". But it's probably worth mentioning that the queue might not be "strictly fair". What *is* fair, though -- and maybe more fair than in the past? -- is that pipelined requests have to get back in line instead of jumping the queue. I think the old blocking JIO connector would allow pipelined requests to skip the queue, but all NIO-based connectors are "more fair" than that. Before Virtual Threads were introduced, the Executor would use a queue of requests to dispatch to available (non-Virtual) threads which would use that thread until the request was complete, then return the thread to the pool. With Virtual Threads, the same thing is essentially still happening except that (a) Tomcat no longer manages the thread pool (a JVM-defined one is being used instead) and (b) requests are immediately handed a (Virtual) thread to carry their execution. I believe there are some subtle differences in how Tomcat will behave, now. As an example, if I have two applications running, say, the Tomcat Manager application and the Tomcat Examples application, without using Virtual Threads, each application's thread pools should be "fair" within the context of each application: requests are processed in the order they are received in Manager and Examples, separately. If all requests are equally "expensive" and everything is "fair", then requests to the Examples application are scheduled alongside those to the Manager application, and they can both execute simultaneously as well as separately-manage the order in which the requests are processed. The above assumes each application has a separate thread pool (which implies a separate Connector). Yes, I'm sorry, I completely skipped over the "fact" that a separate would be used for such a "priority" application such as the Manager. My example was assuming that each application had the possibility to use a separate . Once Virtual Threads are introduced, the requests are filed into a single JVM-wide thread-scheduling system where activity in one application can affect the other. Correct. Let's replace Examples with RealWorldApp, an application that is expected to be
Re: Virtual Threads
On 05/09/2023 20:38, Christopher Schultz wrote: All, I have some questions about Virtual Threads and their use within Tomcat. Note that only Tomcat 11 currently has support for Virtual Threads when running on a version 19 or later JVM. Not quite. All current versions support virtual threads when running on Java 21 or later. My (admittedly limited) understanding is that the use of Virtual Threads, specifically within Tomcat, will allow Tomcat to use fewer platform threads (e.g. native OS threads) to support a larger number of JVM/application threads for request-processing. The justification for this improvement is that request-processing threads spend a lot of time in an I/O-blocked state which, when using Virtual Threads, would no longer consume a platform thread. There appears to only be one setting to enable Virtual Threads in Tomcat: or In both cases, there is only one setting to affect the number of "threads" (by any description) which the Executor will ultimately use (Connectors without an explicit Executor will create a non-shared Executor to be used internally). That setting is "maxThreads" which limits the total number of "threads" that the Executor will create. The implementation of the VirtualThreadExecutor does not seem to have any upper-bound on the number of Virtual Threads that will be created at all. I believe this means that a large number of incoming requests (i.e. a burst) will result in the creation of a large number of Virtual Threads. Without an upper-bound, we are expecting that the JVM's (virtual) thread scheduler will schedule each application thread to be mounted to a platform thread in the order it was added to the queue (essentially in request-order). The virtual thread scheduler uses a work stealing queue so it isn't quite FIFO. Before Virtual Threads were introduced, the Executor would use a queue of requests to dispatch to available (non-Virtual) threads which would use that thread until the request was complete, then return the thread to the pool. With Virtual Threads, the same thing is essentially still happening except that (a) Tomcat no longer manages the thread pool (a JVM-defined one is being used instead) and (b) requests are immediately handed a (Virtual) thread to carry their execution. I believe there are some subtle differences in how Tomcat will behave, now. As an example, if I have two applications running, say, the Tomcat Manager application and the Tomcat Examples application, without using Virtual Threads, each application's thread pools should be "fair" within the context of each application: requests are processed in the order they are received in Manager and Examples, separately. If all requests are equally "expensive" and everything is "fair", then requests to the Examples application are scheduled alongside those to the Manager application, and they can both execute simultaneously as well as separately-manage the order in which the requests are processed. The above assumes each application has a separate thread pool (which implies a separate Connector). Once Virtual Threads are introduced, the requests are filed into a single JVM-wide thread-scheduling system where activity in one application can affect the other. Correct. Let's replace Examples with RealWorldApp, an application that is expected to be used by users. Maybe a LOT of users. Without Virtual Threads, a high number of requests to RealWorldApp will not cause starvation of requests to (maybe the more important, at least for admins) the Manager. Once Virtual Threads are introduced, a limitless number of requests can be queued in front of a request to Manager, which can experience starvation. While Tomcat did not previously implement any specific priority-queuing of requests, the use of separate Executors for each application effectively provided that kind of thing. Yes, each Executor can be configured separately to either use Virtual Threads or not, and so Manager can be configured to NOT use Virtual Threads while RealWorldApp can be configured to use Virtual Threads and the balance is "restored". But it is no longer possible to have RealWorldApp and RealWorldApp2 and Manager all with equally probable request-scheduling when using Virtual Threads. You can pick some subset of applications to get (essentially) priority by NOT using Virtual Threads, but the whole set of applications running in a single JVM will share a single Executor with FIFO behavior. If an application creates Virtual Threads (hey, why not?! they are cheaper!) then they will be scheduled alongside the request-processing threads as well. Do I understand things correctly? Is my scenario of request-starvation for a little-used but potentially critical application (e.g. Manager) a real concern, or am I missing something fundamental about the way Virtual Threads are sch
Virtual Threads
All, I have some questions about Virtual Threads and their use within Tomcat. Note that only Tomcat 11 currently has support for Virtual Threads when running on a version 19 or later JVM. My (admittedly limited) understanding is that the use of Virtual Threads, specifically within Tomcat, will allow Tomcat to use fewer platform threads (e.g. native OS threads) to support a larger number of JVM/application threads for request-processing. The justification for this improvement is that request-processing threads spend a lot of time in an I/O-blocked state which, when using Virtual Threads, would no longer consume a platform thread. There appears to only be one setting to enable Virtual Threads in Tomcat: or In both cases, there is only one setting to affect the number of "threads" (by any description) which the Executor will ultimately use (Connectors without an explicit Executor will create a non-shared Executor to be used internally). That setting is "maxThreads" which limits the total number of "threads" that the Executor will create. The implementation of the VirtualThreadExecutor does not seem to have any upper-bound on the number of Virtual Threads that will be created at all. I believe this means that a large number of incoming requests (i.e. a burst) will result in the creation of a large number of Virtual Threads. Without an upper-bound, we are expecting that the JVM's (virtual) thread scheduler will schedule each application thread to be mounted to a platform thread in the order it was added to the queue (essentially in request-order). Before Virtual Threads were introduced, the Executor would use a queue of requests to dispatch to available (non-Virtual) threads which would use that thread until the request was complete, then return the thread to the pool. With Virtual Threads, the same thing is essentially still happening except that (a) Tomcat no longer manages the thread pool (a JVM-defined one is being used instead) and (b) requests are immediately handed a (Virtual) thread to carry their execution. I believe there are some subtle differences in how Tomcat will behave, now. As an example, if I have two applications running, say, the Tomcat Manager application and the Tomcat Examples application, without using Virtual Threads, each application's thread pools should be "fair" within the context of each application: requests are processed in the order they are received in Manager and Examples, separately. If all requests are equally "expensive" and everything is "fair", then requests to the Examples application are scheduled alongside those to the Manager application, and they can both execute simultaneously as well as separately-manage the order in which the requests are processed. Once Virtual Threads are introduced, the requests are filed into a single JVM-wide thread-scheduling system where activity in one application can affect the other. Let's replace Examples with RealWorldApp, an application that is expected to be used by users. Maybe a LOT of users. Without Virtual Threads, a high number of requests to RealWorldApp will not cause starvation of requests to (maybe the more important, at least for admins) the Manager. Once Virtual Threads are introduced, a limitless number of requests can be queued in front of a request to Manager, which can experience starvation. While Tomcat did not previously implement any specific priority-queuing of requests, the use of separate Executors for each application effectively provided that kind of thing. Yes, each Executor can be configured separately to either use Virtual Threads or not, and so Manager can be configured to NOT use Virtual Threads while RealWorldApp can be configured to use Virtual Threads and the balance is "restored". But it is no longer possible to have RealWorldApp and RealWorldApp2 and Manager all with equally probable request-scheduling when using Virtual Threads. You can pick some subset of applications to get (essentially) priority by NOT using Virtual Threads, but the whole set of applications running in a single JVM will share a single Executor with FIFO behavior. If an application creates Virtual Threads (hey, why not?! they are cheaper!) then they will be scheduled alongside the request-processing threads as well. Do I understand things correctly? Is my scenario of request-starvation for a little-used but potentially critical application (e.g. Manager) a real concern, or am I missing something fundamental about the way Virtual Threads are scheduled relative to other Virtual Threads, and relative to non-virtual threads? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
On 28/07/2023 13:50, Rémy Maucherat wrote:
On Thu, Jul 27, 2023 at 5:04 PM Mark Thomas wrote:
I've refactored things to the point where the unit tests run without
generating any warnings for pinning. I suspect further issues will be
identified over time and we can address those as they are found.
Ok, so the changes for Loom are: sync -> equivalent lock. That's ok
but it's a bit riskier (syncs are hard to avoid releasing).
True, but each lock I added should follow the
lock.lock();
try {
// Do stuff
} finally {
lock.unlock();
}
pattern. This is also one of things that SpotBugs checks for so we
should catch if any new locks get added incorrectly or if any of these
get changed.
Mark
Mark
On 25/07/2023 10:21, Mark Thomas wrote:
Never mind. Pretty much as soon as I hit send I managed to trigger the
issue.
Mark
On 25/07/2023 10:19, Mark Thomas wrote:
Daniel,
How did you trigger the pinning? I'm running some basic tests with
-Djdk.tracePinnedThreads=short and I'm not seeing any pinned threads
reported.
Mark
On 07/07/2023 13:45, Daniel Andres Pelaez Lopez wrote:
Mark,
Thanks for letting me know. I will wait for the August release to test.
Regards.
El jue, 6 jul 2023 a las 15:13, Mark Thomas ()
escribió:
6 Jul 2023 20:09:01 Daniel Andres Pelaez Lopez :
I am aware Tomcat community did a great effort to move Tomat to
Virtual Threads friendly, but I am not sure why HTTP2 was not part of
that effort?
The plan was always to see where the bottlenecks were as folks start to
experiment with Loom support and fix issues as they arose. It helps
focus
effort on where it is really needed.
These fixes look fairly simple. We should be able to get them done for
the August releases (the July releases have already been tagged). We
can
make -dev builds available earlier or you can build Tomcat from
source to
test the changes if you are interested.
As it happens I've spent most of today looking at ThreadLocal vs
SynchronizedStack vs new Object() in various places in the Tomcat code
base without reaching a clear conclusion. Which kind of proves the
point
that if we guess where bottlenecks might be we'll probably be wrong.
Mark
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
On Thu, Jul 27, 2023 at 5:04 PM Mark Thomas wrote: > > I've refactored things to the point where the unit tests run without > generating any warnings for pinning. I suspect further issues will be > identified over time and we can address those as they are found. Ok, so the changes for Loom are: sync -> equivalent lock. That's ok but it's a bit riskier (syncs are hard to avoid releasing). > Mark > > > On 25/07/2023 10:21, Mark Thomas wrote: > > Never mind. Pretty much as soon as I hit send I managed to trigger the > > issue. > > > > Mark > > > > > > On 25/07/2023 10:19, Mark Thomas wrote: > >> Daniel, > >> > >> How did you trigger the pinning? I'm running some basic tests with > >> -Djdk.tracePinnedThreads=short and I'm not seeing any pinned threads > >> reported. > >> > >> Mark > >> > >> > >> On 07/07/2023 13:45, Daniel Andres Pelaez Lopez wrote: > >>> Mark, > >>> > >>> Thanks for letting me know. I will wait for the August release to test. > >>> > >>> Regards. > >>> > >>> > >>> El jue, 6 jul 2023 a las 15:13, Mark Thomas () > >>> escribió: > >>>> > >>>> > >>>> 6 Jul 2023 20:09:01 Daniel Andres Pelaez Lopez : > >>>> > >>>>> I am aware Tomcat community did a great effort to move Tomat to > >>>>> Virtual Threads friendly, but I am not sure why HTTP2 was not part of > >>>>> that effort? > >>>> > >>>> The plan was always to see where the bottlenecks were as folks start to > >>>> experiment with Loom support and fix issues as they arose. It helps > >>>> focus > >>>> effort on where it is really needed. > >>>> > >>>> These fixes look fairly simple. We should be able to get them done for > >>>> the August releases (the July releases have already been tagged). We > >>>> can > >>>> make -dev builds available earlier or you can build Tomcat from > >>>> source to > >>>> test the changes if you are interested. > >>>> > >>>> As it happens I've spent most of today looking at ThreadLocal vs > >>>> SynchronizedStack vs new Object() in various places in the Tomcat code > >>>> base without reaching a clear conclusion. Which kind of proves the > >>>> point > >>>> that if we guess where bottlenecks might be we'll probably be wrong. > >>>> > >>>> Mark > >>>> > >>>> - > >>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >>>> For additional commands, e-mail: users-h...@tomcat.apache.org > >>>> > >>> > >>> > >> > >> - > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > >> For additional commands, e-mail: users-h...@tomcat.apache.org > >> > > > > - > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
I've refactored things to the point where the unit tests run without generating any warnings for pinning. I suspect further issues will be identified over time and we can address those as they are found. Mark On 25/07/2023 10:21, Mark Thomas wrote: Never mind. Pretty much as soon as I hit send I managed to trigger the issue. Mark On 25/07/2023 10:19, Mark Thomas wrote: Daniel, How did you trigger the pinning? I'm running some basic tests with -Djdk.tracePinnedThreads=short and I'm not seeing any pinned threads reported. Mark On 07/07/2023 13:45, Daniel Andres Pelaez Lopez wrote: Mark, Thanks for letting me know. I will wait for the August release to test. Regards. El jue, 6 jul 2023 a las 15:13, Mark Thomas () escribió: 6 Jul 2023 20:09:01 Daniel Andres Pelaez Lopez : I am aware Tomcat community did a great effort to move Tomat to Virtual Threads friendly, but I am not sure why HTTP2 was not part of that effort? The plan was always to see where the bottlenecks were as folks start to experiment with Loom support and fix issues as they arose. It helps focus effort on where it is really needed. These fixes look fairly simple. We should be able to get them done for the August releases (the July releases have already been tagged). We can make -dev builds available earlier or you can build Tomcat from source to test the changes if you are interested. As it happens I've spent most of today looking at ThreadLocal vs SynchronizedStack vs new Object() in various places in the Tomcat code base without reaching a clear conclusion. Which kind of proves the point that if we guess where bottlenecks might be we'll probably be wrong. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
Never mind. Pretty much as soon as I hit send I managed to trigger the issue. Mark On 25/07/2023 10:19, Mark Thomas wrote: Daniel, How did you trigger the pinning? I'm running some basic tests with -Djdk.tracePinnedThreads=short and I'm not seeing any pinned threads reported. Mark On 07/07/2023 13:45, Daniel Andres Pelaez Lopez wrote: Mark, Thanks for letting me know. I will wait for the August release to test. Regards. El jue, 6 jul 2023 a las 15:13, Mark Thomas () escribió: 6 Jul 2023 20:09:01 Daniel Andres Pelaez Lopez : I am aware Tomcat community did a great effort to move Tomat to Virtual Threads friendly, but I am not sure why HTTP2 was not part of that effort? The plan was always to see where the bottlenecks were as folks start to experiment with Loom support and fix issues as they arose. It helps focus effort on where it is really needed. These fixes look fairly simple. We should be able to get them done for the August releases (the July releases have already been tagged). We can make -dev builds available earlier or you can build Tomcat from source to test the changes if you are interested. As it happens I've spent most of today looking at ThreadLocal vs SynchronizedStack vs new Object() in various places in the Tomcat code base without reaching a clear conclusion. Which kind of proves the point that if we guess where bottlenecks might be we'll probably be wrong. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
Daniel, How did you trigger the pinning? I'm running some basic tests with -Djdk.tracePinnedThreads=short and I'm not seeing any pinned threads reported. Mark On 07/07/2023 13:45, Daniel Andres Pelaez Lopez wrote: Mark, Thanks for letting me know. I will wait for the August release to test. Regards. El jue, 6 jul 2023 a las 15:13, Mark Thomas () escribió: 6 Jul 2023 20:09:01 Daniel Andres Pelaez Lopez : I am aware Tomcat community did a great effort to move Tomat to Virtual Threads friendly, but I am not sure why HTTP2 was not part of that effort? The plan was always to see where the bottlenecks were as folks start to experiment with Loom support and fix issues as they arose. It helps focus effort on where it is really needed. These fixes look fairly simple. We should be able to get them done for the August releases (the July releases have already been tagged). We can make -dev builds available earlier or you can build Tomcat from source to test the changes if you are interested. As it happens I've spent most of today looking at ThreadLocal vs SynchronizedStack vs new Object() in various places in the Tomcat code base without reaching a clear conclusion. Which kind of proves the point that if we guess where bottlenecks might be we'll probably be wrong. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
Mark, Thanks for letting me know. I will wait for the August release to test. Regards. El jue, 6 jul 2023 a las 15:13, Mark Thomas () escribió: > > > 6 Jul 2023 20:09:01 Daniel Andres Pelaez Lopez : > > > I am aware Tomcat community did a great effort to move Tomat to > > Virtual Threads friendly, but I am not sure why HTTP2 was not part of > > that effort? > > The plan was always to see where the bottlenecks were as folks start to > experiment with Loom support and fix issues as they arose. It helps focus > effort on where it is really needed. > > These fixes look fairly simple. We should be able to get them done for > the August releases (the July releases have already been tagged). We can > make -dev builds available earlier or you can build Tomcat from source to > test the changes if you are interested. > > As it happens I've spent most of today looking at ThreadLocal vs > SynchronizedStack vs new Object() in various places in the Tomcat code > base without reaching a clear conclusion. Which kind of proves the point > that if we guess where bottlenecks might be we'll probably be wrong. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > -- Daniel Andrés Pelaez López Master’s Degree in IT Architectures, Universidad de los Andes. Software Construction Specialist, Universidad de los Andes. Bachelor's Degree in Computer Sciences, Universidad del Quindio. e. estigm...@gmail.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
6 Jul 2023 20:09:01 Daniel Andres Pelaez Lopez : I am aware Tomcat community did a great effort to move Tomat to Virtual Threads friendly, but I am not sure why HTTP2 was not part of that effort? The plan was always to see where the bottlenecks were as folks start to experiment with Loom support and fix issues as they arose. It helps focus effort on where it is really needed. These fixes look fairly simple. We should be able to get them done for the August releases (the July releases have already been tagged). We can make -dev builds available earlier or you can build Tomcat from source to test the changes if you are interested. As it happens I've spent most of today looking at ThreadLocal vs SynchronizedStack vs new Object() in various places in the Tomcat code base without reaching a clear conclusion. Which kind of proves the point that if we guess where bottlenecks might be we'll probably be wrong. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Pinned threads for HTTP2 using Virtual Threads on Tomcat 10.1.7
Hi community, I am working on a Spring Boot + Tomcat embedded application using Virtual Threads (https://openjdk.org/jeps/425), and everything looks great until we activate HTTP2. We started to see the following logs: Thread[#72,ForkJoinPool-1-worker-9,5,CarrierThreads] org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:86) <== monitors:1 Thread[#70,ForkJoinPool-1-worker-7,5,CarrierThreads] org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1609) <== monitors:1 Thread[#41,ForkJoinPool-1-worker-4,5,CarrierThreads] org.apache.coyote.http2.Http2AsyncUpgradeHandler.writeHeaders(Http2AsyncUpgradeHandler.java:199) <== monitors:1 org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:86) <== monitors:1 Thread[#40,ForkJoinPool-1-worker-3,5,CarrierThreads] org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:1000) <== monitors:1 org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:940) <== monitors:1 Thread[#61,ForkJoinPool-1-worker-6,5,CarrierThreads] Thread[#72,ForkJoinPool-1-worker-9,5,CarrierThreads] org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:86) <== monitors:1 Thread[#70,ForkJoinPool-1-worker-7,5,CarrierThreads] org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper$NioOperationState.run(NioEndpoint.java:1609) <== monitors:1 Thread[#41,ForkJoinPool-1-worker-4,5,CarrierThreads] org.apache.coyote.http2.Http2AsyncUpgradeHandler.writeHeaders(Http2AsyncUpgradeHandler.java:199) <== monitors:1 org.apache.coyote.http2.StreamProcessor.process(StreamProcessor.java:86) <== monitors:1 Thread[#40,ForkJoinPool-1-worker-3,5,CarrierThreads] org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:1000) <== monitors:1 org.apache.coyote.http2.Stream$StreamOutputBuffer.flush(Stream.java:940) <== monitors:1 Thread[#61,ForkJoinPool-1-worker-6,5,CarrierThreads] Inspecting the org.apache.coyote.http2.Stream class, we found several synchronized keywords (https://github.com/apache/tomcat/blob/main/java/org/apache/coyote/http2/Stream.java#L230). >From the JEP, this keyword will pin platform threads and might reduce the general throughput of the application. I am aware Tomcat community did a great effort to move Tomat to Virtual Threads friendly, but I am not sure why HTTP2 was not part of that effort? -- Daniel - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcats starting with 200 threads
Jon, Try using an . You are using a non-nuanced configuration and will therefore get inflexible threading behavior. Thanks, -chris On 9/20/22 02:47, Jonathan Yom-Tov wrote: hi Christopher, Configuration follows: On Mon, Sep 19, 2022 at 7:45 PM Christopher Schultz < ch...@christopherschultz.net> wrote: Jon, On 9/19/22 10:46, Jonathan Yom-Tov wrote: Sometimes one of our production Tomcats will start with the maximum (200) number of threads in the https pool. That is, it doesn't start with some minimum and works its way up to the maximum, it immediately starts with the maximum. There's no reason for it since most of the threads stay idle through the lifetime of the Tomcat. The issue is that this takes up memory and eventually something pushes the Tomcat over the edge and it dies with an out of memory error. Any ideas on how to debug or solve this? The Tomcat version is 9.0.64.0, running on Amazon Linux 2 (amd64) and Java Corretto 1.8.0_312-b07. Can you post your configuration? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcats starting with 200 threads
hi Christopher, Configuration follows: On Mon, Sep 19, 2022 at 7:45 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > Jon, > > On 9/19/22 10:46, Jonathan Yom-Tov wrote: > > Sometimes one of our production Tomcats will start with the maximum (200) > > number of threads in the https pool. That is, it doesn't start with some > > minimum and works its way up to the maximum, it immediately starts with > the > > maximum. There's no reason for it since most of the threads stay idle > > through the lifetime of the Tomcat. The issue is that this takes up > memory > > and eventually something pushes the Tomcat over the edge and it dies with > > an out of memory error. Any ideas on how to debug or solve this? > > > > The Tomcat version is 9.0.64.0, running on Amazon Linux 2 (amd64) and > Java > > Corretto 1.8.0_312-b07. > > Can you post your configuration? > > -chris > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: tomcats starting with 200 threads
Jon, On 9/19/22 10:46, Jonathan Yom-Tov wrote: Sometimes one of our production Tomcats will start with the maximum (200) number of threads in the https pool. That is, it doesn't start with some minimum and works its way up to the maximum, it immediately starts with the maximum. There's no reason for it since most of the threads stay idle through the lifetime of the Tomcat. The issue is that this takes up memory and eventually something pushes the Tomcat over the edge and it dies with an out of memory error. Any ideas on how to debug or solve this? The Tomcat version is 9.0.64.0, running on Amazon Linux 2 (amd64) and Java Corretto 1.8.0_312-b07. Can you post your configuration? -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcats starting with 200 threads
If the machine / VM / container you are running Tomcat on can't handle 200 threads you should be using a smaller value for maxThreads. Mark On 19/09/2022 15:46, Jonathan Yom-Tov wrote: hi, Sometimes one of our production Tomcats will start with the maximum (200) number of threads in the https pool. That is, it doesn't start with some minimum and works its way up to the maximum, it immediately starts with the maximum. There's no reason for it since most of the threads stay idle through the lifetime of the Tomcat. The issue is that this takes up memory and eventually something pushes the Tomcat over the edge and it dies with an out of memory error. Any ideas on how to debug or solve this? The Tomcat version is 9.0.64.0, running on Amazon Linux 2 (amd64) and Java Corretto 1.8.0_312-b07. Thanks, Jon. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
tomcats starting with 200 threads
hi, Sometimes one of our production Tomcats will start with the maximum (200) number of threads in the https pool. That is, it doesn't start with some minimum and works its way up to the maximum, it immediately starts with the maximum. There's no reason for it since most of the threads stay idle through the lifetime of the Tomcat. The issue is that this takes up memory and eventually something pushes the Tomcat over the edge and it dies with an out of memory error. Any ideas on how to debug or solve this? The Tomcat version is 9.0.64.0, running on Amazon Linux 2 (amd64) and Java Corretto 1.8.0_312-b07. Thanks, Jon.
Re: [OT] Running out of httpd worker threads (was: WELCOME to users@tomcat.apache.org)
Sahil, On 5/11/22 13:56, Verma, Sahil wrote: In our production environment, ApacheTomcat services went down. We have checked the logs and found below error - [Thu May 05 10:34:51.441668 2022] [mpm_event:error] [pid 27440:tid 140464737793792] AH00484: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting Please help to find the root cause of the issue why services got stopped. Kindly let us know if any other information required. This is actually a log message from Apache httpd, not Apache Tomcat. There are, however, plenty of Apache httpd experts on this mailing list as well. Note that the log message says [error] but it's not telling you there was a failure, it's just telling you that your server is under enough load (simultaneous client requests) that you may want to reconfigure the server to handle more requests. Whether that's appropriate depends upon how stresses your hardware is under those circumstances. -chris -Original Message- From: users-h...@tomcat.apache.org Sent: Wednesday, May 11, 2022 11:25 PM To: Verma, Sahil Subject: WELCOME to users@tomcat.apache.org Hi! This is the ezmlm program. I'm managing the users@tomcat.apache.org mailing list. I'm working for my owner, who can be reached at users-ow...@tomcat.apache.org. Acknowledgment: I have added the address sverm...@dxc.com to the users mailing list. Welcome to users@tomcat.apache.org! Please save this message so that you know the address you are subscribed under, in case you later want to unsubscribe or change your subscription address. --- Administrative commands for the users list --- I can handle administrative requests automatically. Please do not send them to the list address! Instead, send your message to the correct command address: To subscribe to the list, send a message to: To remove your address from the list, send a message to: Send mail to the following for info and FAQ for this list: Similar addresses exist for the digest list: To get messages 123 through 145 (a maximum of 100 per request), mail: To get an index with subject and author for messages 123-456 , mail: They are always returned as sets of 100, max 2000 per request, so you'll actually get 100-499. To receive all messages with the same subject as message 12345, send a short message to: The messages should contain one line or word of text to avoid being treated as sp@m, but I will ignore their content. Only the ADDRESS you send to is important. You can start a subscription for an alternate address, for example "john@host.domain", just add a hyphen and your address (with '=' instead of '@') after the command word: To stop subscription for this address, mail: In both cases, I'll send a confirmation message to that address. When you receive it, simply reply to it to complete your subscription. If despite following these instructions, you do not get the desired results, please contact my owner at users-ow...@tomcat.apache.org. Please be patient, my owner is a lot slower than I am ;-) --- Enclosed is a copy of the request I received. Return-Path: Received: (qmail 52135 invoked by uid 99); 11 May 2022 17:54:43 - Received: from spamproc1-he-de.apache.org (HELO spamproc1-he-de.apache.org) (116.203.196.100) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 11 May 2022 17:54:43 + Received: from localhost (localhost [127.0.0.1]) by spamproc1-he-de.apache.org (ASF Mail Server at spamproc1-he-de.apache.org) with ESMTP id BCD701FF0FC for ; Wed, 11 May 2022 17:54:42 + (UTC) X-Virus-Scanned: Debian amavisd-new at spamproc1-he-de.apache.org X-Spam-Flag: NO X-Spam-Score: 0.099 X-Spam-Level: X-Spam-Status: No, score=0.099 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, POISEN_SPAM_PILL=0.1, POISEN_SPAM_PILL_1=0.1, POISEN_SPAM_PILL_3=0.1, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamproc1-he-de.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=dxc.com Received: from mx1-he-de.apache.org ([116.203.227.195]) by localhost (spamproc1-he-de.apache.org [116.203.196.100]) (amavisd-new, port 10024) with ESMTP id TcSHZayvXWFL for ; Wed, 11 May 2022 17:54:41 + (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=67.219.250.2; helo=mail1.bemta35.messagelabs.com; envelope-from=sverm...@dxc.com; receiver= Received: from mail1.bemta35.messagelabs.com (mail1.bemta35.messagelabs.com [67.219.250.2]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 68A197ED15 for ; Wed, 11 May 2022 17:54:41 + (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=dxc.com; s=DXCSelector1; t=1652291673; i=@dxc.com;
RE: clearReferencesThreads issues warning about 2 threads, spawned by JDK in printing components
Thanks Mark, Unfortunately the URL in the error gets blocked by our security, so I looked it up on my home system now. Dream * Excel * Explore * Inspire Jon McAlexander Infrastructure Engineer Asst Vice President Middleware Product Engineering Enterprise CIO | EAS | Middleware | Infrastructure Solutions 8080 Cobblestone Rd | Urbandale, IA 50322 MAC: F4469-010 Tel 515-988-2508 | Cell 515-988-2508 jonmcalexan...@wellsfargo.com This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this for the addressee, you must not use, copy, disclose, or take any action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply e-mail and delete this message. Thank you for your cooperation. > -Original Message- > From: Mark Thomas > Sent: Monday, August 23, 2021 3:06 AM > To: users@tomcat.apache.org > Subject: Re: clearReferencesThreads issues warning about 2 threads, > spawned by JDK in printing components > > On 23/08/2021 08:10, Thomas Hoffmann (Speed4Trade GmbH) wrote: > > > > > Is there anything, the application can prevent this? > > Yes. Call Thread.setContextClassLoader(ClassLoader) before calling the code > that creates those threads, passing the common class loader. > Afterwards, reset the TCCL back to the web application class loader. > > > Should Tomcat maybe skip the warning if the thread is demonized? > > No. The threads have the web app class loader as their TCCL so you have a > potential memory leak. The warning is correct. > > > Or maybe these threads should be ignored when checking for orphaned > threads? > > No, for the same reason. > > > It was tested with Tomcat 9.0.52, Windows 10, Coretto-JDK 11.0.12_7. > > You might want to consider raising a bug against the JDK. It could be argued > that those threads should be created with a specific class loader to avoid > memory leaks in container environments. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org
Re: clearReferencesThreads issues warning about 2 threads, spawned by JDK in printing components
Mark, On 8/23/21 04:05, Mark Thomas wrote: On 23/08/2021 08:10, Thomas Hoffmann (Speed4Trade GmbH) wrote: Is there anything, the application can prevent this? Yes. Call Thread.setContextClassLoader(ClassLoader) before calling the code that creates those threads, passing the common class loader. Afterwards, reset the TCCL back to the web application class loader. Should Tomcat maybe skip the warning if the thread is demonized? No. The threads have the web app class loader as their TCCL so you have a potential memory leak. The warning is correct. Or maybe these threads should be ignored when checking for orphaned threads? No, for the same reason. It was tested with Tomcat 9.0.52, Windows 10, Coretto-JDK 11.0.12_7. You might want to consider raising a bug against the JDK. It could be argued that those threads should be created with a specific class loader to avoid memory leaks in container environments. +1 I looked at the JREMemoryLeakPreventionListener and it looks like it only initializes static stuff in the class using Class.forName(className(). I wonder if it would be helpful to also try to call a zero-arg constructor for the class if one exists. In this case, it would allow Tomcat (using this listener) to initialize this class (via instantiation) so the application doesn't have to mess-around with the TCCL. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: clearReferencesThreads issues warning about 2 threads, spawned by JDK in printing components
> Is there anything, the application can prevent this? Yes. Call Thread.setContextClassLoader(ClassLoader) before calling the code that creates those threads, passing the common class loader. Afterwards, reset the TCCL back to the web application class loader. > Should Tomcat maybe skip the warning if the thread is demonized? No. The threads have the web app class loader as their TCCL so you have a potential memory leak. The warning is correct. > Or maybe these threads should be ignored when checking for orphaned threads? No, for the same reason. > It was tested with Tomcat 9.0.52, Windows 10, Coretto-JDK 11.0.12_7. You might want to consider raising a bug against the JDK. It could be argued that those threads should be created with a specific class loader to avoid memory leaks in container environments. Mark Great, I will test this out and change the classloader temporarily. Many thanks for your quick help!
Re: clearReferencesThreads issues warning about 2 threads, spawned by JDK in printing components
On 23/08/2021 08:10, Thomas Hoffmann (Speed4Trade GmbH) wrote: Is there anything, the application can prevent this? Yes. Call Thread.setContextClassLoader(ClassLoader) before calling the code that creates those threads, passing the common class loader. Afterwards, reset the TCCL back to the web application class loader. Should Tomcat maybe skip the warning if the thread is demonized? No. The threads have the web app class loader as their TCCL so you have a potential memory leak. The warning is correct. Or maybe these threads should be ignored when checking for orphaned threads? No, for the same reason. It was tested with Tomcat 9.0.52, Windows 10, Coretto-JDK 11.0.12_7. You might want to consider raising a bug against the JDK. It could be argued that those threads should be created with a specific class loader to avoid memory leaks in container environments. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
clearReferencesThreads issues warning about 2 threads, spawned by JDK in printing components
Hello, we have an internal application which also supports printing. For this reason, we use the following code snippet: DocFlavor flavor = DocFlavor.SERVICE_FORMATTED.PRINTABLE; PrintRequestAttributeSet patts = new HashPrintRequestAttributeSet(); PrintServiceLookup.lookupPrintServices(flavor, patts); This triggers the JDK to spawn two threads, which can be seen in the OpenJDK-Sources: https://github.com/AdoptOpenJDK/openjdk-jdk11/blob/master/src/java.desktop/windows/classes/sun/print/PrintServiceLookupProvider.java (see around line no 134 ff) --> Thread thr = new Thread(null, new PrinterChangeListener(), "PrinterListener", 0, false); ... Thread remThr = new Thread(null, new RemotePrinterChangeListener(), "RemotePrinterListener", 0, false); During Undeployment or stopping Tomcat, we therefore get the following 2 Warnings: 21-Aug-2021 01:11:59.254 WARNUNG [Thread-50] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [PrinterListener] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.desktop@11.0.12/sun.print.PrintServiceLookupProvider.notifyLocalPrinterChange(Native Method) java.desktop@11.0.12/sun.print.PrintServiceLookupProvider$PrinterChangeListener.run(PrintServiceLookupProvider.java:316) java.base@11.0.12/java.lang.Thread.run(Thread.java:829) 21-Aug-2021 01:11:59.254 WARNUNG [Thread-50] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [ROOT] appears to have started a thread named [RemotePrinterListener] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread: java.desktop@11.0.12/sun.print.PrintServiceLookupProvider.notifyRemotePrinterChange(Native Method) java.desktop@11.0.12/sun.print.PrintServiceLookupProvider$RemotePrinterChangeListener.run(PrintServiceLookupProvider.java:323) java.base@11.0.12/java.lang.Thread.run(Thread.java:829) Despite the threads are deamonized, the warning is printed. Is there anything, the application can prevent this? Should Tomcat maybe skip the warning if the thread is demonized? Or maybe these threads should be ignored when checking for orphaned threads? It was tested with Tomcat 9.0.52, Windows 10, Coretto-JDK 11.0.12_7. Thanks in advance! Thomas - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: No Logger information when maximum threads reached. Tomcat 8
This is the third time you have posted essentially the same query to this list. Please see the responses to your previous 2 posts. Mark On 10/11/2020 10:16, Aquib Khan wrote: > Hi, > > > > We have a usecase where we want that our application should indicate > when it reaches maximum thread limit. Our application is deployed in > tomcat. Maximum thread limit of tomcat is 200, so If 200 threads are > reached, does tomcat provide any logger information? > > > > We tried making multiple rest calls and reducing max threads to 2-3. We > faced ConnectionRefused and socketTimeout Exceptions on client side. But > no exception or logger in Catalina logs. We changed Logging level as > well in logging.properties. > > > > We tried Changing values of maxThreads, minspareThread, acceptCount in > server.xml. We used Jmeter for making rest calls. > > > > Can we get any help here? Does tomcat provides any information when it > reaches max thread limit? Do we have to change any property in > server.xml or in logging.properties > > > > We are using tomcat 8. In previous tomcat versions, Exception used to > get printed when threads reaches limit. But in this version we didn’t > got any exception. > > > > Attaching server.xml and logging.properties snapshot. Please do reply > > > > Any help in this regard would be highly appreciable. > > > > Thanks in Advance. > > > > Regards, > > Aquib > > > > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
No Logger information when maximum threads reached. Tomcat 8
Hi, We have a usecase where we want that our application should indicate when it reaches maximum thread limit. Our application is deployed in tomcat. Maximum thread limit of tomcat is 200, so If 200 threads are reached, does tomcat provide any logger information? We tried making multiple rest calls and reducing max threads to 2-3. We faced ConnectionRefused and socketTimeout Exceptions on client side. But no exception or logger in Catalina logs. We changed Logging level as well in logging.properties. We tried Changing values of maxThreads, minspareThread, acceptCount in server.xml. We used Jmeter for making rest calls. Can we get any help here? Does tomcat provides any information when it reaches max thread limit? Do we have to change any property in server.xml or in logging.properties We are using tomcat 8. In previous tomcat versions, Exception used to get printed when threads reaches limit. But in this version we didn't got any exception. Attaching server.xml and logging.properties snapshot. Please do reply Any help in this regard would be highly appreciable. Thanks in Advance. Regards, Aquib - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Enable Logging to Print Max Threads
Aquib, On 10/28/20 08:31, Aquib Khan wrote: Hi, We are using Tomcat Version : 8.5.53. OS Name: Linux OS Version: 3.10.0-1127.18.2.el7.x86_64 We have a requirement where we wanted a logger to get printed in Catalina.out when Tomcat reaches it’s max thread limit. We reduced values of properties : /minSpareThreads, maxThreads and acceptCount/ in server.xml. Also changed logging level to ALL/DEBUG in logging.properties. When we made higher number of requests to tomcat we face */java.net.SocketTimeoutException: Read timed out/ *Exception on client side but no Log or Exception on Tomcat side. What change we have to perform, so that tomcat indicates it’s thread limit has reached or it is about to reach that limit. Tomcat doesn't bother to log when it hits its maximum thread limit because it's an unremarkable event. Who cares that you've hit your max? Well, obviously YOU care. The MXBeans for thread pools, etc. don't have any subscribable events. Theoretically, you could listen for attribute change notifications from an MXBean but none have been programmed into Tomcat. Attaching screenshot of server.xml and Logging.properties. Screenshots don't make it to this list. You gotta use text to communicate. If it's not important to know the exact moment you hit your high-water mark, you could poll Tomcat via JMX to find out what the current max-threads is and the current active-threads. If you have active-threads=max-threads=max-max-threads then you know that your server is saturated. -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Enable Logging to Print Max Threads
Hi, We are using Tomcat Version : 8.5.53. OS Name:Linux OS Version: 3.10.0-1127.18.2.el7.x86_64 We have a requirement where we wanted a logger to get printed in Catalina.out when Tomcat reaches it's max thread limit. We reduced values of properties : minSpareThreads, maxThreads and acceptCount in server.xml. Also changed logging level to ALL/DEBUG in logging.properties. When we made higher number of requests to tomcat we face java.net.SocketTimeoutException: Read timed out Exception on client side but no Log or Exception on Tomcat side. What change we have to perform, so that tomcat indicates it's thread limit has reached or it is about to reach that limit. Attaching screenshot of server.xml and Logging.properties. Any help would be appreciable Regards, Aquib - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 8/27/20 18:14, David wrote: >> I used the http to 8080 in order to read the Tomcat webmanager >> stats. I originally had issues with the JVM being too small, >> running out of memory, CPU spiking, threads maxing out, and >> whole system instability. Getting more machine memory and upping >> the JVM allocation has remedied all of that except for apparently >> the thread issue. What is the memory size of the server and of the JVM? >> I'm unsure that they were aging at that time as I couldn't get >> into anything, but with no room for GC to take place it would >> make sense that the threads would not be released. That's not usually an issue, unless the application is being a significant amount of memory during a request and then releasing it after the request has completed. >> My intention was to restart Tomcat nightly to lessen the chance >> of an occurrence until I could find a way to restart Tomcat based >> on the thread count and script a thread dump at the same time, >> (likely through Solarwinds). Now that you've explained that the >> NIO threads are a part of the system threads, I may be able to >> script something like that directly on the system, with a >> chrontab to check count, if > 295 contains NIO dump thread to / systemctl stop-start tomcat. I wouldn't do that. Just because the threads exist does not mean they are stuck. They may be doing useful work or otherwise running just fine. I would look for other ways to detect problems. >> That's very warming as it seems a viable way to get the data I >> need without posing much impact to users. Your explanation of >> threads leads me to believe that the nightly restart may be >> rather moot as it could likely be exhaustion on the downstream >> causing the backup on the front end. I didn't see these >> connected in this way and assumed they were asynchronous and >> independent processes. There are timeouts configured for all the >> DB2 backend connections, and I was in the mindset of the least >> timeout would kill all connections upstream/downstream by >> presenting the application a forcibly closed by remote host or a >> timeout. If you can suffer through a few more incidents, you can probably get a LOT more information about the root problem and maybe even get it solved, instead of just trying to stop the bleeding. >> I greatly appreciate the assistance, In looking through various >> articles none of this was really discussed because either >> everyone knows it, or maybe it was discussed on a level where I >> couldn't understand it, there certainly doesn't seem to be any >> other instances of connections being open for 18-45minutes or if >> there is it's not an issue for them. If you have a load-balancer (which you do), then I'd expect HTTP keep-alived to keep those connections open literally all day long, only maybe expiring when you have configured them to expire "just in case" or maybe after some amount of inactivity. For a lb-environment, I'd want those keep-alive timeouts to be fairly high so you don't waste any time re-constructing sockets between the lb and the app server . When an lb is NOT in the mix, you generally want /low/ keep-alive timeouts because you can't rely on clients sticking around for very long and you want to get them off your doorstep ASAP. >> During a normal glance at the manager page, there are no >> connections and maybe like 5 empty lines in a "Ready" stage, >> even if I spam the server's logon landing page I can never see a >> persistent connection, so it baffled me as to how connections >> could hang and build up, so I'm thinking something was perhaps >> messed up with the backend. If by "backend" you mean like databasse, etc. then that is probably the issue. The login page is (realtively) static, so it's very difficult to put Tomcat under such load that it's hosing just giving you that same page over and over again. I don't know what your "spamming" strategy is, but you might want to use a real load-generating tool like ApacheBench (ab) or, even better, JMeter which can actually swarm among several machines to basically DDoS your internal servers, which can be useful sometimes for stress-testing. But your tests really do have to comprise a realistic scenario, not just hammering on the login page all day. >> The webapp names /URL's for the oldest connections didn't >> coincide between the two outages, so I kind of brushed it off as >> being application specific, however it may still be. > >> I need it to occur again and get some dumps! Unfortunately, yes. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG w
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
On Thu, Aug 27, 2020 at 4:30 PM Christopher Schultz wrote: > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > David, > > On 8/27/20 17:14, David wrote: > > Thank you all for the replies! > > > > On Thu, Aug 27, 2020 at 3:53 PM Christopher Schultz > > wrote: > >> > > David, > > > > On 8/27/20 13:57, David wrote: > >>>> On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz > >>>> wrote: > >>>>> > >>>> David, > >>>> > >>>> On 8/27/20 10:48, David wrote: > >>>>>>> In the last two weeks I've had two occurrences where a > >>>>>>> single CentOS 7 production server hosting a public > >>>>>>> webpage has become unresponsive. The first time, all > >>>>>>> 300 available "https-jsse-nio-8443" threads were > >>>>>>> consumed, with the max age being around 45minutes, and > >>>>>>> all in a "S" status. This time all 300 were consumed in > >>>>>>> "S" status with the oldest being around ~16minutes. A > >>>>>>> restart of Tomcat on both occasions freed these threads > >>>>>>> and the website became responsive again. The > >>>>>>> connections are post/get methods which shouldn't take > >>>>>>> very long at all. > >>>>>>> > >>>>>>> CPU/MEM/JVM all appear to be within normal operating > >>>>>>> limits. I've not had much luck searching for articles > >>>>>>> for this behavior nor finding remedies. The default > >>>>>>> timeout values are used in both Tomcat and in the > >>>>>>> applications that run within as far as I can tell. > >>>>>>> Hopefully someone will have some insight on why the > >>>>>>> behavior could be occurring, why isn't Tomcat killing > >>>>>>> the connections? Even in a RST/ACK status, shouldn't > >>>>>>> Tomcat terminate the connection without an ACK from the > >>>>>>> client after the default timeout? > >>>> > >>>> Can you please post: > >>>> > >>>> 1. Complete Tomcat version > >>>>> I can't find anything more granular than 9.0.29, is there > >>>>> a command to show a sub patch level? > > > > 9.0.29 is the patch-level, so that's fine. You are about 10 > > versions out of date (~1 year). Any chance for an upgrade? > > > >> They had to re-dev many apps last year when we upgraded from I > >> want to say 1 or 3 or something equally as horrific. Hopefully > >> they are forward compatible with the newer releases and if not > >> should surely be tackled now before later, I will certainly bring > >> this to the table! > > I've rarely been bitten by an upgrade from foo.bar.x to foo.bar.y. > There is a recent caveat if you are using the AJP connector, but you > are not so it's not an issue for you. > > >>>> 2. Connector configuration (possibly redacted) > >>>>> This is the 8443 section of the server.xml *8080 is > >>>>> available during the outage and I'm able to curl the > >>>>> management page to see the 300 used threads, their status, > >>>>> and age* > >>>>> > >>>>> [snip] > >>>>> > >>>>> >>>>> connectionTimeout="2" redirectPort="8443" /> [snip] > >>>>> >>>>> protocol="org.apache.coyote.http11.Http11NioProtocol" > >>>>> maxThreads="300" SSLEnabled="true" > > >>>>> >>>>> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" > >>>>> > >>>>> > certificateKeystorePassword="redacted" type="RSA" /> > >>>>> [snip] >>>>> port="8443" > >>>>> protocol="org.apache.coyote.http11.Http11NioProtocol" > >>>>> maxThreads="300" SSLEnabled="true" > >>>>> protocols="TLSv1.2"> >>>>> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" > >>>>> > >>>>> > certificateKeystorePassword="redacted" type="RSA&qu
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 8/27/20 17:14, David wrote: > Thank you all for the replies! > > On Thu, Aug 27, 2020 at 3:53 PM Christopher Schultz > wrote: >> > David, > > On 8/27/20 13:57, David wrote: >>>> On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz >>>> wrote: >>>>> >>>> David, >>>> >>>> On 8/27/20 10:48, David wrote: >>>>>>> In the last two weeks I've had two occurrences where a >>>>>>> single CentOS 7 production server hosting a public >>>>>>> webpage has become unresponsive. The first time, all >>>>>>> 300 available "https-jsse-nio-8443" threads were >>>>>>> consumed, with the max age being around 45minutes, and >>>>>>> all in a "S" status. This time all 300 were consumed in >>>>>>> "S" status with the oldest being around ~16minutes. A >>>>>>> restart of Tomcat on both occasions freed these threads >>>>>>> and the website became responsive again. The >>>>>>> connections are post/get methods which shouldn't take >>>>>>> very long at all. >>>>>>> >>>>>>> CPU/MEM/JVM all appear to be within normal operating >>>>>>> limits. I've not had much luck searching for articles >>>>>>> for this behavior nor finding remedies. The default >>>>>>> timeout values are used in both Tomcat and in the >>>>>>> applications that run within as far as I can tell. >>>>>>> Hopefully someone will have some insight on why the >>>>>>> behavior could be occurring, why isn't Tomcat killing >>>>>>> the connections? Even in a RST/ACK status, shouldn't >>>>>>> Tomcat terminate the connection without an ACK from the >>>>>>> client after the default timeout? >>>> >>>> Can you please post: >>>> >>>> 1. Complete Tomcat version >>>>> I can't find anything more granular than 9.0.29, is there >>>>> a command to show a sub patch level? > > 9.0.29 is the patch-level, so that's fine. You are about 10 > versions out of date (~1 year). Any chance for an upgrade? > >> They had to re-dev many apps last year when we upgraded from I >> want to say 1 or 3 or something equally as horrific. Hopefully >> they are forward compatible with the newer releases and if not >> should surely be tackled now before later, I will certainly bring >> this to the table! I've rarely been bitten by an upgrade from foo.bar.x to foo.bar.y. There is a recent caveat if you are using the AJP connector, but you are not so it's not an issue for you. >>>> 2. Connector configuration (possibly redacted) >>>>> This is the 8443 section of the server.xml *8080 is >>>>> available during the outage and I'm able to curl the >>>>> management page to see the 300 used threads, their status, >>>>> and age* >>>>> >>>>> [snip] >>>>> >>>>> >>>> connectionTimeout="2" redirectPort="8443" /> [snip] >>>>> >>>> protocol="org.apache.coyote.http11.Http11NioProtocol" >>>>> maxThreads="300" SSLEnabled="true" > >>>>> >>>> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" >>>>> >>>>> certificateKeystorePassword="redacted" type="RSA" /> >>>>> [snip] >>>> port="8443" >>>>> protocol="org.apache.coyote.http11.Http11NioProtocol" >>>>> maxThreads="300" SSLEnabled="true" > >>>> protocols="TLSv1.2"> >>>> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" >>>>> >>>>> certificateKeystorePassword="redacted" type="RSA" /> >>>>> > > What, two connectors on one port? Do you get errors when starting? >> No errors, one is "with HTTP2" should I delete the other former? Well, one of them will succeed in starting the and other one should fail. Did you copy/paste your config without modification? Weird you don't have any errors. Usually you'll get an IOException or whatever binding to the port twice. > I don't see anything obviously problematic in the above > configuration (o
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
Thank you all for the replies! On Thu, Aug 27, 2020 at 3:53 PM Christopher Schultz wrote: > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > David, > > On 8/27/20 13:57, David wrote: > > On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz > > wrote: > >> > > David, > > > > On 8/27/20 10:48, David wrote: > >>>> In the last two weeks I've had two occurrences where a > >>>> single CentOS 7 production server hosting a public webpage > >>>> has become unresponsive. The first time, all 300 available > >>>> "https-jsse-nio-8443" threads were consumed, with the max age > >>>> being around 45minutes, and all in a "S" status. This time > >>>> all 300 were consumed in "S" status with the oldest being > >>>> around ~16minutes. A restart of Tomcat on both occasions > >>>> freed these threads and the website became responsive again. > >>>> The connections are post/get methods which shouldn't take > >>>> very long at all. > >>>> > >>>> CPU/MEM/JVM all appear to be within normal operating limits. > >>>> I've not had much luck searching for articles for this > >>>> behavior nor finding remedies. The default timeout values are > >>>> used in both Tomcat and in the applications that run within > >>>> as far as I can tell. Hopefully someone will have some > >>>> insight on why the behavior could be occurring, why isn't > >>>> Tomcat killing the connections? Even in a RST/ACK status, > >>>> shouldn't Tomcat terminate the connection without an ACK from > >>>> the client after the default timeout? > > > > Can you please post: > > > > 1. Complete Tomcat version > >> I can't find anything more granular than 9.0.29, is there a > >> command to show a sub patch level? > > 9.0.29 is the patch-level, so that's fine. You are about 10 versions > out of date (~1 year). Any chance for an upgrade? They had to re-dev many apps last year when we upgraded from I want to say 1 or 3 or something equally as horrific. Hopefully they are forward compatible with the newer releases and if not should surely be tackled now before later, I will certainly bring this to the table! > > > 2. Connector configuration (possibly redacted) > >> This is the 8443 section of the server.xml *8080 is available > >> during the outage and I'm able to curl the management page to see > >> the 300 used threads, their status, and age* >> name="Catalina"> > >> > >> [snip] > >> > >> >> connectionTimeout="2" redirectPort="8443" /> [snip] > >> >> protocol="org.apache.coyote.http11.Http11NioProtocol" > >> maxThreads="300" SSLEnabled="true" > > >> >> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" > >> certificateKeystorePassword="redacted" type="RSA" /> > >> [snip] >> protocol="org.apache.coyote.http11.Http11NioProtocol" > >> maxThreads="300" SSLEnabled="true" > >> protocols="TLSv1.2"> >> certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" > >> certificateKeystorePassword="redacted" type="RSA" /> > >> > > What, two connectors on one port? Do you get errors when starting? No errors, one is "with HTTP2" should I delete the other former? > > I don't see anything obviously problematic in the above configuration > (other than the double-definition of the 8443 connector). > > 300 tied-up connections (from your initial report) sounds like a > significant number: probably the thread count. Yes sir, that's the NIO thread count for the 8443 connector. > > Mark (as is often the case) is right: take some thread dumps next time > everything locks up and see what all those threads are doing. Often, > it's something like everything is awaiting on a db connection and the > db pool has been exhausted or something. Relatively simple quick-fixes > are available for that, and better, longer-term fixes as well. > Mark/Chris is there a way to dump the connector threads specifically? Or simply is it all contained as a machine/process thread? Sorry I'm not really a Linux guy. > > Do you have a single F5 or a group of them? > >> A group of them, several HA pairs depending on internal or > >> external and application. This server is behind one HA pa
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Felix, On 8/27/20 16:09, Felix Schumacher wrote: > > Am 27.08.20 um 19:35 schrieb Christopher Schultz: >> David, >> >> On 8/27/20 10:48, David wrote: >>> In the last two weeks I've had two occurrences where a single >>> CentOS 7 production server hosting a public webpage has become >>> unresponsive. The first time, all 300 available >>> "https-jsse-nio-8443" threads were consumed, with the max age >>> being around 45minutes, and all in a "S" status. This time all >>> 300 were consumed in "S" status with the oldest being around >>> ~16minutes. A restart of Tomcat on both occasions freed these >>> threads and the website became responsive again. The >>> connections are post/get methods which shouldn't take very long >>> at all. >> >>> CPU/MEM/JVM all appear to be within normal operating limits. >>> I've not had much luck searching for articles for this behavior >>> nor finding remedies. The default timeout values are used in >>> both Tomcat and in the applications that run within as far as I >>> can tell. Hopefully someone will have some insight on why the >>> behavior could be occurring, why isn't Tomcat killing the >>> connections? Even in a RST/ACK status, shouldn't Tomcat >>> terminate the connection without an ACK from the client after >>> the default timeout? >> >> Can you please post: >> >> 1. Complete Tomcat version 2. Connector configuration (possibly >> redacted) >> >>> Is there a graceful way to script the termination of threads >>> in case Tomcat isn't able to for whatever reason? >> >> Not really. > > (First look at Marks response on determining the root cause) > > Well, there might be a way (if it is sane, I don't know). You can > configure a valve to look for seemingly stuck threads and try to > interrupt them: > > http://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Stuck_Thread _Detection_Valve > > There are a few caveats there. First it is only working, when > both conditions are true > > * the servlets are synchronous * the stuck thread can be "freed" > with an Interrupt > > But really, if your threads are stuck for more than 15 minutes, you > have ample of time to take a thread dump and hopefully find the > root cause, so that you don't need this valve. This is a good idea as a band-aid, but the reality is that if you need the StuckThreadDetectionValve then your application is probably broken and needs to be fixed. Here are things that can be broken which might cause thread exhaustion: 1. Poor resource management. Things like db connections pools which can leak and/or not be refilled by the application. Everything stops when the db pool dries up. 2. Failure to set proper IO timeouts. Guess what the default read timeout is on a socket? Forever! If you read from a socket you might never hear back. Sounds like a problem. Set your read timeouts, kids. You might need to do this on your HTTP connections (and pools, and factories, and connection-wrappers like Apache http-client), your database config (usually in the config URL), and any remote-API libraries you are using (which use e.g. HTTP under the hood). - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9IICoACgkQHPApP6U8 pFgkuQ/+NE7tC+wWXoP2Ntv0yljJHyasRY/3dVewoNUxfO4CwcEkhSpK5YEkiHd3 sE7jygxEn3SHtHJ0WQPBWMAzL9RoLnglbAsxVXuWCzbQzd3tGCKOxQevCN3y+2ft jffqMEqOCgrN4kvKivj75V3alFQotT+jbZm1nJEwuQCLSJCqiHWcyCLlJF9Y6axn Thvsv40bnTKCPgqezo/0AYiYjQ9xIatTC3QDw129E7bofNKPBLk7LWcbg9CQBu+T iboA8IIxFgrOFYn66Mgx4kcJcQTRJ2XgdJ1v8p+mSITWH3UkLa5OhZeTqU6x2LDl LPuY8eC6y9QUqpFeEtaL72ZpDdYAn7Vcu4B3+D4Oobh7o2EJNQijIQ6A2QKIFw6e eBACKL0JJMwvfxVnp3nKIuoB3yOemMGZ8NpqUNcEn5mjmZubRWXXJXjtjjF5pGYW RRbMXvs3tFhLGsqnjVHQ/AV5MyuYKfl4Tqhvrz0u2oh0A8uo5Kq3CuHBDcLhLjs1 RkDiZuSdVugRFeq6hcQAyqO6LQ/QRhqtQ1hscecr9Iv8grvs8gzi4PvlurgBFqEF AuWe0V2WY0IJ9S7BqUUDr3Ij+0CQgxeQ70yyOztWjsT0B6ZPdOChm5Meu1+qi2ni EuT6Q5Lo2KHTqhrvi/RbTUXs+D6LSNFN6QFOzWtKWAr+gyrQjKg= =Ew/J -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 8/27/20 13:57, David wrote: > On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz > wrote: >> > David, > > On 8/27/20 10:48, David wrote: >>>> In the last two weeks I've had two occurrences where a >>>> single CentOS 7 production server hosting a public webpage >>>> has become unresponsive. The first time, all 300 available >>>> "https-jsse-nio-8443" threads were consumed, with the max age >>>> being around 45minutes, and all in a "S" status. This time >>>> all 300 were consumed in "S" status with the oldest being >>>> around ~16minutes. A restart of Tomcat on both occasions >>>> freed these threads and the website became responsive again. >>>> The connections are post/get methods which shouldn't take >>>> very long at all. >>>> >>>> CPU/MEM/JVM all appear to be within normal operating limits. >>>> I've not had much luck searching for articles for this >>>> behavior nor finding remedies. The default timeout values are >>>> used in both Tomcat and in the applications that run within >>>> as far as I can tell. Hopefully someone will have some >>>> insight on why the behavior could be occurring, why isn't >>>> Tomcat killing the connections? Even in a RST/ACK status, >>>> shouldn't Tomcat terminate the connection without an ACK from >>>> the client after the default timeout? > > Can you please post: > > 1. Complete Tomcat version >> I can't find anything more granular than 9.0.29, is there a >> command to show a sub patch level? 9.0.29 is the patch-level, so that's fine. You are about 10 versions out of date (~1 year). Any chance for an upgrade? > 2. Connector configuration (possibly redacted) >> This is the 8443 section of the server.xml *8080 is available >> during the outage and I'm able to curl the management page to see >> the 300 used threads, their status, and age* > name="Catalina"> >> >> [snip] >> >> > connectionTimeout="2" redirectPort="8443" /> [snip] >> > protocol="org.apache.coyote.http11.Http11NioProtocol" >> maxThreads="300" SSLEnabled="true" > >> > certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" >> certificateKeystorePassword="redacted" type="RSA" /> >> [snip] > protocol="org.apache.coyote.http11.Http11NioProtocol" >> maxThreads="300" SSLEnabled="true" > > protocols="TLSv1.2"> > certificateKeystoreFile="/opt/apache-tomcat-9.0.29/redacted.jks" >> certificateKeystorePassword="redacted" type="RSA" /> >> What, two connectors on one port? Do you get errors when starting? I don't see anything obviously problematic in the above configuration (other than the double-definition of the 8443 connector). 300 tied-up connections (from your initial report) sounds like a significant number: probably the thread count. Mark (as is often the case) is right: take some thread dumps next time everything locks up and see what all those threads are doing. Often, it's something like everything is awaiting on a db connection and the db pool has been exhausted or something. Relatively simple quick-fixes are available for that, and better, longer-term fixes as well. > Do you have a single F5 or a group of them? >> A group of them, several HA pairs depending on internal or >> external and application. This server is behind one HA pair and >> is a single server. Okay. Just remember that each F5 can make some large number of connections to Tomcat, so you need to make sure you can handle them. This was a much bigger deal back in the BIO days when thread limit = connection limit, and the thread limit was usually something like 250 - - 300. NIO is much better, and the default connection limit is 10k which "ought to be enough for anyone"[1]. - -chris [1] With apologies to Bill gates, who apparently never said anything of the sort. -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9IHUYACgkQHPApP6U8 pFgcMhAAsN/Fc0nG4EJ/aaxZtj46g7FW2UDLa3HcGI+r8mvI5pYlCxWO0Cm4oDHn PAEUsjNgDcyLbWPa+hIfTWZ2v594w8ACrprpdNWHoPhZ316LudpG3G8RWwrIVsOa pn6MmX79rvds1Htl2cbsZkaaNCg/3+dx5VgyQtexHopSP9FpU1swDwex4fIf/pFz jcl4SB6eMnKxHwf/avwEy6sfdN05ALCl6KfJBCA6vnRvMT8hYVGs5B/bDdPRU5zL 0cNIAlNaxrcw0G13cuOhg5KYG+eeKBKl2R/OiSmyn4+Xp7zzbl3G3i4GvfbYrwqe BFTcTGT3cTE3vwMcHmsskh2soxYcH3etWtJ2/XsrKoKdRqXpWybVyNEvHcUwhxdP h7SAN5V8D2+9a8Hhh8y/hUCHBOT70THUyBipYweV26wUj4ipOAiYiJ2UaCATwNzf E7Giv6D4Y9WQCU119HaQ65TLmvGTtfzctM5pJzrnRbI7LOpuo9/bNYxkYNoU8U9r sAgY4t3kvKNttetFnwdY5+JTM+yrFolYFkYMFv8vpaVyiumP4+dpbkniRAmLabWl O0kIn/bRTkek4ic/qCuawBi1Rc1hV1/1uUE1+t8XHG7Sfdd0vwUabZ8ZRxNUBWcc EliCfzyMosWcsgU2puNduPyXDeRxKb5gfe4VdfaH5xvfdqIpfgw= =SesB -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
Am 27.08.20 um 19:35 schrieb Christopher Schultz: > David, > > On 8/27/20 10:48, David wrote: > > In the last two weeks I've had two occurrences where a single > > CentOS 7 production server hosting a public webpage has become > > unresponsive. The first time, all 300 available > > "https-jsse-nio-8443" threads were consumed, with the max age being > > around 45minutes, and all in a "S" status. This time all 300 were > > consumed in "S" status with the oldest being around ~16minutes. A > > restart of Tomcat on both occasions freed these threads and the > > website became responsive again. The connections are post/get > > methods which shouldn't take very long at all. > > > CPU/MEM/JVM all appear to be within normal operating limits. I've > > not had much luck searching for articles for this behavior nor > > finding remedies. The default timeout values are used in both > > Tomcat and in the applications that run within as far as I can > > tell. Hopefully someone will have some insight on why the behavior > > could be occurring, why isn't Tomcat killing the connections? Even > > in a RST/ACK status, shouldn't Tomcat terminate the connection > > without an ACK from the client after the default timeout? > > Can you please post: > > 1. Complete Tomcat version > 2. Connector configuration (possibly redacted) > > > Is there a graceful way to script the termination of threads in > > case Tomcat isn't able to for whatever reason? > > Not really. (First look at Marks response on determining the root cause) Well, there might be a way (if it is sane, I don't know). You can configure a valve to look for seemingly stuck threads and try to interrupt them: http://tomcat.apache.org/tomcat-9.0-doc/config/valve.html#Stuck_Thread_Detection_Valve There are a few caveats there. First it is only working, when both conditions are true * the servlets are synchronous * the stuck thread can be "freed" with an Interrupt But really, if your threads are stuck for more than 15 minutes, you have ample of time to take a thread dump and hopefully find the root cause, so that you don't need this valve. Felix > > > My research for killing threads results in system threads or > > application threads, not Tomcat Connector connection threads, so > > I'm not sure if this is even viable. I'm also looking into ways to > > terminate these aged sessions via the F5. At this time I'm open to > > any suggestions that would be able to automate a resolution to > > keep the system from experiencing downtime, or for any insight on > > where to look for a root cause. Thanks in advance for any guidance > > you can lend. > It might actually be the F5 itself, especially if it opens up a large > number of connections to Tomcat and then tries to open additional ones > for some reason. If it opens 300 connections (which are then e.g. > leaked by the F5 internally) but the 301st is refused, then your > server is essentially inert from that point forward. > > NIO connectors default to max 10k connections so that's not likely the > actual problem, here, but it could be for some configurations. > > Do you have a single F5 or a group of them? > > -chris > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
On 27/08/2020 18:57, David wrote: > On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz > wrote: >>>> Is there a graceful way to script the termination of threads in >>>> case Tomcat isn't able to for whatever reason? > > Not really. What you can do is take a thread dump when this happens so you can see what the threads are doing. That should provide some insight to where the problem is. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
On Thu, Aug 27, 2020 at 12:35 PM Christopher Schultz wrote: > > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > David, > > On 8/27/20 10:48, David wrote: > > In the last two weeks I've had two occurrences where a single > > CentOS 7 production server hosting a public webpage has become > > unresponsive. The first time, all 300 available > > "https-jsse-nio-8443" threads were consumed, with the max age being > > around 45minutes, and all in a "S" status. This time all 300 were > > consumed in "S" status with the oldest being around ~16minutes. A > > restart of Tomcat on both occasions freed these threads and the > > website became responsive again. The connections are post/get > > methods which shouldn't take very long at all. > > > > CPU/MEM/JVM all appear to be within normal operating limits. I've > > not had much luck searching for articles for this behavior nor > > finding remedies. The default timeout values are used in both > > Tomcat and in the applications that run within as far as I can > > tell. Hopefully someone will have some insight on why the behavior > > could be occurring, why isn't Tomcat killing the connections? Even > > in a RST/ACK status, shouldn't Tomcat terminate the connection > > without an ACK from the client after the default timeout? > > Can you please post: > > 1. Complete Tomcat version I can't find anything more granular than 9.0.29, is there a command to show a sub patch level? > 2. Connector configuration (possibly redacted) This is the 8443 section of the server.xml *8080 is available during the outage and I'm able to curl the management page to see the 300 used threads, their status, and age* > > > Is there a graceful way to script the termination of threads in > > case Tomcat isn't able to for whatever reason? > > Not really. > > > My research for killing threads results in system threads or > > application threads, not Tomcat Connector connection threads, so > > I'm not sure if this is even viable. I'm also looking into ways to > > terminate these aged sessions via the F5. At this time I'm open to > > any suggestions that would be able to automate a resolution to > > keep the system from experiencing downtime, or for any insight on > > where to look for a root cause. Thanks in advance for any guidance > > you can lend. > It might actually be the F5 itself, especially if it opens up a large > number of connections to Tomcat and then tries to open additional ones > for some reason. If it opens 300 connections (which are then e.g. > leaked by the F5 internally) but the 301st is refused, then your > server is essentially inert from that point forward. > > NIO connectors default to max 10k connections so that's not likely the > actual problem, here, but it could be for some configurations. > > Do you have a single F5 or a group of them? A group of them, several HA pairs depending on internal or external and application. This server is behind one HA pair and is a single server. > > - -chris Thank you Chris! David > -BEGIN PGP SIGNATURE- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9H7tIACgkQHPApP6U8 > pFjR1hAAldbVnHDrV0W4aPLvcDwO/zi7qvrCscNjnJWhmR1m9TrevlrSb0EZvCJo > gTl7DXYEiZ9sBEdgs6AavHlk8jQ+ZbXbp8lsMElW5X9QoxxUD3YyJEpDSeHOG7/S > 2CyCYGzAQER0RlzBn9w97bCKWvUWoWDeLApd/pwdATjAo53hDtdNGdz6WdNLEzKm > g/BCZP0ynHZu7pMzSeZsOUBUXEKhDwHU+71fJo+WIJ4Gtiyb4xf2qkewvjQtuOGl > o/yESHNCJy09JAs3xK9W6eEVp981/Fuo4qDH32MJaXXbZRaNk32AwqngXKUhTW2l > BBl0jHoFIj+YJYc6AgVlv0la5qDIqP2VTv4ujOLBeF/95oP4sVRobIN4TiFyH6vv > ImvvRq55ML5NvKJv8g9Tj0aY5PusfwxcwyMCVovIof49vQXJUy7SbtgRB3eqgT8W > WwdBiGNsyWZpVjpr/CGBkBZmuR4wckeq0J5O/XGRFS9pK1jXH4gPnxe58vJmjA+P > RiSdp3SsU0P94SuF843CW+vmWyUu6SApCybUTwo5yiFXP2e/1+M9/fUGsykXpszU > zUvMcj9LWJ1QR3TbvEnwilsge4HKbUM3ZsFaujDjAVy6TAOgGS/dtVZ2UyMcrlOd > JMe3GeaOdM+ej27l5D8Eq6jaQcCfy+Mg9HxsYsbyrgrw3AhBhmo= > =eVIu > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 8/27/20 10:48, David wrote: > In the last two weeks I've had two occurrences where a single > CentOS 7 production server hosting a public webpage has become > unresponsive. The first time, all 300 available > "https-jsse-nio-8443" threads were consumed, with the max age being > around 45minutes, and all in a "S" status. This time all 300 were > consumed in "S" status with the oldest being around ~16minutes. A > restart of Tomcat on both occasions freed these threads and the > website became responsive again. The connections are post/get > methods which shouldn't take very long at all. > > CPU/MEM/JVM all appear to be within normal operating limits. I've > not had much luck searching for articles for this behavior nor > finding remedies. The default timeout values are used in both > Tomcat and in the applications that run within as far as I can > tell. Hopefully someone will have some insight on why the behavior > could be occurring, why isn't Tomcat killing the connections? Even > in a RST/ACK status, shouldn't Tomcat terminate the connection > without an ACK from the client after the default timeout? Can you please post: 1. Complete Tomcat version 2. Connector configuration (possibly redacted) > Is there a graceful way to script the termination of threads in > case Tomcat isn't able to for whatever reason? Not really. > My research for killing threads results in system threads or > application threads, not Tomcat Connector connection threads, so > I'm not sure if this is even viable. I'm also looking into ways to > terminate these aged sessions via the F5. At this time I'm open to > any suggestions that would be able to automate a resolution to > keep the system from experiencing downtime, or for any insight on > where to look for a root cause. Thanks in advance for any guidance > you can lend. It might actually be the F5 itself, especially if it opens up a large number of connections to Tomcat and then tries to open additional ones for some reason. If it opens 300 connections (which are then e.g. leaked by the F5 internally) but the 301st is refused, then your server is essentially inert from that point forward. NIO connectors default to max 10k connections so that's not likely the actual problem, here, but it could be for some configurations. Do you have a single F5 or a group of them? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9H7tIACgkQHPApP6U8 pFjR1hAAldbVnHDrV0W4aPLvcDwO/zi7qvrCscNjnJWhmR1m9TrevlrSb0EZvCJo gTl7DXYEiZ9sBEdgs6AavHlk8jQ+ZbXbp8lsMElW5X9QoxxUD3YyJEpDSeHOG7/S 2CyCYGzAQER0RlzBn9w97bCKWvUWoWDeLApd/pwdATjAo53hDtdNGdz6WdNLEzKm g/BCZP0ynHZu7pMzSeZsOUBUXEKhDwHU+71fJo+WIJ4Gtiyb4xf2qkewvjQtuOGl o/yESHNCJy09JAs3xK9W6eEVp981/Fuo4qDH32MJaXXbZRaNk32AwqngXKUhTW2l BBl0jHoFIj+YJYc6AgVlv0la5qDIqP2VTv4ujOLBeF/95oP4sVRobIN4TiFyH6vv ImvvRq55ML5NvKJv8g9Tj0aY5PusfwxcwyMCVovIof49vQXJUy7SbtgRB3eqgT8W WwdBiGNsyWZpVjpr/CGBkBZmuR4wckeq0J5O/XGRFS9pK1jXH4gPnxe58vJmjA+P RiSdp3SsU0P94SuF843CW+vmWyUu6SApCybUTwo5yiFXP2e/1+M9/fUGsykXpszU zUvMcj9LWJ1QR3TbvEnwilsge4HKbUM3ZsFaujDjAVy6TAOgGS/dtVZ2UyMcrlOd JMe3GeaOdM+ej27l5D8Eq6jaQcCfy+Mg9HxsYsbyrgrw3AhBhmo= =eVIu -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat 9.0.29 - HTTPS threads age, max connections reached, Tomcat not responding on 8443
In the last two weeks I've had two occurrences where a single CentOS 7 production server hosting a public webpage has become unresponsive. The first time, all 300 available "https-jsse-nio-8443" threads were consumed, with the max age being around 45minutes, and all in a "S" status. This time all 300 were consumed in "S" status with the oldest being around ~16minutes. A restart of Tomcat on both occasions freed these threads and the website became responsive again. The connections are post/get methods which shouldn't take very long at all. CPU/MEM/JVM all appear to be within normal operating limits. I've not had much luck searching for articles for this behavior nor finding remedies. The default timeout values are used in both Tomcat and in the applications that run within as far as I can tell. Hopefully someone will have some insight on why the behavior could be occurring, why isn't Tomcat killing the connections? Even in a RST/ACK status, shouldn't Tomcat terminate the connection without an ACK from the client after the default timeout? Is there a graceful way to script the termination of threads in case Tomcat isn't able to for whatever reason? My research for killing threads results in system threads or application threads, not Tomcat Connector connection threads, so I'm not sure if this is even viable. I'm also looking into ways to terminate these aged sessions via the F5.At this time I'm open to any suggestions that would be able to automate a resolution to keep the system from experiencing downtime, or for any insight on where to look for a root cause. Thanks in advance for any guidance you can lend. Thanks, David
Re: Websocket semaphore lock on close() blocks all tomcat threads
On 14/07/2020 21:08, Mark Thomas wrote: > On 14/07/2020 20:57, Sridhar Rao wrote: >> >> We notice a behavior with tomcat where it becomes unresponsive and all >> http threads go into a timed wait state and the node becomes unresponsive. >> >> Tomcat Version: 8.5.47 > > > >> Could this be a tomcat defect? > > Possibly. > > Let me take a look. I don't recall anything along these lines being > fixed since 8.5.47 but there might be. Yep. Looks like an issue in all current versions. Fixed for 10.0.x with back-ports to follow shortly. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Websocket semaphore lock on close() blocks all tomcat threads
On 14/07/2020 20:57, Sridhar Rao wrote: > > We notice a behavior with tomcat where it becomes unresponsive and all > http threads go into a timed wait state and the node becomes unresponsive. > > Tomcat Version: 8.5.47 > Could this be a tomcat defect? Possibly. Let me take a look. I don't recall anything along these lines being fixed since 8.5.47 but there might be. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Raghav, On 4/30/20 03:02, Ragavendhiran Bhiman (rabhiman) wrote: > Hello Chris, > > Please see my reply below in line. > > Thanks & Regards, > > Raghav > > On 30/04/20, 9:23 AM, "Christopher Schultz" wrote: > > Raghav, > > On 4/29/20 22:26, Ragavendhiran Bhiman (rabhiman) wrote: >> The below is the executer element. > >> > maxThreads="450" minSpareThreads="5"/> > >> I also captured the network pcap and able to see many RST >> packets in between that is marked as RED in wireshark. > > Okay, so far you have told us: > > 1. You are using Tomcat 8.5.29 >> [Raghav] yes correct. > 2. You have an with 450 threads in it >> [Raghav] Executor has 450 threads. > > 3. You see "lots of threads" >> [Raghav] Yes above 450 threads or all the. 450 threads are alive. >> How many more than 450? Or are you only concerned that all 450 threads are in use when you think they should not be? > 4. You are seeing lots of RST packets. >> [Raghav]I can provide one snapshot of pcap analyisis after the >> application data exchange only I am able to see the RST packet >> which I see could be ok or sent for socket closing. RST packets in and of themselves aren't cause for alarm. > We can't help you without more details. Pretend we aren't looking > at your screen as you investigate. > > How many threads are you seeing? More than 450? How many, exactly? > What are the names of the threads? > >> [Raghav] Yes all the 450 threads are alive. Okay, what's the problem? > Some things I have noticed that seem ... suspicious. > > 1. Your sslImplementationName is invalid. >> [Raghav] is the SSL implementation name has to be changed? What >> is the SSL name to be used? Usually it doesn't need to be specified. Sometimes, it makes sense to override the default. But you have overridden the default with a class that does not exist in Tomcat. I'm not sure what Tomcat will do if the class is invalid. Or maybe your developers have patched Tomcat to include a new SSLImplementation class? That would make sense if (a) you have an odd-looking sslImplementationName configuration setting and (b) your I/O channels are behaving strangely (because you have a broken I/O implementation). > 2. You have a 5-minute keepAliveTimeout -- which sounds insanely > high -- and an infinite number of keepalive requests. Are you > fronting Tomcat with a load-balancer or other reverse-proxy? >> [Raghav]. Yes, 5 minutes timeout is high. I already informed to >> the team. But people are saying previous versions are working >> properly where the large number of threads are not there. Okay, but this is still probably not a good idea. Are you indeed using a load-balancer? If you are, then having a long keep-alive timeout is actually a *good* thing, because it will improve performance between the lb and Tomcat. But no individual real-world client needs a 5-minute keep-alive timeout. >> I think the threads are based on the number of requests is it >> true? It's complicated. Usually, your thread pool will contain the number of threads equal to the high-water-mark of the number of simultaneous request (plus or minus a few, depending on timing). If it's not configured correctly, those threads will never be released. Whether or not that's a Bad Thing in your environment is up to you. > 3. You have sendReasonPhrase="true" which indicates that you are > working with clients which violate the HTTP specification. >> [Raghav] this doesn't matter. I don’t know how this has been set >> to true. > > 4. You have an infinite "maxSavePostSize" setting. Are you > expecting many users to perform unauthenticated POSTs where the > POST body needs to be very large, and saved-and-replayed during the > authentication step? >> [Raghav] This is not known. > > 5. Your keystoreType is PKCS11 which is usually a hardware > keystore. Fine. But you have a truststoreType of PKCS11 as well. > Are you using a hardware trust store as well? >> [Raghav] Not sure about this. There are many ciphers defined >> along with this as well. The ciphers are about communication with the client. The keystore is about getting the crypto started. The trust store is about trusting clients connecting to you. If you aren't using client TLS certificates, then you shouldn't have a trust store configured at all. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl6rMgkACgkQHPApP6U8 pFgDGRAAwpdgRhpTzSvXZQJurphNFBPHdmdXbq05gm6bdKpRcViMCU9J1SWQhAZU tO5md6T0EvWDIhr8jRfnAkH/9f1U5iF1dzIncdOjspJAfbkt0mUVd6ARbWr/NPLl iE78
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
Hello Chris,
Please see my reply below in line.
Thanks & Regards,
Raghav
On 30/04/20, 9:23 AM, "Christopher Schultz"
wrote:
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Raghav,
On 4/29/20 22:26, Ragavendhiran Bhiman (rabhiman) wrote:
> The below is the executer element.
>
> maxThreads="450" minSpareThreads="5"/>
>
> I also captured the network pcap and able to see many RST packets
> in between that is marked as RED in wireshark.
Okay, so far you have told us:
1. You are using Tomcat 8.5.29
[Raghav] yes correct.
2. You have an with 450 threads in it
[Raghav] Executor has 450 threads.
3. You see "lots of threads"
[Raghav] Yes above 450 threads or all the. 450 threads are alive.
4. You are seeing lots of RST packets.
[Raghav]I can provide one snapshot of pcap analyisis after the application data
exchange only I am able to see the RST packet which I see could be ok or sent
for socket closing.
We can't help you without more details. Pretend we aren't looking at
your screen as you investigate.
How many threads are you seeing? More than 450? How many, exactly?
What are the names of the threads?
[Raghav] Yes all the 450 threads are alive.
Some things I have noticed that seem ... suspicious.
1. Your sslImplementationName is invalid.
[Raghav] is the SSL implementation name has to be changed? What is the SSL name
to be used?
2. You have a 5-minute keepAliveTimeout -- which sounds insanely high
- -- and an infinite number of keepalive requests. Are you fronting
Tomcat with a load-balancer or other reverse-proxy?
[Raghav]. Yes, 5 minutes timeout is high. I already informed to the team. But
people are saying previous versions are working properly where the large number
of threads are not there.
I think the threads are based on the number of requests is it true?
3. You have sendReasonPhrase="true" which indicates that you are
working with clients which violate the HTTP specification.
[Raghav] this doesn't matter. I don’t know how this has been set to true.
4. You have an infinite "maxSavePostSize" setting. Are you expecting
many users to perform unauthenticated POSTs where the POST body needs
to be very large, and saved-and-replayed during the authentication step?
[Raghav] This is not known.
5. Your keystoreType is PKCS11 which is usually a hardware keystore.
Fine. But you have a truststoreType of PKCS11 as well. Are you using a
hardware trust store as well?
[Raghav] Not sure about this. There are many ciphers defined along with this as
well.
- -chris
>
> Thanks & Regards,
>
> Raghav
>
>
> On 29/04/20, 9:52 PM, "Mark Thomas" wrote:
>
> On 29/04/2020 14:53, Ragavendhiran Bhiman (rabhiman) wrote:
>> Yes you are correct apache tomcat version 8.5.29 being used.
>>
>> On 29/04/20, 7:22 PM, "Ragavendhiran Bhiman (rabhiman)"
>> wrote:
>>
>> Hi Mark,
>>
>> We have configured 450 threads for port number 443 with the
>> following executer
>
> That is a Connector element, not the executor element. We need
> both.
>
> Mark
>
>
>>
>> > protocol="org.apache.coyote.http11.Http11NioProtocol"
>> SSLEnabled="true" maxKeepAliveRequests="-1"
>> keepAliveTimeout="30" executor="AdminExecutorPool"
>> maxSavePostSize="-1" scheme="https" secure="true"
>> enableLookups="false" disableUploadTimeout="true"
>> acceptCount="100" compression="on"
>> compressableMimeType="text/html,text/json,text/javascript,text/css,ap
plication/javascript"
>
>>
> sslEnabledProtocols="${sslEnabledProtocolsHighSecurity}" server="
> "
>> allowUnsafeLegacyRenegotiation="false" clientAuth="false"
>> bindOnInit="false" URIEncoding="UTF-8"
>> useBodyEncodingForURI="true" keystoreType="PKCS11"
>> keyAlias="tomcat" truststoreType="PKCS11"
>> sendReasonPhrase="true"
>> sslImplementationName="org.apache.tomcat.util.net.jsse.IseJSSEImpleme
ntation"
>
>>
>
>> />
>>
>> I could see 450 threads open for servicing the clients in one
>> specific setup only what could be the reason?
>>
>> Thanks a lot.
>>
>> Regards,
>>
>> Raghav
>>
>>
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256
Raghav,
On 4/29/20 22:26, Ragavendhiran Bhiman (rabhiman) wrote:
> The below is the executer element.
>
> maxThreads="450" minSpareThreads="5"/>
>
> I also captured the network pcap and able to see many RST packets
> in between that is marked as RED in wireshark.
Okay, so far you have told us:
1. You are using Tomcat 8.5.29
2. You have an with 450 threads in it
3. You see "lots of threads"
4. You are seeing lots of RST packets
We can't help you without more details. Pretend we aren't looking at
your screen as you investigate.
How many threads are you seeing? More than 450? How many, exactly?
What are the names of the threads?
Some things I have noticed that seem ... suspicious.
1. Your sslImplementationName is invalid.
2. You have a 5-minute keepAliveTimeout -- which sounds insanely high
- -- and an infinite number of keepalive requests. Are you fronting
Tomcat with a load-balancer or other reverse-proxy?
3. You have sendReasonPhrase="true" which indicates that you are
working with clients which violate the HTTP specification.
4. You have an infinite "maxSavePostSize" setting. Are you expecting
many users to perform unauthenticated POSTs where the POST body needs
to be very large, and saved-and-replayed during the authentication step?
5. Your keystoreType is PKCS11 which is usually a hardware keystore.
Fine. But you have a truststoreType of PKCS11 as well. Are you using a
hardware trust store as well?
- -chris
>
> Thanks & Regards,
>
> Raghav
>
>
> On 29/04/20, 9:52 PM, "Mark Thomas" wrote:
>
> On 29/04/2020 14:53, Ragavendhiran Bhiman (rabhiman) wrote:
>> Yes you are correct apache tomcat version 8.5.29 being used.
>>
>> On 29/04/20, 7:22 PM, "Ragavendhiran Bhiman (rabhiman)"
>> wrote:
>>
>> Hi Mark,
>>
>> We have configured 450 threads for port number 443 with the
>> following executer
>
> That is a Connector element, not the executor element. We need
> both.
>
> Mark
>
>
>>
>> > protocol="org.apache.coyote.http11.Http11NioProtocol"
>> SSLEnabled="true" maxKeepAliveRequests="-1"
>> keepAliveTimeout="30" executor="AdminExecutorPool"
>> maxSavePostSize="-1" scheme="https" secure="true"
>> enableLookups="false" disableUploadTimeout="true"
>> acceptCount="100" compression="on"
>> compressableMimeType="text/html,text/json,text/javascript,text/css,ap
plication/javascript"
>
>>
> sslEnabledProtocols="${sslEnabledProtocolsHighSecurity}" server="
> "
>> allowUnsafeLegacyRenegotiation="false" clientAuth="false"
>> bindOnInit="false" URIEncoding="UTF-8"
>> useBodyEncodingForURI="true" keystoreType="PKCS11"
>> keyAlias="tomcat" truststoreType="PKCS11"
>> sendReasonPhrase="true"
>> sslImplementationName="org.apache.tomcat.util.net.jsse.IseJSSEImpleme
ntation"
>
>>
>
>> />
>>
>> I could see 450 threads open for servicing the clients in one
>> specific setup only what could be the reason?
>>
>> Thanks a lot.
>>
>> Regards,
>>
>> Raghav
>>
>>
>> On 29/04/20, 7:18 PM, "Mark Thomas" wrote:
>>
>> On 29/04/2020 14:24, Ragavendhiran Bhiman (rabhiman) wrote:
>>> Apache version 8.5.29
>>
>> Given this is the Apache Tomcat mailing list and that that is a
>> valid, although rather old, Tomcat version number I assume you
>> mean you are using Apache Tomcat 8.5.29.
>>
>> Generally, please also include JVM vendor and version being used
>> as well as OS.
>>
>>
>>
>>> Hi,
>>>
>>> I am seeing too many open threads to port number 443 with
>>> TLSv1.2, what could be the primary reason for the same?
>>
>> Open threads? That doesn't make sense. Do you mean open ports,
>> threads (idle, active, both) or something else?
>>
>> How are you defining "too many"? More than you expect? There is
>> an error? Something else?
>>
>>> How can I analyze the problem ? Any particular pointers if you
>>> could provide will be more helpful.
>>
>> That depends on what the problem turns out to be.
>>
>> Mark
>>
>> -
>
>>
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
>> For additional commands, e-
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
Hi Mark,
The below is the executer element.
I also captured the network pcap and able to see many RST packets in between
that is marked as RED in wireshark.
Thanks & Regards,
Raghav
On 29/04/20, 9:52 PM, "Mark Thomas" wrote:
On 29/04/2020 14:53, Ragavendhiran Bhiman (rabhiman) wrote:
> Yes you are correct apache tomcat version 8.5.29 being used.
>
> On 29/04/20, 7:22 PM, "Ragavendhiran Bhiman (rabhiman)"
wrote:
>
> Hi Mark,
>
> We have configured 450 threads for port number 443 with the following
executer
That is a Connector element, not the executor element. We need both.
Mark
>
> port="443"
protocol="org.apache.coyote.http11.Http11NioProtocol" SSLEnabled="true"
> maxKeepAliveRequests="-1"
> keepAliveTimeout="30"
> executor="AdminExecutorPool"
> maxSavePostSize="-1"
> scheme="https" secure="true"
> enableLookups="false" disableUploadTimeout="true"
> acceptCount="100"
> compression="on"
>
compressableMimeType="text/html,text/json,text/javascript,text/css,application/javascript"
> sslEnabledProtocols="${sslEnabledProtocolsHighSecurity}"
server=" "
> allowUnsafeLegacyRenegotiation="false"
> clientAuth="false"
> bindOnInit="false"
> URIEncoding="UTF-8"
> useBodyEncodingForURI="true"
> keystoreType="PKCS11"
> keyAlias="tomcat"
> truststoreType="PKCS11"
> sendReasonPhrase="true"
>
sslImplementationName="org.apache.tomcat.util.net.jsse.IseJSSEImplementation"
>
> />
>
> I could see 450 threads open for servicing the clients in one
specific setup only what could be the reason?
>
> Thanks a lot.
>
> Regards,
>
> Raghav
>
>
> On 29/04/20, 7:18 PM, "Mark Thomas" wrote:
>
> On 29/04/2020 14:24, Ragavendhiran Bhiman (rabhiman) wrote:
> > Apache version 8.5.29
>
> Given this is the Apache Tomcat mailing list and that that is a
valid,
> although rather old, Tomcat version number I assume you mean you
are
> using Apache Tomcat 8.5.29.
>
> Generally, please also include JVM vendor and version being used
as well
> as OS.
>
>
>
> > Hi,
> >
> > I am seeing too many open threads to port number 443 with
TLSv1.2, what could be the primary reason for the same?
>
> Open threads? That doesn't make sense. Do you mean open ports,
threads
> (idle, active, both) or something else?
>
> How are you defining "too many"? More than you expect? There is an
> error? Something else?
>
> > How can I analyze the problem ? Any particular pointers if you
could provide will be more helpful.
>
> That depends on what the problem turns out to be.
>
> Mark
>
>
-
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
On 29/04/2020 14:53, Ragavendhiran Bhiman (rabhiman) wrote:
> Yes you are correct apache tomcat version 8.5.29 being used.
>
> On 29/04/20, 7:22 PM, "Ragavendhiran Bhiman (rabhiman)"
> wrote:
>
> Hi Mark,
>
> We have configured 450 threads for port number 443 with the following
> executer
That is a Connector element, not the executor element. We need both.
Mark
>
> port="443" protocol="org.apache.coyote.http11.Http11NioProtocol"
> SSLEnabled="true"
> maxKeepAliveRequests="-1"
> keepAliveTimeout="30"
> executor="AdminExecutorPool"
> maxSavePostSize="-1"
> scheme="https" secure="true"
> enableLookups="false" disableUploadTimeout="true"
> acceptCount="100"
> compression="on"
>
> compressableMimeType="text/html,text/json,text/javascript,text/css,application/javascript"
> sslEnabledProtocols="${sslEnabledProtocolsHighSecurity}"
> server=" "
> allowUnsafeLegacyRenegotiation="false"
> clientAuth="false"
> bindOnInit="false"
> URIEncoding="UTF-8"
> useBodyEncodingForURI="true"
> keystoreType="PKCS11"
> keyAlias="tomcat"
> truststoreType="PKCS11"
> sendReasonPhrase="true"
>
> sslImplementationName="org.apache.tomcat.util.net.jsse.IseJSSEImplementation"
>
> />
>
> I could see 450 threads open for servicing the clients in one specific
> setup only what could be the reason?
>
> Thanks a lot.
>
> Regards,
>
> Raghav
>
>
> On 29/04/20, 7:18 PM, "Mark Thomas" wrote:
>
> On 29/04/2020 14:24, Ragavendhiran Bhiman (rabhiman) wrote:
> > Apache version 8.5.29
>
> Given this is the Apache Tomcat mailing list and that that is a valid,
> although rather old, Tomcat version number I assume you mean you are
> using Apache Tomcat 8.5.29.
>
> Generally, please also include JVM vendor and version being used as
> well
> as OS.
>
>
>
> > Hi,
> >
> > I am seeing too many open threads to port number 443 with TLSv1.2,
> what could be the primary reason for the same?
>
> Open threads? That doesn't make sense. Do you mean open ports, threads
> (idle, active, both) or something else?
>
> How are you defining "too many"? More than you expect? There is an
> error? Something else?
>
> > How can I analyze the problem ? Any particular pointers if you
> could provide will be more helpful.
>
> That depends on what the problem turns out to be.
>
> Mark
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
>
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
Yes you are correct apache tomcat version 8.5.29 being used.
On 29/04/20, 7:22 PM, "Ragavendhiran Bhiman (rabhiman)"
wrote:
Hi Mark,
We have configured 450 threads for port number 443 with the following
executer
I could see 450 threads open for servicing the clients in one specific
setup only what could be the reason?
Thanks a lot.
Regards,
Raghav
On 29/04/20, 7:18 PM, "Mark Thomas" wrote:
On 29/04/2020 14:24, Ragavendhiran Bhiman (rabhiman) wrote:
> Apache version 8.5.29
Given this is the Apache Tomcat mailing list and that that is a valid,
although rather old, Tomcat version number I assume you mean you are
using Apache Tomcat 8.5.29.
Generally, please also include JVM vendor and version being used as well
as OS.
> Hi,
>
> I am seeing too many open threads to port number 443 with TLSv1.2,
what could be the primary reason for the same?
Open threads? That doesn't make sense. Do you mean open ports, threads
(idle, active, both) or something else?
How are you defining "too many"? More than you expect? There is an
error? Something else?
> How can I analyze the problem ? Any particular pointers if you could
provide will be more helpful.
That depends on what the problem turns out to be.
Mark
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
Hi Mark, We have configured 450 threads for port number 443 with the following executer I could see 450 threads open for servicing the clients in one specific setup only what could be the reason? Thanks a lot. Regards, Raghav On 29/04/20, 7:18 PM, "Mark Thomas" wrote: On 29/04/2020 14:24, Ragavendhiran Bhiman (rabhiman) wrote: > Apache version 8.5.29 Given this is the Apache Tomcat mailing list and that that is a valid, although rather old, Tomcat version number I assume you mean you are using Apache Tomcat 8.5.29. Generally, please also include JVM vendor and version being used as well as OS. > Hi, > > I am seeing too many open threads to port number 443 with TLSv1.2, what could be the primary reason for the same? Open threads? That doesn't make sense. Do you mean open ports, threads (idle, active, both) or something else? How are you defining "too many"? More than you expect? There is an error? Something else? > How can I analyze the problem ? Any particular pointers if you could provide will be more helpful. That depends on what the problem turns out to be. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
On 29/04/2020 14:24, Ragavendhiran Bhiman (rabhiman) wrote: > Apache version 8.5.29 Given this is the Apache Tomcat mailing list and that that is a valid, although rather old, Tomcat version number I assume you mean you are using Apache Tomcat 8.5.29. Generally, please also include JVM vendor and version being used as well as OS. > Hi, > > I am seeing too many open threads to port number 443 with TLSv1.2, what could > be the primary reason for the same? Open threads? That doesn't make sense. Do you mean open ports, threads (idle, active, both) or something else? How are you defining "too many"? More than you expect? There is an error? Something else? > How can I analyze the problem ? Any particular pointers if you could provide > will be more helpful. That depends on what the problem turns out to be. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
Apache version 8.5.29 From: "Ragavendhiran Bhiman (rabhiman)" Date: Wednesday, 29 April 2020 at 6:50 PM To: "users-ow...@tomcat.apache.org" , "users@tomcat.apache.org" Subject: Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads Adding tomcat users as well. From: "Ragavendhiran Bhiman (rabhiman)" Date: Wednesday, 29 April 2020 at 6:45 PM To: "users-ow...@tomcat.apache.org" Subject: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads Hi, I am seeing too many open threads to port number 443 with TLSv1.2, what could be the primary reason for the same? How can I analyze the problem ? Any particular pointers if you could provide will be more helpful. Thanks & Regards, Raghavendran +91 8220757651.
Re: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads
Adding tomcat users as well. From: "Ragavendhiran Bhiman (rabhiman)" Date: Wednesday, 29 April 2020 at 6:45 PM To: "users-ow...@tomcat.apache.org" Subject: Some questions regarding the TLS1.2 port 443 continuously communicating and too many open threads Hi, I am seeing too many open threads to port number 443 with TLSv1.2, what could be the primary reason for the same? How can I analyze the problem ? Any particular pointers if you could provide will be more helpful. Thanks & Regards, Raghavendran +91 8220757651.
Re: manager app, Complete Server Status, not shown list of NIO threads
Hi, Dne 05.12.2018 v 16:39 Christopher Schultz napsal(a): -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jan, On 12/5/18 06:59, Jan Vávra wrote: Hi, Dne 04.12.2018 v 17:49 Christopher Schultz napsal(a): Jan, On 12/4/18 10:10, Jan Vávra wrote: Hello, I'm using Apache Tomcat/8.5.35, jvm 1.8.0_192-b12, Windows Server 2012 R2 and at Complete Server Status page I can see list of all http-nio threads and I can see a header of ajp-nio threads. But there is displayed only a label Max threads: and nothing more. In the localhost.log is an exception: 04-Dec-2018 10:31:38.109 SEVERE [96] org.apache.catalina.core.StandardHostValve.invoke Exception Processing null java.lang.NullPointerException at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrappe rVa lve.java:236) at org.apache.catalina.core.StandardContextValve.invoke(StandardContex tVa lve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenti cat orBase.java:493) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve .ja va:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve .ja va:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAc ces sLogValve.java:650) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineV alv e.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.j ava :342) at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:479) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorL igh t.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(Abstra ctP rotocol.java:806) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEnd poi nt.java:1498) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorB ase .java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto r.j ava:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut or. java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Task Thr ead.java:61) at java.lang.Thread.run(Thread.java:748) At the version Apache Tomcat/8.5.34 this problem doesn't occur. Did I found a bug? Something looks seriously broken if you got an NPE on that line of code. Either the Valve doesn't have a Container or the Container doesn't have a logger, neither of which should ever happen. What's your use-case? Is this embedded, or using Tomcat "normally" like starting-up from a script/service and deploying web applications to it? The tomcat is started as a Windows service tomcat7.exe. The services was installed using service.bat install. User connections are received in Apache Server and reverse proxied using mod_ajp. The manager app I've opened locally at RDP at url http://localhost:8080/manager/html. Did you upgrade from 8.5.34 and then it broke? How did you perform the upgrade? I upgraded from 8.5.9 to 8.5.35 just copying from apache-tomcat-8.5.35-windows-x64.zip. I didn't copy only the conf/server.xml If I copied files from 8.5.34 the problem didn't occur. So you took files from the ZIP file and dropped them on top of your existing Tomcat installation? Yes. Before that I've made a directory synchronization to view what files are new, modified and same. Which files? All of them? And then I've copied all of them except server.xml where I have my customizations. Generally, conf/server.xml is customized for a particular environment. If you copied everything, you will probably have lost something important. I do not understand how I can lost something important, if I've copied everything. There could be a jar hell when in the zip would be jar with newer version numbers in name. But this is not this issue for upgrade from 8.5.9 to 8.5.39. No new jar is in the zip. I've also text compared source a my server.xml to inspect some new sections in source server.xml and have nothing found. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlwH8UUACgkQHPApP6U8 pFgD+BAAwoP66vzrt5yMgiEPW9h8iUc3c8C7TXVTBLcPB1zGhjlE0r+OEhpDv2u/ nvtlLOHMJKfqDKmDsRL8CN/qjdieZaGS4jGgDmISgBlUAMskzKf0KO5IFTqME/1p VKd74+cPaEZTOBqS0iwJ3iEuUczy5ArxyHeCwe7gtKLecUJot9p8MmAZ3vbVZfs+ wKHYvz2TgI4DW6LHYlVVkzE5fxsJcJXMZj0DoDIjvzy1IGfMbsIYgnUpPtqh4neC uGBc1GaOIjH4uK81jr6A9CxtFaSpsek+1cZzXKDZwn0LKaY6hzAieXc+EH+22Cp3 DjPqOzwJA2fBW5ZuOUpSnCpxvtZtz8HhuouzZA8HgsgZdXhfGiBv980CigMUoOoh 71gXSi5AgOcsarD3R+mwu5UAgmYKEtIcslfrpQEE7CziRCLJhyuEZvSt6Yz48v/B q5Jpd2/FT/qf91C/X/l1Ndd21kApQYZw6C4AAOIuxP7ZLdtUjlme2dKIB46w88pq zwluTi3Rnk2qFBYfTlaK5TCINj3d1r0kguzMwxoEyDbK86FOOqX9qlEKnty3JdBw ieOpQ4+t+O1/ogLfqFwX2Tce9MdWtDFMkWS/FLW1O4LdLklvKElb6Q75ZUmRqrrQ FS4IvMfXaMWkOr53MM/G7Zs37kp7F5k4MF7yyafNTjFP3yK8/Og= =R5+S -END PGP SIGNATURE- - To unsubscribe, e-mail: user
Re: manager app, Complete Server Status, not shown list of NIO threads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jan, On 12/5/18 06:59, Jan Vávra wrote: > Hi, > > Dne 04.12.2018 v 17:49 Christopher Schultz napsal(a): Jan, > > On 12/4/18 10:10, Jan Vávra wrote: >>>> Hello, I'm using Apache Tomcat/8.5.35, jvm 1.8.0_192-b12, >>>> Windows Server 2012 R2 and at Complete Server Status page I >>>> can see list of all http-nio threads and I can see a header >>>> of ajp-nio threads. But there is displayed only a label Max >>>> threads: and nothing more. >>>> >>>> In the localhost.log is an exception: 04-Dec-2018 >>>> 10:31:38.109 SEVERE [96] >>>> org.apache.catalina.core.StandardHostValve.invoke Exception >>>> Processing null java.lang.NullPointerException at >>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrappe rVa > >>>> lve.java:236) >>>> at >>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContex tVa > >>>> lve.java:96) >>>> at >>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenti cat > >>>> orBase.java:493) >>>> at >>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve .ja > >>>> va:140) >>>> at >>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve .ja > >>>> va:81) >>>> at >>>> org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAc ces > >>>> sLogValve.java:650) >>>> at >>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineV alv > >>>> e.java:87) >>>> at >>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.j ava > >>>> :342) >>>> > at > org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:479) >>>> at >>>> org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorL igh > >>>> t.java:66) >>>> at >>>> org.apache.coyote.AbstractProtocol$ConnectionHandler.process(Abstra ctP > >>>> rotocol.java:806) >>>> at >>>> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEnd poi > >>>> nt.java:1498) >>>> at >>>> org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorB ase > >>>> .java:49) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecuto r.j > >>>> ava:1149) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecut or. > >>>> java:624) >>>> at >>>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(Task Thr > >>>> ead.java:61) >>>> at java.lang.Thread.run(Thread.java:748) >>>> >>>> At the version Apache Tomcat/8.5.34 this problem doesn't >>>> occur. Did I found a bug? > Something looks seriously broken if you got an NPE on that line of > code. Either the Valve doesn't have a Container or the Container > doesn't have a logger, neither of which should ever happen. > > What's your use-case? Is this embedded, or using Tomcat "normally" > like starting-up from a script/service and deploying web > applications to it? >> The tomcat is started as a Windows service tomcat7.exe. The >> services was installed using service.bat install. User >> connections are received in Apache Server and reverse proxied >> using mod_ajp. The manager app I've opened locally at RDP at url >> http://localhost:8080/manager/html. > > > Did you upgrade from 8.5.34 and then it broke? How did you perform > the upgrade? >> I upgraded from 8.5.9 to 8.5.35 just copying from >> apache-tomcat-8.5.35-windows-x64.zip. I didn't copy only the >> conf/server.xml If I copied files from 8.5.34 the problem didn't >> occur. So you took files from the ZIP file and dropped them on top of your existing Tomcat installation? Which files? All of them? Generally, conf/server.xml is customized for a particular environment. If you copied everything, you will probably have lost something important. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlwH8UUACgkQHPApP6U8 pFgD+BAAwoP66vzrt5yMgiEPW9h8iUc3c8C7TXVTBLcPB1zGhjlE0r+OEhpDv2u/ nvtlLOHMJKfqDKmDsRL8CN/qjdieZaGS4jGgDmISgBlUAMskzKf0KO5IFTqME/1p VKd74+cPaEZTOBqS0iwJ3iEuUczy5ArxyHeCwe7gtKLecUJot9p8MmAZ3vbVZfs+ wKHYvz2TgI4DW6LHYlVVkzE5fxsJcJXMZj0DoDIjvzy1IGfMbsIYgnUpPtqh4neC uGBc1GaOIjH4uK81jr6A9CxtFaSpsek+1cZzXKDZwn0LKaY6hzAieXc+EH+22Cp3 DjPqOzwJA2fBW5ZuOUpSnCpxvtZtz8HhuouzZA8HgsgZdXhfGiBv980CigMUoOoh 71gXSi5AgOcsarD3R+mwu5UAgmYKEtIcslfrpQEE7CziRCLJhyuEZvSt6Yz48v/B q5Jpd2/FT/qf91C/X/l1Ndd21kApQYZw6C4AAOIuxP7ZLdtUjlme2dKIB46w88pq zwluTi3Rnk2qFBYfTlaK5TCINj3d1r0kguzMwxoEyDbK86FOOqX9qlEKnty3JdBw ieOpQ4+t+O1/ogLfqFwX2Tce9MdWtDFMkWS/FLW1O4LdLklvKElb6Q75ZUmRqrrQ FS4IvMfXaMWkOr53MM/G7Zs37kp7F5k4MF7yyafNTjFP3yK8/Og= =R5+S -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: manager app, Complete Server Status, not shown list of NIO threads
Hi, Dne 04.12.2018 v 17:49 Christopher Schultz napsal(a): -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jan, On 12/4/18 10:10, Jan Vávra wrote: Hello, I'm using Apache Tomcat/8.5.35, jvm 1.8.0_192-b12, Windows Server 2012 R2 and at Complete Server Status page I can see list of all http-nio threads and I can see a header of ajp-nio threads. But there is displayed only a label Max threads: and nothing more. In the localhost.log is an exception: 04-Dec-2018 10:31:38.109 SEVERE [96] org.apache.catalina.core.StandardHostValve.invoke Exception Processing null java.lang.NullPointerException at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa lve.java:236) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa lve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticat orBase.java:493) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja va:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja va:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAcces sLogValve.java:650) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv e.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java :342) at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:479) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLigh t.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractP rotocol.java:806) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoi nt.java:1498) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase .java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j ava:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThr ead.java:61) at java.lang.Thread.run(Thread.java:748) At the version Apache Tomcat/8.5.34 this problem doesn't occur. Did I found a bug? Something looks seriously broken if you got an NPE on that line of code. Either the Valve doesn't have a Container or the Container doesn't have a logger, neither of which should ever happen. What's your use-case? Is this embedded, or using Tomcat "normally" like starting-up from a script/service and deploying web applications to it? The tomcat is started as a Windows service tomcat7.exe. The services was installed using service.bat install. User connections are received in Apache Server and reverse proxied using mod_ajp. The manager app I've opened locally at RDP at url http://localhost:8080/manager/html. Did you upgrade from 8.5.34 and then it broke? How did you perform the upgrade? I upgraded from 8.5.9 to 8.5.35 just copying from apache-tomcat-8.5.35-windows-x64.zip. I didn't copy only the conf/server.xml If I copied files from 8.5.34 the problem didn't occur. - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlwGsBUACgkQHPApP6U8 pFg7RxAAvmaaNGfOSGut8l5xHktCpf7B8Uy4/nChr/gAeGuY6Vk0vKcB88cdecGU q70+atyU6r03sUWuIbfM9OQc7DhPS5mZHifCOM8Yn77ZK6pJERYg+tz4m/f7M979 Cj7QUIHl8/i21qULL6Y5OQyV/N5U9lAnKzGRJrVcDghcl+pYrEBE3P+VyOXduTvr rOjYxNq2NP+D0B4JTy1D+Wa7+BFsqrAwm3TjQ7vxCg3wW4OWwDUM7SKYCCDMCYHV kLHIgJFRSwoZIGhDI4HYGjssY1H1q4DCTJPduQi0RZI/IcvrHNjRff3S9HXnzs5n 86efvPHh8SUmp42dFwWzlXq6E7O9tzoCOTbAn+SEr3EQG8H2m2cQ8Y66zaoXZGMY hN+cPtMfimwCpI3bO46yEB6YipZbf4fjn6MXIbJk+GWEafuviEg0t/uIUGRslBpl CpzJb1lgZZUIdmn3xvdPsCzo1Ot5saC2lEJbzCnBjjR8d9RqlrGZKBn5L6h43zxV QlEK8YfF11YckXKSM/Kr8/AorYzmifcd6aAQJJtCPkQWE49YDzbCOIAqy6ooR7Br fCpfs1cUsemNn0nfUYWMN/4X/NDfrsFhdxLveI8/bjMCFfYCxyH0V5wYN69oZaWL N8Vt74TrJ4gqukQTwvq85xdgVdLj9zQgTtVci5+AamjDe6ydTjI= =Wtha -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Jan. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: manager app, Complete Server Status, not shown list of NIO threads
On 04/12/2018 15:10, Jan Vávra wrote: > Hello, > I'm using Apache Tomcat/8.5.35, jvm 1.8.0_192-b12, Windows Server 2012 > R2 and at Complete Server Status page I can see list of all http-nio > threads and I can see a header of ajp-nio threads. But there is > displayed only a label Max threads: > and nothing more. That matches this: https://bz.apache.org/bugzilla/show_bug.cgi?id=62918 > In the localhost.log is an exception: > 04-Dec-2018 10:31:38.109 SEVERE [96] > org.apache.catalina.core.StandardHostValve.invoke Exception Processing null > java.lang.NullPointerException > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:236) although the exception is different. Odd. I'll echo Chris's comments about the exception. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: manager app, Complete Server Status, not shown list of NIO threads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jan, On 12/4/18 10:10, Jan Vávra wrote: > Hello, I'm using Apache Tomcat/8.5.35, jvm 1.8.0_192-b12, Windows > Server 2012 R2 and at Complete Server Status page I can see list of > all http-nio threads and I can see a header of ajp-nio threads. But > there is displayed only a label Max threads: and nothing more. > > In the localhost.log is an exception: 04-Dec-2018 10:31:38.109 > SEVERE [96] org.apache.catalina.core.StandardHostValve.invoke > Exception Processing null java.lang.NullPointerException at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperVa lve.java:236) > > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextVa lve.java:96) > > at > org.apache.catalina.authenticator.AuthenticatorBase.invoke(Authenticat orBase.java:493) > > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.ja va:140) > > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.ja va:81) > > at > org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAcces sLogValve.java:650) > > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValv e.java:87) > > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java :342) > > at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:479) > at > org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLigh t.java:66) > > at > org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractP rotocol.java:806) > > at > org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoi nt.java:1498) > > at > org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase .java:49) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j ava:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. java:624) > > at > org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThr ead.java:61) > > at java.lang.Thread.run(Thread.java:748) > > At the version Apache Tomcat/8.5.34 this problem doesn't occur. Did > I found a bug? Something looks seriously broken if you got an NPE on that line of code. Either the Valve doesn't have a Container or the Container doesn't have a logger, neither of which should ever happen. What's your use-case? Is this embedded, or using Tomcat "normally" like starting-up from a script/service and deploying web applications to it? Did you upgrade from 8.5.34 and then it broke? How did you perform the upgrade? - -chris -BEGIN PGP SIGNATURE- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlwGsBUACgkQHPApP6U8 pFg7RxAAvmaaNGfOSGut8l5xHktCpf7B8Uy4/nChr/gAeGuY6Vk0vKcB88cdecGU q70+atyU6r03sUWuIbfM9OQc7DhPS5mZHifCOM8Yn77ZK6pJERYg+tz4m/f7M979 Cj7QUIHl8/i21qULL6Y5OQyV/N5U9lAnKzGRJrVcDghcl+pYrEBE3P+VyOXduTvr rOjYxNq2NP+D0B4JTy1D+Wa7+BFsqrAwm3TjQ7vxCg3wW4OWwDUM7SKYCCDMCYHV kLHIgJFRSwoZIGhDI4HYGjssY1H1q4DCTJPduQi0RZI/IcvrHNjRff3S9HXnzs5n 86efvPHh8SUmp42dFwWzlXq6E7O9tzoCOTbAn+SEr3EQG8H2m2cQ8Y66zaoXZGMY hN+cPtMfimwCpI3bO46yEB6YipZbf4fjn6MXIbJk+GWEafuviEg0t/uIUGRslBpl CpzJb1lgZZUIdmn3xvdPsCzo1Ot5saC2lEJbzCnBjjR8d9RqlrGZKBn5L6h43zxV QlEK8YfF11YckXKSM/Kr8/AorYzmifcd6aAQJJtCPkQWE49YDzbCOIAqy6ooR7Br fCpfs1cUsemNn0nfUYWMN/4X/NDfrsFhdxLveI8/bjMCFfYCxyH0V5wYN69oZaWL N8Vt74TrJ4gqukQTwvq85xdgVdLj9zQgTtVci5+AamjDe6ydTjI= =Wtha -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
manager app, Complete Server Status, not shown list of NIO threads
Hello, I'm using Apache Tomcat/8.5.35, jvm 1.8.0_192-b12, Windows Server 2012 R2 and at Complete Server Status page I can see list of all http-nio threads and I can see a header of ajp-nio threads. But there is displayed only a label Max threads: and nothing more. In the localhost.log is an exception: 04-Dec-2018 10:31:38.109 SEVERE [96] org.apache.catalina.core.StandardHostValve.invoke Exception Processing null java.lang.NullPointerException at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:236) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:650) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:342) at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:479) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:806) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) At the version Apache Tomcat/8.5.34 this problem doesn't occur. Did I found a bug? Jan.
Re: StuckThreadDetectionValve with threads on IO
On 04/10/18 01:08, Behrooz Nobakht wrote: > [1] uses a time based threshold to mark a request thread as "stuck" above > the configured threshold. > > This is definitely useful, but maybe a thread is busy transferring a large > request (IO). This could become > a wrong conclusion for the thread to be marked as stuck. > > A relevant contextual example is that nginx has proxy_read_timeout [2] > (although Tomcat is not by definition a proxy) > The above checks if the server is transmitting any bytes over the wire > (socket read timeout) to decide if a request is stuck or not. > > I am wondering if there's a similar way to achieve the same in Tomcat > (possibly without using NIO connectors)? It should be possible with a custom Valve. It would need to be along similar lines to the StuckThreadDetectionValve but record more information. Namely the response object and the last value of response.getBytesWritten(false). The idea is that on each cycle of the background process, you'd check the current value of response.getBytesWritten(false) and compare it to the last. HTH, Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
StuckThreadDetectionValve with threads on IO
[1] uses a time based threshold to mark a request thread as "stuck" above the configured threshold. This is definitely useful, but maybe a thread is busy transferring a large request (IO). This could become a wrong conclusion for the thread to be marked as stuck. A relevant contextual example is that nginx has proxy_read_timeout [2] (although Tomcat is not by definition a proxy) The above checks if the server is transmitting any bytes over the wire (socket read timeout) to decide if a request is stuck or not. I am wondering if there's a similar way to achieve the same in Tomcat (possibly without using NIO connectors)? Thanks, Behrooz [1]: https://github.com/apache/tomcat/blob/TOMCAT_9_0_12/java/org/apache/catalina/valves/StuckThreadDetectionValve.java [2]: http://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_read_timeout
Re: Servlet Threads Changing Instance Data
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jerry, On 8/15/18 4:00 PM, Jerry Malcolm wrote: > On 8/15/2018 1:50 PM, Olaf Kock wrote: >> Jerry, >> >> >> On 15.08.2018 18:14, Jerry Malcolm wrote: >>> I have a mobile app that issues several http web service calls >>> to initialize. I was making them sequentially with no issues. >>> I then changed to give them all separate threads so they could >>> load asynchronously. Then the bottom fell out. I started >>> getting empty responses and some responses with results of >>> three or four of the calls concatenated. I traced the problem >>> from the app back through apache through mod_jk and found the >>> culprit to be Tomcat. >>> >>> I'm a seasoned Tomcat developer for over 15 years. I've never >>> seen anything like this. But it's really scary. What I found >>> is that sometime during the execution of one servlet call, it's >>> thread data is swapped to thread data of another servlet call. >>> I know this sounds like Twilight Zone. But here is a log >>> output. At the beginning of doGet(), I generated a random text >>> string just to keep track of the thread data/:/ >>> >>> Thread: ajp-nio-8009-exec-24 uid: rclebgb --> Thread: >>> ajp-nio-8009-exec-29 uid: ceycfqd --> Thread: >>> ajp-nio-8009-exec-29 uid: ceycfqd <-- Thread: >>> ajp-nio-8009-exec-24 uid: ceycfqd <-- >>> >>> Note that when thread 24 starts I store the "rcl..." uid. >>> Another call comes in and starts thread 29. By the time thread >>> 24's servlet code is done, it now has thread 29's data. (The >>> uid is just a quick variable for reference. The request >>> object, response object, EVERYTHING is now thread 29's data). >>> >>> This explains why I'm getting empty responses and other >>> response with the data for multiple requests concatenated >>> together. The "rcl..." instance data has totally disappeared, >>> and all of the server calls are now using the "cey..." instance >>> data (i.e. response object). >>> >>> I figure this is some sort of timing/race condition that only >>> occurs with a second call coming in while the first call is >>> still in progress. I can go back to sending the mobile app >>> calls serially and probably work around this. But this is a >>> huge problem. >>> >>> Any ideas? (BTW... Tomcat 9.0.7) >>> >> >> As we don't know which code generates this log output, it's hard >> to judge what actually causes your problem. You say "thread data" >> is being swapped, and the very first aspect that comes to my mind >> is: Servlets are inherently multithreaded, and a common pattern >> of bugs is if a servlet has a member variable that is used for >> request processing: There typically is only one Servlet object >> ever, thus they all share the same state, if the state is stored >> in a member variable. This might be directly in the servlet or in >> some other component or singleton somewhere. >> >> Any state and request processing must be done on the >> request/response pair, and properly threadsafe in every other >> part of your code. >> >> And most likely this is an issue that luckily shows up when >> you're issuing a lot of parallel threads due to parallelizing one >> client. It'd be a lot harder to reproduce if it were individual >> users, who (very) occasionally see the wrong data. Consider >> yourself lucky to have such a nice and reproducible issue. >> >> Olaf > > Olaf, > > I'm having a bit of trouble feeling "lucky" that I have a problem > that is catastrophically blocking my development progress toward a > deadline. Uh hmm. > I'm not sure what you mean by typically there is only one servlet > object. There's one class. But a new instance is created on each > request, right? And all instance variables should be scoped to > that instance alone, right? Nope. I think that's where everything you're observing is coming from, then. > It's fundamental to Java that a private local variable can't > arbitrarily change to another value in the middle of executing a > method other than by the local code changing it itself, right? Since servlet instances are shared, so are all those private variable references. > Yet uid (and all the other variables including request and > response) changed.. somewhere between the beginning and ending of > doGet(). !! I'm not doing any singleton stuff. S
Re: Servlet Threads Changing Instance Data
Am 2018-08-15 um 22:00 schrieb Jerry Malcolm:
On 8/15/2018 1:50 PM, Olaf Kock wrote:
Jerry,
On 15.08.2018 18:14, Jerry Malcolm wrote:
I have a mobile app that issues several http web service calls to
initialize. I was making them sequentially with no issues. I then
changed to give them all separate threads so they could load
asynchronously. Then the bottom fell out. I started getting empty
responses and some responses with results of three or four of the
calls concatenated. I traced the problem from the app back through
apache through mod_jk and found the culprit to be Tomcat.
I'm a seasoned Tomcat developer for over 15 years. I've never seen
anything like this. But it's really scary. What I found is that
sometime during the execution of one servlet call, it's thread data
is swapped to thread data of another servlet call. I know this sounds
like Twilight Zone. But here is a log output. At the beginning of
doGet(), I generated a random text string just to keep track of the
thread data/:/
Thread: ajp-nio-8009-exec-24 uid: rclebgb -->
Thread: ajp-nio-8009-exec-29 uid: ceycfqd -->
Thread: ajp-nio-8009-exec-29 uid: ceycfqd <--
Thread: ajp-nio-8009-exec-24 uid: ceycfqd <--
Note that when thread 24 starts I store the "rcl..." uid. Another
call comes in and starts thread 29. By the time thread 24's servlet
code is done, it now has thread 29's data. (The uid is just a quick
variable for reference. The request object, response object,
EVERYTHING is now thread 29's data).
This explains why I'm getting empty responses and other response with
the data for multiple requests concatenated together. The "rcl..."
instance data has totally disappeared, and all of the server calls
are now using the "cey..." instance data (i.e. response object).
I figure this is some sort of timing/race condition that only occurs
with a second call coming in while the first call is still in
progress. I can go back to sending the mobile app calls serially and
probably work around this. But this is a huge problem.
Any ideas? (BTW... Tomcat 9.0.7)
As we don't know which code generates this log output, it's hard to
judge what actually causes your problem. You say "thread data" is
being swapped, and the very first aspect that comes to my mind is:
Servlets are inherently multithreaded, and a common pattern of bugs is
if a servlet has a member variable that is used for request
processing: There typically is only one Servlet object ever, thus they
all share the same state, if the state is stored in a member variable.
This might be directly in the servlet or in some other component or
singleton somewhere.
Any state and request processing must be done on the request/response
pair, and properly threadsafe in every other part of your code.
And most likely this is an issue that luckily shows up when you're
issuing a lot of parallel threads due to parallelizing one client.
It'd be a lot harder to reproduce if it were individual users, who
(very) occasionally see the wrong data. Consider yourself lucky to
have such a nice and reproducible issue.
Olaf
Olaf,
I'm having a bit of trouble feeling "lucky" that I have a problem that
is catastrophically blocking my development progress toward a deadline.
I'm not sure what you mean by typically there is only one servlet
object. There's one class. But a new instance is created on each
request, right? And all instance variables should be scoped to that
instance alone, right? It's fundamental to Java that a private local
variable can't arbitrarily change to another value in the middle of
executing a method other than by the local code changing it itself,
right? Yet uid (and all the other variables including request and
response) changed.. somewhere between the beginning and ending of
doGet(). !! I'm not doing any singleton stuff. Simply
request-->Servlet-->response.
My servlet code is very basic. It is not multithreaded.
public class MyServlet() {
private String uid; // < PRIVATE access!!
Oh hell, this is so wrong. The servlet instance exists only once in the
webapp classloader. No one is creating a new instace on each an every
request. You *cannot* share a variable like that, it is not threadsafe.
This is your problem. You have to fix that. You also *must* review the
rest of your code. Here is your exact problem:
https://stackoverflow.com/a/2184147/696632. Trust Bauke Scholz!
Michael
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Servlet Threads Changing Instance Data
> From: Jerry Malcolm [mailto:techst...@malcolms.com] > Subject: Re: Servlet Threads Changing Instance Data > I'm not sure what you mean by typically there is only one servlet > object. There's one class. But a new instance is created on each > request, right? No - there's only one instance of each defined servlet. All requests for a particular servlet share that instance. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. smime.p7s Description: S/MIME cryptographic signature
Re: Servlet Threads Changing Instance Data
On 8/15/2018 1:50 PM, Olaf Kock wrote:
Jerry,
On 15.08.2018 18:14, Jerry Malcolm wrote:
I have a mobile app that issues several http web service calls to
initialize. I was making them sequentially with no issues. I then
changed to give them all separate threads so they could load
asynchronously. Then the bottom fell out. I started getting empty
responses and some responses with results of three or four of the
calls concatenated. I traced the problem from the app back through
apache through mod_jk and found the culprit to be Tomcat.
I'm a seasoned Tomcat developer for over 15 years. I've never seen
anything like this. But it's really scary. What I found is that
sometime during the execution of one servlet call, it's thread data
is swapped to thread data of another servlet call. I know this sounds
like Twilight Zone. But here is a log output. At the beginning of
doGet(), I generated a random text string just to keep track of the
thread data/:/
Thread: ajp-nio-8009-exec-24 uid: rclebgb -->
Thread: ajp-nio-8009-exec-29 uid: ceycfqd -->
Thread: ajp-nio-8009-exec-29 uid: ceycfqd <--
Thread: ajp-nio-8009-exec-24 uid: ceycfqd <--
Note that when thread 24 starts I store the "rcl..." uid. Another
call comes in and starts thread 29. By the time thread 24's servlet
code is done, it now has thread 29's data. (The uid is just a quick
variable for reference. The request object, response object,
EVERYTHING is now thread 29's data).
This explains why I'm getting empty responses and other response with
the data for multiple requests concatenated together. The "rcl..."
instance data has totally disappeared, and all of the server calls
are now using the "cey..." instance data (i.e. response object).
I figure this is some sort of timing/race condition that only occurs
with a second call coming in while the first call is still in
progress. I can go back to sending the mobile app calls serially and
probably work around this. But this is a huge problem.
Any ideas? (BTW... Tomcat 9.0.7)
As we don't know which code generates this log output, it's hard to
judge what actually causes your problem. You say "thread data" is
being swapped, and the very first aspect that comes to my mind is:
Servlets are inherently multithreaded, and a common pattern of bugs is
if a servlet has a member variable that is used for request
processing: There typically is only one Servlet object ever, thus they
all share the same state, if the state is stored in a member variable.
This might be directly in the servlet or in some other component or
singleton somewhere.
Any state and request processing must be done on the request/response
pair, and properly threadsafe in every other part of your code.
And most likely this is an issue that luckily shows up when you're
issuing a lot of parallel threads due to parallelizing one client.
It'd be a lot harder to reproduce if it were individual users, who
(very) occasionally see the wrong data. Consider yourself lucky to
have such a nice and reproducible issue.
Olaf
Olaf,
I'm having a bit of trouble feeling "lucky" that I have a problem that
is catastrophically blocking my development progress toward a deadline.
I'm not sure what you mean by typically there is only one servlet
object. There's one class. But a new instance is created on each
request, right? And all instance variables should be scoped to that
instance alone, right? It's fundamental to Java that a private local
variable can't arbitrarily change to another value in the middle of
executing a method other than by the local code changing it itself,
right? Yet uid (and all the other variables including request and
response) changed.. somewhere between the beginning and ending of
doGet(). !! I'm not doing any singleton stuff. Simply
request-->Servlet-->response.
My servlet code is very basic. It is not multithreaded.
public class MyServlet() {
private String uid; // < PRIVATE access!!
doGet() {
uid = createUID();
System.out.println( "Thread: " + Thread.currentThread().getName() +
" uid: " + uid + " -->");
(no new threads, totally synchronous, simply get
a couple of vars from a JDBC call)
System.out.println( "Thread: " + Thread.currentThread().getName() +
" uid: " + uid + " <--");
}
That's it. I can think of nothing I could be doing that would cause the
PRIVATE uid varible to be different on the 2nd println from the 1st
println. And furthermore, the 2nd println has a uid from a DIFFERENT
request.
What could I possibly be doing in this code above that could cause this?
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Servlet Threads Changing Instance Data
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Jerry, On 8/15/18 12:14 PM, Jerry Malcolm wrote: > I have a mobile app that issues several http web service calls to > initialize. I was making them sequentially with no issues. I > then changed to give them all separate threads so they could load > asynchronously. Then the bottom fell out. I started getting > empty responses and some responses with results of three or four of > the calls concatenated. I traced the problem from the app back > through apache through mod_jk and found the culprit to be Tomcat. > > I'm a seasoned Tomcat developer for over 15 years. I've never > seen anything like this. But it's really scary. What I found is > that sometime during the execution of one servlet call, it's thread > data is swapped to thread data of another servlet call. I know > this sounds like Twilight Zone. But here is a log output. At the > beginning of doGet(), I generated a random text string just to keep > track of the thread data/:/ > > Thread: ajp-nio-8009-exec-24 uid: rclebgb --> Thread: > ajp-nio-8009-exec-29 uid: ceycfqd --> Thread: ajp-nio-8009-exec-29 > uid: ceycfqd <-- Thread: ajp-nio-8009-exec-24 uid: ceycfqd <-- > > Note that when thread 24 starts I store the "rcl..." uid. Another > call comes in and starts thread 29. By the time thread 24's > servlet code is done, it now has thread 29's data. (The uid is > just a quick variable for reference. The request object, response > object, EVERYTHING is now thread 29's data). > > This explains why I'm getting empty responses and other response > with the data for multiple requests concatenated together. The > "rcl..." instance data has totally disappeared, and all of the > server calls are now using the "cey..." instance data (i.e. > response object). > > I figure this is some sort of timing/race condition that only > occurs with a second call coming in while the first call is still > in progress. I can go back to sending the mobile app calls serially > and probably work around this. But this is a huge problem. > > Any ideas? (BTW... Tomcat 9.0.7) Some more specific questions thatn what Olaf has already asked: 1. Where is your "uid" being stored? 2. What does your code look like? Can you just post some of it? We really don't care what your application is doing, of course. Just the parts that generate+log the "uids" you are showing to us. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlt0g7gACgkQHPApP6U8 pFjxcQ/9GdFR302AfkCzCW2WDLqOYlcV/WqGHv2qIq3G5Eyv6b8fRHlqncg6QGbv JZOMT1DHvfWYnMZ/XOcVbF1AdRfpCx4TUPuuaw4YoeS4kqghGKM3L545+Uavm5Qi 8I1GR5NL87fpFExifWrjqVgDIWGXmsHHG0TPwyWq16ALJ8IQ0jStbGdft82AEZeF jUkbt2/+aDvKsGJhqROQ/Z+QnA+F9u6NW2vZTP2zhb2S9Max1L/MHySx3WLZv4aN 72sS592+KquSMOw1068Wj+BdsIN6ag6HOaNc/o3WCRivcLd/bQbZ4+v8UoegruRW DK57voXd28lMBZI0SOU4sCLJpRctdprA8+zIsEiHKx4ln6UD+SHg3C9MGHhED5vp zRI6dFTcuzoQ3f6/LTp3zNXqBsxTWPyF7C82CphRP0BOfQT526a4D0GkAO4BpdAU Xm9tEpy1jQEyLplOWRctrSdy/e1uaqlJ3F+3/dkZEn+zfV9hpnMQ76BS8Mrei+Ke aH8K5f6Ahak4iaey69nUKe8vVhGHex3uq8oo94Ud+mMVlSeWNFvcY8Kb7ByjAcUb z3SsqTsGRHcmdJGg3Ku3FNpLEi9HkdZaAhGxENBg1lOd0oKHOEE/fsvnB83tartc Aqjan2IdjHtil5vYRiQQqqmZ372KLIpxTfle1cj0PoJgd/P52DI= =tZjO -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Servlet Threads Changing Instance Data
Jerry, On 15.08.2018 18:14, Jerry Malcolm wrote: I have a mobile app that issues several http web service calls to initialize. I was making them sequentially with no issues. I then changed to give them all separate threads so they could load asynchronously. Then the bottom fell out. I started getting empty responses and some responses with results of three or four of the calls concatenated. I traced the problem from the app back through apache through mod_jk and found the culprit to be Tomcat. I'm a seasoned Tomcat developer for over 15 years. I've never seen anything like this. But it's really scary. What I found is that sometime during the execution of one servlet call, it's thread data is swapped to thread data of another servlet call. I know this sounds like Twilight Zone. But here is a log output. At the beginning of doGet(), I generated a random text string just to keep track of the thread data/:/ Thread: ajp-nio-8009-exec-24 uid: rclebgb --> Thread: ajp-nio-8009-exec-29 uid: ceycfqd --> Thread: ajp-nio-8009-exec-29 uid: ceycfqd <-- Thread: ajp-nio-8009-exec-24 uid: ceycfqd <-- Note that when thread 24 starts I store the "rcl..." uid. Another call comes in and starts thread 29. By the time thread 24's servlet code is done, it now has thread 29's data. (The uid is just a quick variable for reference. The request object, response object, EVERYTHING is now thread 29's data). This explains why I'm getting empty responses and other response with the data for multiple requests concatenated together. The "rcl..." instance data has totally disappeared, and all of the server calls are now using the "cey..." instance data (i.e. response object). I figure this is some sort of timing/race condition that only occurs with a second call coming in while the first call is still in progress. I can go back to sending the mobile app calls serially and probably work around this. But this is a huge problem. Any ideas? (BTW... Tomcat 9.0.7) As we don't know which code generates this log output, it's hard to judge what actually causes your problem. You say "thread data" is being swapped, and the very first aspect that comes to my mind is: Servlets are inherently multithreaded, and a common pattern of bugs is if a servlet has a member variable that is used for request processing: There typically is only one Servlet object ever, thus they all share the same state, if the state is stored in a member variable. This might be directly in the servlet or in some other component or singleton somewhere. Any state and request processing must be done on the request/response pair, and properly threadsafe in every other part of your code. And most likely this is an issue that luckily shows up when you're issuing a lot of parallel threads due to parallelizing one client. It'd be a lot harder to reproduce if it were individual users, who (very) occasionally see the wrong data. Consider yourself lucky to have such a nice and reproducible issue. Olaf - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Servlet Threads Changing Instance Data
I have a mobile app that issues several http web service calls to initialize. I was making them sequentially with no issues. I then changed to give them all separate threads so they could load asynchronously. Then the bottom fell out. I started getting empty responses and some responses with results of three or four of the calls concatenated. I traced the problem from the app back through apache through mod_jk and found the culprit to be Tomcat. I'm a seasoned Tomcat developer for over 15 years. I've never seen anything like this. But it's really scary. What I found is that sometime during the execution of one servlet call, it's thread data is swapped to thread data of another servlet call. I know this sounds like Twilight Zone. But here is a log output. At the beginning of doGet(), I generated a random text string just to keep track of the thread data/:/ Thread: ajp-nio-8009-exec-24 uid: rclebgb --> Thread: ajp-nio-8009-exec-29 uid: ceycfqd --> Thread: ajp-nio-8009-exec-29 uid: ceycfqd <-- Thread: ajp-nio-8009-exec-24 uid: ceycfqd <-- Note that when thread 24 starts I store the "rcl..." uid. Another call comes in and starts thread 29. By the time thread 24's servlet code is done, it now has thread 29's data. (The uid is just a quick variable for reference. The request object, response object, EVERYTHING is now thread 29's data). This explains why I'm getting empty responses and other response with the data for multiple requests concatenated together. The "rcl..." instance data has totally disappeared, and all of the server calls are now using the "cey..." instance data (i.e. response object). I figure this is some sort of timing/race condition that only occurs with a second call coming in while the first call is still in progress. I can go back to sending the mobile app calls serially and probably work around this. But this is a huge problem. Any ideas? (BTW... Tomcat 9.0.7) Jerry/ / - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Which connector defines "http-nio-auto-1-exec-*" threads?
Thanks for reply. As a matter of fact, right after sending my question, I realized that a bug in the installation code ended up adding another connector with port number zero which wasn't in the original server.xml. Problem resolved. /Jong >>> Rémy Maucherat <r...@apache.org> 11/22/2017 10:15 AM >>> On Wed, Nov 22, 2017 at 3:24 PM, Jong Kim <jong@microfocus.com> wrote: > Tomcat version: 8.0.47 > OS: SUSE Linux Enterprise Server 12 (SP3) > > I have a Tomcat installation where server.xml defines two connectors - > NIO2 connector on port 8443 and AJP connector on port 8009 - The two > connector definitions are shown below. > > keystoreFile="/mycerts/keystore" keystorePass="xxx" maxThreads="150" > port="8443" protocol="org.apache.coyote.http11.Http11Nio2Protocol" > scheme="https" secure="true" sslProtocol="TLS" acceptCount="0" > sslEnabledProtocols="TLSv1.2" ciphers="HIGH:!3DES:!EXP:!aNULL:!MD5"/> > > protocol="AJP/1.3" redirectPort="8443" tomcatAuthentication="false"/> > > We disable non-secure http access on port 8080 by removing the connector > definition. > > However, when we start and run this Tomcat, it creates three protocol > handler rather than expected two (as shown below). > > 17-Nov-2017 19:13:21.790 INFO [main] org.apache.coyote.AbstractProtocol.start > Starting ProtocolHandler ["http-nio-auto-1-46276"] > 17-Nov-2017 19:13:21.796 INFO [main] org.apache.coyote.AbstractProtocol.start > Starting ProtocolHandler ["http-nio2-8443"] > 17-Nov-2017 19:13:21.797 INFO [main] org.apache.coyote.AbstractProtocol.start > Starting ProtocolHandler ["ajp-nio-8009"] > > And then, when I look at the connector thread pool, it creates a set of > threads with this name pattern - "http-nio-auto-1-exec-" - in > addition to the two thread pools used by NIO2 and AJP connectors. > > So, the question - What is this so-called "auto" connector for? And why is > Tomcat creating it when I did NOT configure the system to create a > non-secure http connector? Also, the listening port number used by this > "auto" connector seems random (46276 in this case) and changes from run to > run, so how could it be ever used for anything? > > Thanks in advance for help > The auto port is used for a connector that doesn't specify a port. I am not aware of automatic connector creation except if using embedded, but that doesn't look to be your case here with that server.xml fragment. Any other information ? Rémy > /Jong > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Which connector defines "http-nio-auto-1-exec-*" threads?
On Wed, Nov 22, 2017 at 3:24 PM, Jong Kim <jong@microfocus.com> wrote: > Tomcat version: 8.0.47 > OS: SUSE Linux Enterprise Server 12 (SP3) > > I have a Tomcat installation where server.xml defines two connectors - > NIO2 connector on port 8443 and AJP connector on port 8009 - The two > connector definitions are shown below. > > keystoreFile="/mycerts/keystore" keystorePass="xxx" maxThreads="150" > port="8443" protocol="org.apache.coyote.http11.Http11Nio2Protocol" > scheme="https" secure="true" sslProtocol="TLS" acceptCount="0" > sslEnabledProtocols="TLSv1.2" ciphers="HIGH:!3DES:!EXP:!aNULL:!MD5"/> > > protocol="AJP/1.3" redirectPort="8443" tomcatAuthentication="false"/> > > We disable non-secure http access on port 8080 by removing the connector > definition. > > However, when we start and run this Tomcat, it creates three protocol > handler rather than expected two (as shown below). > > 17-Nov-2017 19:13:21.790 INFO [main] org.apache.coyote.AbstractProtocol.start > Starting ProtocolHandler ["http-nio-auto-1-46276"] > 17-Nov-2017 19:13:21.796 INFO [main] org.apache.coyote.AbstractProtocol.start > Starting ProtocolHandler ["http-nio2-8443"] > 17-Nov-2017 19:13:21.797 INFO [main] org.apache.coyote.AbstractProtocol.start > Starting ProtocolHandler ["ajp-nio-8009"] > > And then, when I look at the connector thread pool, it creates a set of > threads with this name pattern - "http-nio-auto-1-exec-" - in > addition to the two thread pools used by NIO2 and AJP connectors. > > So, the question - What is this so-called "auto" connector for? And why is > Tomcat creating it when I did NOT configure the system to create a > non-secure http connector? Also, the listening port number used by this > "auto" connector seems random (46276 in this case) and changes from run to > run, so how could it be ever used for anything? > > Thanks in advance for help > The auto port is used for a connector that doesn't specify a port. I am not aware of automatic connector creation except if using embedded, but that doesn't look to be your case here with that server.xml fragment. Any other information ? Rémy > /Jong > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org >
Which connector defines "http-nio-auto-1-exec-*" threads?
Tomcat version: 8.0.47 OS: SUSE Linux Enterprise Server 12 (SP3) I have a Tomcat installation where server.xml defines two connectors - NIO2 connector on port 8443 and AJP connector on port 8009 - The two connector definitions are shown below. We disable non-secure http access on port 8080 by removing the connector definition. However, when we start and run this Tomcat, it creates three protocol handler rather than expected two (as shown below). 17-Nov-2017 19:13:21.790 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-auto-1-46276"] 17-Nov-2017 19:13:21.796 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio2-8443"] 17-Nov-2017 19:13:21.797 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"] And then, when I look at the connector thread pool, it creates a set of threads with this name pattern - "http-nio-auto-1-exec-" - in addition to the two thread pools used by NIO2 and AJP connectors. So, the question - What is this so-called "auto" connector for? And why is Tomcat creating it when I did NOT configure the system to create a non-secure http connector? Also, the listening port number used by this "auto" connector seems random (46276 in this case) and changes from run to run, so how could it be ever used for anything? Thanks in advance for help /Jong - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Per EndPoint Threads???
Owen Rubel oru...@gmail.com On Tue, Aug 15, 2017 at 8:23 AM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Owen, > > On 8/13/17 10:46 AM, Owen Rubel wrote: > > Owen Rubel oru...@gmail.com > > > > On Sun, Aug 13, 2017 at 5:57 AM, Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > > Owen, > > > > On 8/12/17 12:47 PM, Owen Rubel wrote: > >>>> What I am talking about is something that improves > >>>> communication as we notice that communication channel needing > >>>> more resources. Not caching what is communicated... improving > >>>> the CHANNEL for communicating the resource (whatever it may > >>>> be). > > > > If the channel is an HTTP connection (or TCP; the application > > protocol isn't terribly relevant), then you are limited by the > > following: > > > > 1. Network bandwidth 2. Available threads (to service a particular > > request) 3. Hardware resources on the server > > (CPU/memory/disk/etc.) > > > > Let's ignore 1 and 3 for now, since you are primarily concerned > > with concurrency, and concurrency is useless if the other resources > > are constrained or otherwise limiting the equation. > > > > Let's say we had "per endpoint" thread pools, so that e.g. /create > > had its own thread pool, and /show had another one, etc. What would > > that buy us? > > > > (Let's ignore for now the fact that one set of threads must always > > be used to decode the request to decide where it's going, like > > /create or /show.) > > > > If we have a limited total number of threads (e.g. 10), then we > > could "reserve" some of them so that we could always have 2 threads > > for /create even if all the other threads in the system (the other > > 8) were being used for something else. If we had 2 threads for > > /create and 2 threads for /show, then only 6 would remain for e.g. > > /edit or /delete. So if 6 threads were already being used for /edit > > or /delete, the 7th incoming request would be queued, but anyone > > making a request for /show or /create would (if a thread in those > > pools is available) be serviced immediately. > > > > I can see some utility in this ability, because it would allow the > > container to ensure that some resources were never starved... or, > > rather, that they have some priority over certain other services. > > In other words, the service could enjoy guaranteed provisioning > > for certain endpoints. > > > > As it stands, Tomcat (and, I would venture a guess, most if not > > all other containers) implements a fair request pipeline where > > requests are (at least roughly) serviced in the order in which they > > are received. Rather than guaranteeing provisioning for a > > particular endpoint, the closest thing that could be implemented > > (at the application level) would be a > > resource-availability-limiting mechanism, such as counting the > > number of in-flight requests and rejecting those which exceed some > > threshold with e.g. a 503 response. > > > > Unfortunately, that doesn't actually prioritize some requests, it > > merely rejects others in order to attempt to prioritize those > > others. It also starves endpoints even when there is no reason to > > do so (e.g. in the 10-thread scenario, if all 4 /show and /create > > threads are idle, but 6 requests are already in process for the > > other endpoints, a 7th request for those other endpoints will be > > rejected). > > > > I believe that per-endpoint provisioning is a possibility, but I > > don't think that the potential gains are worth the certain > > complexity of the system required to implement it. > > > > There are other ways to handle heterogeneous service requests in a > > way that doesn't starve one type of request in favor of another. > > One obvious solution is horizontal scaling with a load-balancer. An > > LB can be used to implement a sort of guaranteed-provisioning for > > certain endpoints by providing more back-end servers for certain > > endpoints. If you want to make sure that /show can be called by any > > client at any time, then make sure you spin-up 1000 /show servers > > and register them with the load-balancer. You can survive with only > > maybe 10 nodes servicing /delete requests; others will either wait > > in a queue or receive a 503 from the lb. > > > > For my money, I'd maximize the number of threads available
Re: Per EndPoint Threads???
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Owen, On 8/13/17 10:46 AM, Owen Rubel wrote: > Owen Rubel oru...@gmail.com > > On Sun, Aug 13, 2017 at 5:57 AM, Christopher Schultz < > ch...@christopherschultz.net> wrote: > > Owen, > > On 8/12/17 12:47 PM, Owen Rubel wrote: >>>> What I am talking about is something that improves >>>> communication as we notice that communication channel needing >>>> more resources. Not caching what is communicated... improving >>>> the CHANNEL for communicating the resource (whatever it may >>>> be). > > If the channel is an HTTP connection (or TCP; the application > protocol isn't terribly relevant), then you are limited by the > following: > > 1. Network bandwidth 2. Available threads (to service a particular > request) 3. Hardware resources on the server > (CPU/memory/disk/etc.) > > Let's ignore 1 and 3 for now, since you are primarily concerned > with concurrency, and concurrency is useless if the other resources > are constrained or otherwise limiting the equation. > > Let's say we had "per endpoint" thread pools, so that e.g. /create > had its own thread pool, and /show had another one, etc. What would > that buy us? > > (Let's ignore for now the fact that one set of threads must always > be used to decode the request to decide where it's going, like > /create or /show.) > > If we have a limited total number of threads (e.g. 10), then we > could "reserve" some of them so that we could always have 2 threads > for /create even if all the other threads in the system (the other > 8) were being used for something else. If we had 2 threads for > /create and 2 threads for /show, then only 6 would remain for e.g. > /edit or /delete. So if 6 threads were already being used for /edit > or /delete, the 7th incoming request would be queued, but anyone > making a request for /show or /create would (if a thread in those > pools is available) be serviced immediately. > > I can see some utility in this ability, because it would allow the > container to ensure that some resources were never starved... or, > rather, that they have some priority over certain other services. > In other words, the service could enjoy guaranteed provisioning > for certain endpoints. > > As it stands, Tomcat (and, I would venture a guess, most if not > all other containers) implements a fair request pipeline where > requests are (at least roughly) serviced in the order in which they > are received. Rather than guaranteeing provisioning for a > particular endpoint, the closest thing that could be implemented > (at the application level) would be a > resource-availability-limiting mechanism, such as counting the > number of in-flight requests and rejecting those which exceed some > threshold with e.g. a 503 response. > > Unfortunately, that doesn't actually prioritize some requests, it > merely rejects others in order to attempt to prioritize those > others. It also starves endpoints even when there is no reason to > do so (e.g. in the 10-thread scenario, if all 4 /show and /create > threads are idle, but 6 requests are already in process for the > other endpoints, a 7th request for those other endpoints will be > rejected). > > I believe that per-endpoint provisioning is a possibility, but I > don't think that the potential gains are worth the certain > complexity of the system required to implement it. > > There are other ways to handle heterogeneous service requests in a > way that doesn't starve one type of request in favor of another. > One obvious solution is horizontal scaling with a load-balancer. An > LB can be used to implement a sort of guaranteed-provisioning for > certain endpoints by providing more back-end servers for certain > endpoints. If you want to make sure that /show can be called by any > client at any time, then make sure you spin-up 1000 /show servers > and register them with the load-balancer. You can survive with only > maybe 10 nodes servicing /delete requests; others will either wait > in a queue or receive a 503 from the lb. > > For my money, I'd maximize the number of threads available for all > requests (whether within a single server, or across a large > cluster) and not require that they be available for any particular > endpoint. Once you have to depart from a single server, you MUST > have something like a load-balancer involved, and therefore the > above solution becomes not only more practical but also more > powerful. > > Since relying on a one-box-wonder to run a high-availability web > service isn't practical, provisioning is necessarily above the > cluster-node level, and so the problem has
Re: Per EndPoint Threads???
Owen Rubel oru...@gmail.com On Sun, Aug 13, 2017 at 5:57 AM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Owen, > > On 8/12/17 12:47 PM, Owen Rubel wrote: > > What I am talking about is something that improves communication as > > we notice that communication channel needing more resources. Not > > caching what is communicated... improving the CHANNEL for > > communicating the resource (whatever it may be). > > If the channel is an HTTP connection (or TCP; the application protocol > isn't terribly relevant), then you are limited by the following: > > 1. Network bandwidth > 2. Available threads (to service a particular request) > 3. Hardware resources on the server (CPU/memory/disk/etc.) > > Let's ignore 1 and 3 for now, since you are primarily concerned with > concurrency, and concurrency is useless if the other resources are > constrained or otherwise limiting the equation. > > Let's say we had "per endpoint" thread pools, so that e.g. /create had > its own thread pool, and /show had another one, etc. What would that > buy us? > > (Let's ignore for now the fact that one set of threads must always be > used to decode the request to decide where it's going, like /create or > /show.) > > If we have a limited total number of threads (e.g. 10), then we could > "reserve" some of them so that we could always have 2 threads for > /create even if all the other threads in the system (the other 8) were > being used for something else. If we had 2 threads for /create and 2 > threads for /show, then only 6 would remain for e.g. /edit or /delete. > So if 6 threads were already being used for /edit or /delete, the 7th > incoming request would be queued, but anyone making a request for > /show or /create would (if a thread in those pools is available) be > serviced immediately. > > I can see some utility in this ability, because it would allow the > container to ensure that some resources were never starved... or, > rather, that they have some priority over certain other services. In > other words, the service could enjoy guaranteed provisioning for > certain endpoints. > > As it stands, Tomcat (and, I would venture a guess, most if not all > other containers) implements a fair request pipeline where requests > are (at least roughly) serviced in the order in which they are > received. Rather than guaranteeing provisioning for a particular > endpoint, the closest thing that could be implemented (at the > application level) would be a resource-availability-limiting > mechanism, such as counting the number of in-flight requests and > rejecting those which exceed some threshold with e.g. a 503 response. > > Unfortunately, that doesn't actually prioritize some requests, it > merely rejects others in order to attempt to prioritize those others. > It also starves endpoints even when there is no reason to do so (e.g. > in the 10-thread scenario, if all 4 /show and /create threads are > idle, but 6 requests are already in process for the other endpoints, a > 7th request for those other endpoints will be rejected). > > I believe that per-endpoint provisioning is a possibility, but I don't > think that the potential gains are worth the certain complexity of the > system required to implement it. > > There are other ways to handle heterogeneous service requests in a way > that doesn't starve one type of request in favor of another. One > obvious solution is horizontal scaling with a load-balancer. An LB can > be used to implement a sort of guaranteed-provisioning for certain > endpoints by providing more back-end servers for certain endpoints. If > you want to make sure that /show can be called by any client at any > time, then make sure you spin-up 1000 /show servers and register them > with the load-balancer. You can survive with only maybe 10 nodes > servicing /delete requests; others will either wait in a queue or > receive a 503 from the lb. > > For my money, I'd maximize the number of threads available for all > requests (whether within a single server, or across a large cluster) > and not require that they be available for any particular endpoint. > Once you have to depart from a single server, you MUST have something > like a load-balancer involved, and therefore the above solution > becomes not only more practical but also more powerful. > > Since relying on a one-box-wonder to run a high-availability web > service isn't practical, provisioning is necessarily above the > cluster-node level, and so the problem has effectively moved from the > app server to the load-balancer (or reverse proxy). I believe the > application server is an inapprop
Re: Per EndPoint Threads???
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Owen, On 8/12/17 12:47 PM, Owen Rubel wrote: > What I am talking about is something that improves communication as > we notice that communication channel needing more resources. Not > caching what is communicated... improving the CHANNEL for > communicating the resource (whatever it may be). If the channel is an HTTP connection (or TCP; the application protocol isn't terribly relevant), then you are limited by the following: 1. Network bandwidth 2. Available threads (to service a particular request) 3. Hardware resources on the server (CPU/memory/disk/etc.) Let's ignore 1 and 3 for now, since you are primarily concerned with concurrency, and concurrency is useless if the other resources are constrained or otherwise limiting the equation. Let's say we had "per endpoint" thread pools, so that e.g. /create had its own thread pool, and /show had another one, etc. What would that buy us? (Let's ignore for now the fact that one set of threads must always be used to decode the request to decide where it's going, like /create or /show.) If we have a limited total number of threads (e.g. 10), then we could "reserve" some of them so that we could always have 2 threads for /create even if all the other threads in the system (the other 8) were being used for something else. If we had 2 threads for /create and 2 threads for /show, then only 6 would remain for e.g. /edit or /delete. So if 6 threads were already being used for /edit or /delete, the 7th incoming request would be queued, but anyone making a request for /show or /create would (if a thread in those pools is available) be serviced immediately. I can see some utility in this ability, because it would allow the container to ensure that some resources were never starved... or, rather, that they have some priority over certain other services. In other words, the service could enjoy guaranteed provisioning for certain endpoints. As it stands, Tomcat (and, I would venture a guess, most if not all other containers) implements a fair request pipeline where requests are (at least roughly) serviced in the order in which they are received. Rather than guaranteeing provisioning for a particular endpoint, the closest thing that could be implemented (at the application level) would be a resource-availability-limiting mechanism, such as counting the number of in-flight requests and rejecting those which exceed some threshold with e.g. a 503 response. Unfortunately, that doesn't actually prioritize some requests, it merely rejects others in order to attempt to prioritize those others. It also starves endpoints even when there is no reason to do so (e.g. in the 10-thread scenario, if all 4 /show and /create threads are idle, but 6 requests are already in process for the other endpoints, a 7th request for those other endpoints will be rejected). I believe that per-endpoint provisioning is a possibility, but I don't think that the potential gains are worth the certain complexity of the system required to implement it. There are other ways to handle heterogeneous service requests in a way that doesn't starve one type of request in favor of another. One obvious solution is horizontal scaling with a load-balancer. An LB can be used to implement a sort of guaranteed-provisioning for certain endpoints by providing more back-end servers for certain endpoints. If you want to make sure that /show can be called by any client at any time, then make sure you spin-up 1000 /show servers and register them with the load-balancer. You can survive with only maybe 10 nodes servicing /delete requests; others will either wait in a queue or receive a 503 from the lb. For my money, I'd maximize the number of threads available for all requests (whether within a single server, or across a large cluster) and not require that they be available for any particular endpoint. Once you have to depart from a single server, you MUST have something like a load-balancer involved, and therefore the above solution becomes not only more practical but also more powerful. Since relying on a one-box-wonder to run a high-availability web service isn't practical, provisioning is necessarily above the cluster-node level, and so the problem has effectively moved from the app server to the load-balancer (or reverse proxy). I believe the application server is an inappropriate place to implement this type of provisioning because it's too small-scale. The app server should serve requests as quickly as possible, and arranging for this kind of provisioning would add a level of complexity that would jeopardize performance of all requests within the application server. > But like you said, this is not something that is doable so I'll > look elsewhere. I think it's doable, just not worth it given the orthogonal solutions available. Some things are better-implemented at other layers of the application (as a whole system) and per
Re: Per EndPoint Threads???
On Sat, Aug 12, 2017 at 3:13 PM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Owen, > > On 8/12/17 12:47 PM, Owen Rubel wrote: > > On Sat, Aug 12, 2017 at 9:36 AM, Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > > Owen, > > > > On 8/12/17 11:21 AM, Owen Rubel wrote: > >>>> On Sat, Aug 12, 2017 at 1:19 AM, Mark Thomas > >>>> <ma...@apache.org> wrote: > >>>> > >>>>> On 12/08/17 06:00, Christopher Schultz wrote: > >>>>>> Owen, > >>>>>> > >>>>>> Please do not top-post. I have re-ordered your post to > >>>>>> be bottom-post. > >>>>>> > >>>>>> On 8/11/17 10:12 PM, Owen Rubel wrote: > >>>>>>> On Fri, Aug 11, 2017 at 5:58 PM, > >>>>>>> <christop...@baus.net> wrote: > >>>>>> > >>>>>>>>> Hi All, > >>>>>>>>> > >>>>>>>>> I'm looking for a way (or a tool) in Tomcat to > >>>>>>>>> associate threads with endpoints. > >>>>>>>> > >>>>>>>> It isn't clear to me why this would be necessary. > >>>>>>>> Threads should be allocated on demand to individual > >>>>>>>> requests. If one route sees more traffic, then it > >>>>>>>> should automatically be allocated more threads. This > >>>>>>>> could starve some requests if the maximum number of > >>>>>>>> threads had been allocated to a lessor used route, > >>>>>>>> while available threads went unused for more commonly > >>>>>>>> used route. > >>>>>> > >>>>>>> Absolutely but it could ramp up more threads as > >>>>>>> needed. > >>>>>> > >>>>>>> I base the logic on neuron and neuralTransmitters. > >>>>>>> When neurons talk to each other, they send back neural > >>>>>>> transmitters to enforce that pathway. > >>>>>> > >>>>>>> If we could do the same through threads by adding > >>>>>>> additional threads for endpoints that receive more > >>>>>>> traffic vs those which do not, it would enforce better > >>>>>>> and faster communication on those paths.> The current > >>>>>>> way Tomcat does it is not dynamic and it just applies > >>>>>>> to ALL pathways equally which is not efficient. > >>>>>> How would this improve efficiency at all? > >>>>>> > >>>>>> There is nothing inherently "showy" or "edity" about a > >>>>>> particular thread; each request-processing thread is > >>>>>> indistinguishable from any other. I don't believe there > >>>>>> is a way to improve the situation even if "per-endpoint" > >>>>>> (whatever that would mean) threads were a possibility. > >>>>>> > >>>>>> What would you attach to a thread that would make it any > >>>>>> better at editing records? Or deleting them? > >>>>> > >>>>> And I'll add that the whole original proposal ignores a > >>>>> number of rather fundamental points about how Servlet > >>>>> containers (and web servers in general) work. To name a > >>>>> few: > >>>>> > >>>>> - Until the request has been parsed (which requires a > >>>>> thread) Tomcat doesn't know which Servlet (endpoint) the > >>>>> request is destined for. Switching processing to a > >>>>> different thread at that point would add significant > >>>>> overhead for no benefit. > >>>>> > >>>>> - Even after parsing, the actual Servlet that processes > >>>>> the request (if any) can change during processing (e.g. a > >>>>> Filter that conditionally forwards to a different Servlet, > >>>>> authentication, etc.) > >>>>> > >>>>> There is nothing about a endpoint specific thread that > >>>>> would allow it to process a req
Re: Per EndPoint Threads???
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Owen, On 8/12/17 12:47 PM, Owen Rubel wrote: > On Sat, Aug 12, 2017 at 9:36 AM, Christopher Schultz < > ch...@christopherschultz.net> wrote: > > Owen, > > On 8/12/17 11:21 AM, Owen Rubel wrote: >>>> On Sat, Aug 12, 2017 at 1:19 AM, Mark Thomas >>>> <ma...@apache.org> wrote: >>>> >>>>> On 12/08/17 06:00, Christopher Schultz wrote: >>>>>> Owen, >>>>>> >>>>>> Please do not top-post. I have re-ordered your post to >>>>>> be bottom-post. >>>>>> >>>>>> On 8/11/17 10:12 PM, Owen Rubel wrote: >>>>>>> On Fri, Aug 11, 2017 at 5:58 PM, >>>>>>> <christop...@baus.net> wrote: >>>>>> >>>>>>>>> Hi All, >>>>>>>>> >>>>>>>>> I'm looking for a way (or a tool) in Tomcat to >>>>>>>>> associate threads with endpoints. >>>>>>>> >>>>>>>> It isn't clear to me why this would be necessary. >>>>>>>> Threads should be allocated on demand to individual >>>>>>>> requests. If one route sees more traffic, then it >>>>>>>> should automatically be allocated more threads. This >>>>>>>> could starve some requests if the maximum number of >>>>>>>> threads had been allocated to a lessor used route, >>>>>>>> while available threads went unused for more commonly >>>>>>>> used route. >>>>>> >>>>>>> Absolutely but it could ramp up more threads as >>>>>>> needed. >>>>>> >>>>>>> I base the logic on neuron and neuralTransmitters. >>>>>>> When neurons talk to each other, they send back neural >>>>>>> transmitters to enforce that pathway. >>>>>> >>>>>>> If we could do the same through threads by adding >>>>>>> additional threads for endpoints that receive more >>>>>>> traffic vs those which do not, it would enforce better >>>>>>> and faster communication on those paths.> The current >>>>>>> way Tomcat does it is not dynamic and it just applies >>>>>>> to ALL pathways equally which is not efficient. >>>>>> How would this improve efficiency at all? >>>>>> >>>>>> There is nothing inherently "showy" or "edity" about a >>>>>> particular thread; each request-processing thread is >>>>>> indistinguishable from any other. I don't believe there >>>>>> is a way to improve the situation even if "per-endpoint" >>>>>> (whatever that would mean) threads were a possibility. >>>>>> >>>>>> What would you attach to a thread that would make it any >>>>>> better at editing records? Or deleting them? >>>>> >>>>> And I'll add that the whole original proposal ignores a >>>>> number of rather fundamental points about how Servlet >>>>> containers (and web servers in general) work. To name a >>>>> few: >>>>> >>>>> - Until the request has been parsed (which requires a >>>>> thread) Tomcat doesn't know which Servlet (endpoint) the >>>>> request is destined for. Switching processing to a >>>>> different thread at that point would add significant >>>>> overhead for no benefit. >>>>> >>>>> - Even after parsing, the actual Servlet that processes >>>>> the request (if any) can change during processing (e.g. a >>>>> Filter that conditionally forwards to a different Servlet, >>>>> authentication, etc.) >>>>> >>>>> There is nothing about a endpoint specific thread that >>>>> would allow it to process a request more efficiently than a >>>>> general thread. >>>>> >>>>> Any per-endpoint thread-pool solution will require the >>>>> additional overhead to switch processing from the general >>>>> parsing thread to the endpoint specific thread. This >>>>> additional cost comes with zero benefits hence it will >>>>> always be less efficient. >>>>&g
Re: Per EndPoint Threads???
On Sat, Aug 12, 2017 at 9:36 AM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Owen, > > On 8/12/17 11:21 AM, Owen Rubel wrote: > > On Sat, Aug 12, 2017 at 1:19 AM, Mark Thomas <ma...@apache.org> > > wrote: > > > >> On 12/08/17 06:00, Christopher Schultz wrote: > >>> Owen, > >>> > >>> Please do not top-post. I have re-ordered your post to be > >>> bottom-post. > >>> > >>> On 8/11/17 10:12 PM, Owen Rubel wrote: > >>>> On Fri, Aug 11, 2017 at 5:58 PM, <christop...@baus.net> > >>>> wrote: > >>> > >>>>>> Hi All, > >>>>>> > >>>>>> I'm looking for a way (or a tool) in Tomcat to associate > >>>>>> threads with endpoints. > >>>>> > >>>>> It isn't clear to me why this would be necessary. Threads > >>>>> should be allocated on demand to individual requests. If > >>>>> one route sees more traffic, then it should automatically > >>>>> be allocated more threads. This could starve some requests > >>>>> if the maximum number of threads had been allocated to a > >>>>> lessor used route, while available threads went unused for > >>>>> more commonly used route. > >>> > >>>> Absolutely but it could ramp up more threads as needed. > >>> > >>>> I base the logic on neuron and neuralTransmitters. When > >>>> neurons talk to each other, they send back neural > >>>> transmitters to enforce that pathway. > >>> > >>>> If we could do the same through threads by adding additional > >>>> threads for endpoints that receive more traffic vs those > >>>> which do not, it would enforce better and faster > >>>> communication on those paths.> The current way Tomcat does it > >>>> is not dynamic and it just applies to ALL pathways equally > >>>> which is not efficient. > >>> How would this improve efficiency at all? > >>> > >>> There is nothing inherently "showy" or "edity" about a > >>> particular thread; each request-processing thread is > >>> indistinguishable from any other. I don't believe there is a > >>> way to improve the situation even if "per-endpoint" (whatever > >>> that would mean) threads were a possibility. > >>> > >>> What would you attach to a thread that would make it any better > >>> at editing records? Or deleting them? > >> > >> And I'll add that the whole original proposal ignores a number of > >> rather fundamental points about how Servlet containers (and web > >> servers in general) work. To name a few: > >> > >> - Until the request has been parsed (which requires a thread) > >> Tomcat doesn't know which Servlet (endpoint) the request is > >> destined for. Switching processing to a different thread at that > >> point would add significant overhead for no benefit. > >> > >> - Even after parsing, the actual Servlet that processes the > >> request (if any) can change during processing (e.g. a Filter that > >> conditionally forwards to a different Servlet, authentication, > >> etc.) > >> > >> There is nothing about a endpoint specific thread that would > >> allow it to process a request more efficiently than a general > >> thread. > >> > >> Any per-endpoint thread-pool solution will require the > >> additional overhead to switch processing from the general parsing > >> thread to the endpoint specific thread. This additional cost > >> comes with zero benefits hence it will always be less efficient. > >> > >> In short, there is no way pre-allocating threads to particular > >> endpoints can improve performance compared to just adding the > >> same number of additional threads to the general thread pool. > > > Ah ok thank you for very concise answer. am chasing a pipe dream I > > guess. Maybe there is another way to get this kind of benefit. > The answer is caching, and that can be done at many levels, but the > thread-level makes the least sense due to the reasons Mark outlined abov > e. > > - -chris > -BEGIN PGP SIGNATURE- > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > i
Re: Per EndPoint Threads???
On Sat, Aug 12, 2017 at 9:36 AM, Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Owen, > > On 8/12/17 11:21 AM, Owen Rubel wrote: > > On Sat, Aug 12, 2017 at 1:19 AM, Mark Thomas <ma...@apache.org> > > wrote: > > > >> On 12/08/17 06:00, Christopher Schultz wrote: > >>> Owen, > >>> > >>> Please do not top-post. I have re-ordered your post to be > >>> bottom-post. > >>> > >>> On 8/11/17 10:12 PM, Owen Rubel wrote: > >>>> On Fri, Aug 11, 2017 at 5:58 PM, <christop...@baus.net> > >>>> wrote: > >>> > >>>>>> Hi All, > >>>>>> > >>>>>> I'm looking for a way (or a tool) in Tomcat to associate > >>>>>> threads with endpoints. > >>>>> > >>>>> It isn't clear to me why this would be necessary. Threads > >>>>> should be allocated on demand to individual requests. If > >>>>> one route sees more traffic, then it should automatically > >>>>> be allocated more threads. This could starve some requests > >>>>> if the maximum number of threads had been allocated to a > >>>>> lessor used route, while available threads went unused for > >>>>> more commonly used route. > >>> > >>>> Absolutely but it could ramp up more threads as needed. > >>> > >>>> I base the logic on neuron and neuralTransmitters. When > >>>> neurons talk to each other, they send back neural > >>>> transmitters to enforce that pathway. > >>> > >>>> If we could do the same through threads by adding additional > >>>> threads for endpoints that receive more traffic vs those > >>>> which do not, it would enforce better and faster > >>>> communication on those paths.> The current way Tomcat does it > >>>> is not dynamic and it just applies to ALL pathways equally > >>>> which is not efficient. > >>> How would this improve efficiency at all? > >>> > >>> There is nothing inherently "showy" or "edity" about a > >>> particular thread; each request-processing thread is > >>> indistinguishable from any other. I don't believe there is a > >>> way to improve the situation even if "per-endpoint" (whatever > >>> that would mean) threads were a possibility. > >>> > >>> What would you attach to a thread that would make it any better > >>> at editing records? Or deleting them? > >> > >> And I'll add that the whole original proposal ignores a number of > >> rather fundamental points about how Servlet containers (and web > >> servers in general) work. To name a few: > >> > >> - Until the request has been parsed (which requires a thread) > >> Tomcat doesn't know which Servlet (endpoint) the request is > >> destined for. Switching processing to a different thread at that > >> point would add significant overhead for no benefit. > >> > >> - Even after parsing, the actual Servlet that processes the > >> request (if any) can change during processing (e.g. a Filter that > >> conditionally forwards to a different Servlet, authentication, > >> etc.) > >> > >> There is nothing about a endpoint specific thread that would > >> allow it to process a request more efficiently than a general > >> thread. > >> > >> Any per-endpoint thread-pool solution will require the > >> additional overhead to switch processing from the general parsing > >> thread to the endpoint specific thread. This additional cost > >> comes with zero benefits hence it will always be less efficient. > >> > >> In short, there is no way pre-allocating threads to particular > >> endpoints can improve performance compared to just adding the > >> same number of additional threads to the general thread pool. > > > Ah ok thank you for very concise answer. am chasing a pipe dream I > > guess. Maybe there is another way to get this kind of benefit. > The answer is caching, and that can be done at many levels, but the > thread-level makes the least sense due to the reasons Mark outlined abov > e. > > - -chris > -BEGIN PGP SIGNATURE- > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ >
Re: Per EndPoint Threads???
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Owen, On 8/12/17 11:21 AM, Owen Rubel wrote: > On Sat, Aug 12, 2017 at 1:19 AM, Mark Thomas <ma...@apache.org> > wrote: > >> On 12/08/17 06:00, Christopher Schultz wrote: >>> Owen, >>> >>> Please do not top-post. I have re-ordered your post to be >>> bottom-post. >>> >>> On 8/11/17 10:12 PM, Owen Rubel wrote: >>>> On Fri, Aug 11, 2017 at 5:58 PM, <christop...@baus.net> >>>> wrote: >>> >>>>>> Hi All, >>>>>> >>>>>> I'm looking for a way (or a tool) in Tomcat to associate >>>>>> threads with endpoints. >>>>> >>>>> It isn't clear to me why this would be necessary. Threads >>>>> should be allocated on demand to individual requests. If >>>>> one route sees more traffic, then it should automatically >>>>> be allocated more threads. This could starve some requests >>>>> if the maximum number of threads had been allocated to a >>>>> lessor used route, while available threads went unused for >>>>> more commonly used route. >>> >>>> Absolutely but it could ramp up more threads as needed. >>> >>>> I base the logic on neuron and neuralTransmitters. When >>>> neurons talk to each other, they send back neural >>>> transmitters to enforce that pathway. >>> >>>> If we could do the same through threads by adding additional >>>> threads for endpoints that receive more traffic vs those >>>> which do not, it would enforce better and faster >>>> communication on those paths.> The current way Tomcat does it >>>> is not dynamic and it just applies to ALL pathways equally >>>> which is not efficient. >>> How would this improve efficiency at all? >>> >>> There is nothing inherently "showy" or "edity" about a >>> particular thread; each request-processing thread is >>> indistinguishable from any other. I don't believe there is a >>> way to improve the situation even if "per-endpoint" (whatever >>> that would mean) threads were a possibility. >>> >>> What would you attach to a thread that would make it any better >>> at editing records? Or deleting them? >> >> And I'll add that the whole original proposal ignores a number of >> rather fundamental points about how Servlet containers (and web >> servers in general) work. To name a few: >> >> - Until the request has been parsed (which requires a thread) >> Tomcat doesn't know which Servlet (endpoint) the request is >> destined for. Switching processing to a different thread at that >> point would add significant overhead for no benefit. >> >> - Even after parsing, the actual Servlet that processes the >> request (if any) can change during processing (e.g. a Filter that >> conditionally forwards to a different Servlet, authentication, >> etc.) >> >> There is nothing about a endpoint specific thread that would >> allow it to process a request more efficiently than a general >> thread. >> >> Any per-endpoint thread-pool solution will require the >> additional overhead to switch processing from the general parsing >> thread to the endpoint specific thread. This additional cost >> comes with zero benefits hence it will always be less efficient. >> >> In short, there is no way pre-allocating threads to particular >> endpoints can improve performance compared to just adding the >> same number of additional threads to the general thread pool. > Ah ok thank you for very concise answer. am chasing a pipe dream I > guess. Maybe there is another way to get this kind of benefit. The answer is caching, and that can be done at many levels, but the thread-level makes the least sense due to the reasons Mark outlined abov e. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlmPLngACgkQHPApP6U8 pFisbw//aiIg0vGmmlm4T/xoEbAKblKf6Qn9zmDzbLY9IbIG7MdsMcuV9hnsasEp iaZs3ROTy3BvWKGoyIGThtRsBPSFmb1H/XuKs4bqxgdRNgcbxEbjkH+1wZCx76Aq aqdIiCFdWvkOll4EqC4UYjNXCMkMBoTGN4GTxGmB8arujOyiC1KVPLY+wiRtXusF BrV3n9G+wN7Qq+rHIvgct1J29xTnPwQWhcdTrR5+IXn7vuNhEe9yxlKyJh4N6Pkt TW8ZlZfUgPnAXYZFvb0UfRK43cOCP4HsgncvIDjnnRJVTnaqRKBuRE4ZVYJG91SN CHUCYAmCR/rUZcOO3VJZ0dE7OEkrtcs6tmRT7j0qfS2qxbAb6YuW5xNYrCTgWKyD 6bUCQsKzcChV4mQPVDjXO/yv1t3dpXeMB+44KwCVB3bFPTediwISzTxInCSd/Kdu I+57Rcrclto8S3+GRsUPRG3dwsNMYMIxHpuzj/LYzLNdoANI8vM5NntYdQ4cwEFM H23i54m00WQ5RLuRJGzker+T5H0NvGlVwFQnqO9kCkA57o1Gi+vk34UuNPVLsqHx sKq6Eb4s3MeslZBPHhJWYXGPx226+T6sEXO1y2UZ9GuWYzfI3MF6/xcFOI2/W3id kYZEnR3R1Xes7GzsSLuCXVRDQco3GhXvSiyLvYC9xwgIsjnM61Q= =Q/vf -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Per EndPoint Threads???
Ah ok thank you for very concise answer. am chasing a pipe dream I guess. Maybe there is another way to get this kind of benefit. Thanks again for your answer. Owen Rubel oru...@gmail.com On Sat, Aug 12, 2017 at 1:19 AM, Mark Thomas <ma...@apache.org> wrote: > On 12/08/17 06:00, Christopher Schultz wrote: > > Owen, > > > > Please do not top-post. I have re-ordered your post to be bottom-post. > > > > On 8/11/17 10:12 PM, Owen Rubel wrote: > >> On Fri, Aug 11, 2017 at 5:58 PM, <christop...@baus.net> wrote: > > > >>>> Hi All, > >>>> > >>>> I'm looking for a way (or a tool) in Tomcat to associate > >>>> threads with endpoints. > >>> > >>> It isn't clear to me why this would be necessary. Threads should > >>> be allocated on demand to individual requests. If one route sees > >>> more traffic, then it should automatically be allocated more > >>> threads. This could starve some requests if the maximum number of > >>> threads had been allocated to a lessor used route, while > >>> available threads went unused for more commonly used route. > > > >> Absolutely but it could ramp up more threads as needed. > > > >> I base the logic on neuron and neuralTransmitters. When neurons > >> talk to each other, they send back neural transmitters to enforce > >> that pathway. > > > >> If we could do the same through threads by adding additional > >> threads for endpoints that receive more traffic vs those which do > >> not, it would enforce better and faster communication on those > >> paths.> The current way Tomcat does it is not dynamic and it just > >> applies to ALL pathways equally which is not efficient. > > How would this improve efficiency at all? > > > > There is nothing inherently "showy" or "edity" about a particular > > thread; each request-processing thread is indistinguishable from any > > other. I don't believe there is a way to improve the situation even if > > "per-endpoint" (whatever that would mean) threads were a possibility. > > > > What would you attach to a thread that would make it any better at > > editing records? Or deleting them? > > And I'll add that the whole original proposal ignores a number of rather > fundamental points about how Servlet containers (and web servers in > general) work. To name a few: > > - Until the request has been parsed (which requires a thread) Tomcat > doesn't know which Servlet (endpoint) the request is destined for. > Switching processing to a different thread at that point would add > significant overhead for no benefit. > > - Even after parsing, the actual Servlet that processes the request (if > any) can change during processing (e.g. a Filter that conditionally > forwards to a different Servlet, authentication, etc.) > > There is nothing about a endpoint specific thread that would allow it to > process a request more efficiently than a general thread. > > Any per-endpoint thread-pool solution will require the additional > overhead to switch processing from the general parsing thread to the > endpoint specific thread. This additional cost comes with zero benefits > hence it will always be less efficient. > > In short, there is no way pre-allocating threads to particular endpoints > can improve performance compared to just adding the same number of > additional threads to the general thread pool. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Per EndPoint Threads???
On 12/08/17 06:00, Christopher Schultz wrote: > Owen, > > Please do not top-post. I have re-ordered your post to be bottom-post. > > On 8/11/17 10:12 PM, Owen Rubel wrote: >> On Fri, Aug 11, 2017 at 5:58 PM, <christop...@baus.net> wrote: > >>>> Hi All, >>>> >>>> I'm looking for a way (or a tool) in Tomcat to associate >>>> threads with endpoints. >>> >>> It isn't clear to me why this would be necessary. Threads should >>> be allocated on demand to individual requests. If one route sees >>> more traffic, then it should automatically be allocated more >>> threads. This could starve some requests if the maximum number of >>> threads had been allocated to a lessor used route, while >>> available threads went unused for more commonly used route. > >> Absolutely but it could ramp up more threads as needed. > >> I base the logic on neuron and neuralTransmitters. When neurons >> talk to each other, they send back neural transmitters to enforce >> that pathway. > >> If we could do the same through threads by adding additional >> threads for endpoints that receive more traffic vs those which do >> not, it would enforce better and faster communication on those >> paths.> The current way Tomcat does it is not dynamic and it just >> applies to ALL pathways equally which is not efficient. > How would this improve efficiency at all? > > There is nothing inherently "showy" or "edity" about a particular > thread; each request-processing thread is indistinguishable from any > other. I don't believe there is a way to improve the situation even if > "per-endpoint" (whatever that would mean) threads were a possibility. > > What would you attach to a thread that would make it any better at > editing records? Or deleting them? And I'll add that the whole original proposal ignores a number of rather fundamental points about how Servlet containers (and web servers in general) work. To name a few: - Until the request has been parsed (which requires a thread) Tomcat doesn't know which Servlet (endpoint) the request is destined for. Switching processing to a different thread at that point would add significant overhead for no benefit. - Even after parsing, the actual Servlet that processes the request (if any) can change during processing (e.g. a Filter that conditionally forwards to a different Servlet, authentication, etc.) There is nothing about a endpoint specific thread that would allow it to process a request more efficiently than a general thread. Any per-endpoint thread-pool solution will require the additional overhead to switch processing from the general parsing thread to the endpoint specific thread. This additional cost comes with zero benefits hence it will always be less efficient. In short, there is no way pre-allocating threads to particular endpoints can improve performance compared to just adding the same number of additional threads to the general thread pool. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Per EndPoint Threads???
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Owen, Please do not top-post. I have re-ordered your post to be bottom-post. On 8/11/17 10:12 PM, Owen Rubel wrote: > On Fri, Aug 11, 2017 at 5:58 PM, <christop...@baus.net> wrote: > >>> Hi All, >>> >>> I'm looking for a way (or a tool) in Tomcat to associate >>> threads with endpoints. >> >> It isn't clear to me why this would be necessary. Threads should >> be allocated on demand to individual requests. If one route sees >> more traffic, then it should automatically be allocated more >> threads. This could starve some requests if the maximum number of >> threads had been allocated to a lessor used route, while >> available threads went unused for more commonly used route. > > Absolutely but it could ramp up more threads as needed. > > I base the logic on neuron and neuralTransmitters. When neurons > talk to each other, they send back neural transmitters to enforce > that pathway. > > If we could do the same through threads by adding additional > threads for endpoints that receive more traffic vs those which do > not, it would enforce better and faster communication on those > paths.> The current way Tomcat does it is not dynamic and it just > applies to ALL pathways equally which is not efficient. How would this improve efficiency at all? There is nothing inherently "showy" or "edity" about a particular thread; each request-processing thread is indistinguishable from any other. I don't believe there is a way to improve the situation even if "per-endpoint" (whatever that would mean) threads were a possibility. What would you attach to a thread that would make it any better at editing records? Or deleting them? - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlmOi2YACgkQHPApP6U8 pFh+ohAAkIDqAaZK3mmQsSAE100a4RMwCyAjT076eiEkqj3MVJHUBuYf2adNlRYe jvcKJCmvu061mW+/kos0+YIrt6ao2j60+fryX1goMOXhBxxrSlioccOwLkBu4HIG SB/AuFIYqIG6S1ICqVunCFJsrYnMuJEX6WfA8O7G+sQWFH54w9XadewabEduu3uO PwoP14a7XFOC8RPp9HM9Rdx8EfADRXrFugN0E5YSjXN5cdMs8bxJcabo8vjVnfNH JDCkvF0tDd+FWj4t/AqXugM6fc6EYb8sSxEifxkdbu701A4doe8n1d1zawd3+qd4 IBVR6jFDHGqRm6cHvmhI8G4Tlx6c5EX29ZGTTdKnPvNloyob0a3/LauPJMr/97Xv eIsj0shEfbUOWgcBWHRMbXbmZRjOAU7wxXtm2KsLZpJ6ZVZe9c7wSRLThYjp0Yyx jgpwHN4sVPGG821trGht29E3v1e2GN1A7nuYbM7A7BK1PHP3MmLozVxAMxAip1T4 hVaVDHc1hd/G79Jvugq/T7atKQfOetLD4vg9ZFGIukaPZwA+3BtMYTNWn/bX2u9d hBsWCw5Abn1SABlQ4cl87OJF9jya4p/P3Kqejyg9jbDbUy9J21QFEP6n5qHy9/vy Jg6cjWpho6s9Ajx690ZNsdudDPoRuBe2TRLkFTOnUXsgwHTmToY= =tiO+ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Per EndPoint Threads???
Absolutely but it could ramp up more threads as needed. I base the logic on neuron and neuralTransmitters. When neurons talk to each other, they send back neural transmitters to enforce that pathway. If we could do the same through threads by adding additional threads for endpoints that receive more traffic vs those which do not, it would enforce better and faster communication on those paths. The current way Tomcat does it is not dynamic and it just applies to ALL pathways equally which is not efficient. Owen Rubel oru...@gmail.com On Fri, Aug 11, 2017 at 5:58 PM, <christop...@baus.net> wrote: > > Hi All, > > > > I'm looking for a way (or a tool) in Tomcat to associate threads with > > endpoints. > > It isn't clear to me why this would be necessary. Threads should be > allocated on demand to individual requests. If one route sees more > traffic, then it should automatically be allocated more threads. This > could starve some requests if the maximum number of threads had been > allocated to a lessor used route, while available threads went unused > for more commonly used route. > > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Per EndPoint Threads???
> Hi All, > > I'm looking for a way (or a tool) in Tomcat to associate threads with > endpoints. It isn't clear to me why this would be necessary. Threads should be allocated on demand to individual requests. If one route sees more traffic, then it should automatically be allocated more threads. This could starve some requests if the maximum number of threads had been allocated to a lessor used route, while available threads went unused for more commonly used route. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Per EndPoint Threads???
Hi All, I'm looking for a way (or a tool) in Tomcat to associate threads with endpoints. The reason being is that on a whole, threads are used not by the whole system but distributed dynamically to specific pieces. Tomcat repeats this process over and over but never stores this knowledge of which pieces endpoints continually have high volume and which have lower volume traffic. Even at startup/restart, these individual endpoints in the system should start with a higher number of threads by DEFAULT as a result of the continual higher traffic. Is there a way to assign/distribute much like 'load balancing' the number of threads across available endpoints??? ie: localhost/v0.1/user/show: 50% localhost/v0.1/user/create: 10% localhost/v0.1/user/edit: 5% localhost/v0.1/user/delete: 2% Owen Rubel oru...@gmail.com
Re: able to see more than 10 threads, maxThread configuration to 10
On 04/05/17 16:42, Jain Shailu wrote: > We have updated the maxThread configuration to 10 at below place in > server.xml of DEV environment. However I was able to see more than 10 > threads. I have also attached the server.xml > > > I need your help to understand if I am doing something wrong or it is > an expected behaviour? It depends. What are the names of the threads? Mark > > > A "Connector" represents an endpoint by which requests are received > and responses are returned. Documentation at : > Java HTTP Connector: /docs/config/http.html (blocking & non-blocking) > Java AJP Connector: /docs/config/ajp.html > APR (HTTP/AJP) Connector: /docs/apr.html > Define a non-SSL HTTP/1.1 Connector on port 8080 > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
able to see more than 10 threads, maxThread configuration to 10
We have updated the maxThread configuration to 10 at below place in server.xml of DEV environment. However I was able to see more than 10 threads. I have also attached the server.xml I need your help to understand if I am doing something wrong or it is an expected behaviour? A "Connector" represents an endpoint by which requests are received and responses are returned. Documentation at : Java HTTP Connector: /docs/config/http.html (blocking & non-blocking) Java AJP Connector: /docs/config/ajp.html APR (HTTP/AJP) Connector: /docs/apr.html Define a non-SSL HTTP/1.1 Connector on port 8080 -- Thanks Shailesh
Re: what could this mean if there are many many http threads in this state?
On 11/04/17 13:35, Johan Compagner wrote: > Thread: http-443-72, state: RUNNABLE, total cpu time: 5583984.375ms, total > user time: 5451312.5ms > java.net.SocketInputStream.socketRead0(Native Method) > java.net.SocketInputStream.socketRead(Unknown Source) > java.net.SocketInputStream.read(Unknown Source) > java.net.SocketInputStream.read(Unknown Source) > sun.security.ssl.InputRecord.readFully(Unknown Source) > sun.security.ssl.InputRecord.read(Unknown Source) > sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source) > sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source) > sun.security.ssl.AppInputStream.read(Unknown Source) > > org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:733) > > org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:364) > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:814) > > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:620) > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > java.lang.Thread.run(Unknown Source) > > it seems that there is an incomming request right? That is the normal waiting state for keep-alive connections with the HTTP BIO connector. Mark > That is not really a thread just in a a waiting state? > that should be something like this: > > Thread: http-443-70, state: WAITING, total cpu time: 7274093.75ms, total > user time: 7110015.625ms > java.lang.Object.wait(Native Method) > java.lang.Object.wait(Unknown Source) > org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458) > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484) > java.lang.Thread.run(Unknown Source) > > > or could the above be a keep alive thing? so the browser keeps the > connection alive for a while to reuse? > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
what could this mean if there are many many http threads in this state?
Thread: http-443-72, state: RUNNABLE, total cpu time: 5583984.375ms, total user time: 5451312.5ms java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.socketRead(Unknown Source) java.net.SocketInputStream.read(Unknown Source) java.net.SocketInputStream.read(Unknown Source) sun.security.ssl.InputRecord.readFully(Unknown Source) sun.security.ssl.InputRecord.read(Unknown Source) sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source) sun.security.ssl.SSLSocketImpl.readDataRecord(Unknown Source) sun.security.ssl.AppInputStream.read(Unknown Source) org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:733) org.apache.coyote.http11.InternalInputBuffer.parseRequestLine(InternalInputBuffer.java:364) org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:814) org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:620) org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) java.lang.Thread.run(Unknown Source) it seems that there is an incomming request right? That is not really a thread just in a a waiting state? that should be something like this: Thread: http-443-70, state: WAITING, total cpu time: 7274093.75ms, total user time: 7110015.625ms java.lang.Object.wait(Native Method) java.lang.Object.wait(Unknown Source) org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458) org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484) java.lang.Thread.run(Unknown Source) or could the above be a keep alive thing? so the browser keeps the connection alive for a while to reuse? -- Johan Compagner Servoy
Re: [OT] Tomcat listener not coming up - no stuck threads
So I wanted to make sure I didn't lose track of this. Did you want me to revise what I put out there, or were you going to bring in your changes? On Thu, Dec 15, 2016 at 5:06 PM Christopher Schultz < ch...@christopherschultz.net> wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > John, > > On 12/15/16 5:03 PM, John D. Ament wrote: > > On Thu, Dec 15, 2016 at 4:06 PM Christopher Schultz < > > ch...@christopherschultz.net> wrote: > > > > Mark, > > > > On 12/15/16 2:44 PM, Mark Thomas wrote: > >>>> On 15/12/2016 19:35, Christopher Schultz wrote: > >>>>> John, > >>>>> > >>>>> On 12/14/16 10:01 PM, John D. Ament wrote: > >>>>>> I was feeling ambitious so I gave this a shot. Here's > >>>>>> what I changed: > >>>>>> https://github.com/johnament/tomcat85/commit/a0281b2d4e0fea7cb0f7 > a9d <https://github.com/johnament/tomcat85/commit/a0281b2d4e0fea7cb0f7a9d> > > > >>>>>> > 3c > > <https://github.com/johnament/tomcat85/commit/a0281b2d4e0fea7cb0f7a9d3 > c> > >>>>> > >>>>>> > > > > > d1e750b48498ff2 > >>>> > >>>> That was pretty much what I was thinking. > > > > > >> If you want it, let me know. I can provide a patch. I'm not > >> sure what tests to add, other than verify no extra threads were > >> created. I need to setup a bugzilla account too. > > > > > > > >>>> > >>>>> My > >>>>> > >>>>> tack was going to be to create an ExecutorService that > >>>>> didn't dispatch. That way, less code has to change. > >>>> > >>>> Seems like more work. And doesn't solve the original problem > >>>> of the main thread holding a lock. > > > > No, it won't hold a lock. The (Abstract)ExecutorService just > > dispatches to the execute(Runnable job) method which can simply > > be: > > > > job.run() > > > > In that case, only one thread is ever involved: the thread that > > submitted the job. > > > > > >> The problem is that tomcat is expecting a ThreadPoolExecutor. It > >> may make sense to wrap it in a delegate that just handles the > >> calls you need, since you're only calling one method on the > >> executor. That would simplify this logic as well. > > That was exactly what I was going to do: create a class that would do > double-duty. So rather than if/else blocks behaving differently for > the case where 1 thread is being used rather than >1, the > ExecutorService (ThreadPoolExecutor) would handle the difference. So > it would be a one-line change to the ContainerBase class. > > - -chris > -BEGIN PGP SIGNATURE- > Comment: GPGTools - http://gpgtools.org > Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ > > iQIcBAEBCAAGBQJYUxPwAAoJEBzwKT+lPKRYmlwP/A/3F64wo3nJ46if8Ldm72hG > 1zCUHD+ticxSX/wnjxjjItpy24jLw74d7ub/zuEGdXbcz0FZkL7HkX7uBqsoq/5K > SR3fS4Vukz7h4aohDrkGFqMYcI48xZOBj2qPP26PNxKgBD4JbGGA/5kP2g1ItVbC > EWhKRhHUJbP+sPqbyw6EGgxuGFczen/nXXhiTgDY5J5heG/4UoAOzCMJSbWzmywG > lpPsa3UPvrErf43WE5AH8TijwxSOOSMCQDgVSDWgo5obqPRUzYE69f1h4pap7j5M > HJlJZyaR+/5H3epfgyim1D4G6+LudvbS8chLIl0Dq9cPcFfbUny118ZSd9zs1n2D > Np0q03wL4fTSfZTe95q8OWTdmGdwpg++GIoL8hsDg6wzkkPnWOX0mC3qc4TZ5Z8P > cr+jW5VW12INwKyJROqe3KfX0IJQHS+MfddO+VDaItumibAeDeNEY+TTRsOWSq7k > MSFH8Uic/S7EoWCTrGILTVgiZ5swAZIgIYSThSEukE3smltdnjZoR9juqwqz8ZgC > ivIhrHx0lyHr36kbtebCgC/G9IRmv1oO5zNSUglCOmB4fsssxX84CUZXrcx1jxA2 > rdxgImaDw1igiLyhA++RUEL33t38wFzzqk0yfuNSZIa3jX7Obs89LY+wS6Oi6lMI > bmjL4dBBuzbKbDgnTn26 > =JFw5 > -END PGP SIGNATURE- > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: [OT] Tomcat listener not coming up - no stuck threads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 John, On 12/15/16 5:03 PM, John D. Ament wrote: > On Thu, Dec 15, 2016 at 4:06 PM Christopher Schultz < > ch...@christopherschultz.net> wrote: > > Mark, > > On 12/15/16 2:44 PM, Mark Thomas wrote: >>>> On 15/12/2016 19:35, Christopher Schultz wrote: >>>>> John, >>>>> >>>>> On 12/14/16 10:01 PM, John D. Ament wrote: >>>>>> I was feeling ambitious so I gave this a shot. Here's >>>>>> what I changed: >>>>>> https://github.com/johnament/tomcat85/commit/a0281b2d4e0fea7cb0f7 a9d > >>>>>> 3c > <https://github.com/johnament/tomcat85/commit/a0281b2d4e0fea7cb0f7a9d3 c> >>>>> >>>>>> > > d1e750b48498ff2 >>>> >>>> That was pretty much what I was thinking. > > >> If you want it, let me know. I can provide a patch. I'm not >> sure what tests to add, other than verify no extra threads were >> created. I need to setup a bugzilla account too. > > > >>>> >>>>> My >>>>> >>>>> tack was going to be to create an ExecutorService that >>>>> didn't dispatch. That way, less code has to change. >>>> >>>> Seems like more work. And doesn't solve the original problem >>>> of the main thread holding a lock. > > No, it won't hold a lock. The (Abstract)ExecutorService just > dispatches to the execute(Runnable job) method which can simply > be: > > job.run() > > In that case, only one thread is ever involved: the thread that > submitted the job. > > >> The problem is that tomcat is expecting a ThreadPoolExecutor. It >> may make sense to wrap it in a delegate that just handles the >> calls you need, since you're only calling one method on the >> executor. That would simplify this logic as well. That was exactly what I was going to do: create a class that would do double-duty. So rather than if/else blocks behaving differently for the case where 1 thread is being used rather than >1, the ExecutorService (ThreadPoolExecutor) would handle the difference. So it would be a one-line change to the ContainerBase class. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJYUxPwAAoJEBzwKT+lPKRYmlwP/A/3F64wo3nJ46if8Ldm72hG 1zCUHD+ticxSX/wnjxjjItpy24jLw74d7ub/zuEGdXbcz0FZkL7HkX7uBqsoq/5K SR3fS4Vukz7h4aohDrkGFqMYcI48xZOBj2qPP26PNxKgBD4JbGGA/5kP2g1ItVbC EWhKRhHUJbP+sPqbyw6EGgxuGFczen/nXXhiTgDY5J5heG/4UoAOzCMJSbWzmywG lpPsa3UPvrErf43WE5AH8TijwxSOOSMCQDgVSDWgo5obqPRUzYE69f1h4pap7j5M HJlJZyaR+/5H3epfgyim1D4G6+LudvbS8chLIl0Dq9cPcFfbUny118ZSd9zs1n2D Np0q03wL4fTSfZTe95q8OWTdmGdwpg++GIoL8hsDg6wzkkPnWOX0mC3qc4TZ5Z8P cr+jW5VW12INwKyJROqe3KfX0IJQHS+MfddO+VDaItumibAeDeNEY+TTRsOWSq7k MSFH8Uic/S7EoWCTrGILTVgiZ5swAZIgIYSThSEukE3smltdnjZoR9juqwqz8ZgC ivIhrHx0lyHr36kbtebCgC/G9IRmv1oO5zNSUglCOmB4fsssxX84CUZXrcx1jxA2 rdxgImaDw1igiLyhA++RUEL33t38wFzzqk0yfuNSZIa3jX7Obs89LY+wS6Oi6lMI bmjL4dBBuzbKbDgnTn26 =JFw5 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
