Re: Tomcat request hooks
Hi Mark, Thanks for the detailed reply. I think I get it now. The problem I was trying to solve was to dynamically control the concurrency/in-flight requests (requests being processed by worker threads + requests in queue) based on the response times of requests (time spent in queue + time spent in worker threads). On Sat, May 9, 2020 at 8:12 PM Mark Thomas wrote: > On 09/05/2020 14:08, Vikas Kumar wrote: > > I did not know about the images being dropped. > > > > Here are the steps I depicted in the image: > > > > Step 0: Request accepted by OS > > Step 1: Request sent to Tomcat. Tomcat accepts the request if it's > internal > > queue is not full, else rejects the request > > Step 2: Tomcat adds the request to the queue if all worker threads are > > busy, else sends the request to one of the free worker threads > > Step 3: Request is picked by the worker thread for processing > > The above sequence isn't quite right. > > Step 0 is the accept queue. See acceptCount in the docs. Note that not > every OS honours this setting and may use a smaller or larger value. It > the accept queue is full, the connection gets dropped. > > Step 1 is the Acceptor thread. This runs in a fairly tight loop. If the > current connection count > maxConnections, the loop pauses. The Acceptor > thread takes the next connection from the accept queue, accepts it > (creates the connection) and then creates a task for the executor to > process that connection. It then loops. If the connection isn't accepted > in time, it will time out. The queue for the executor is essentially > unlimited - although in practice it should never exceed maxConnections > > Step 2 Is the executor. If there are tasks in the queue and spare worker > threads available, the task (== connection) is allocated a worker thread > for processing. > > Step 3 is that the executor/worker thread processes the connection. When > the request is completed, the connection is either kept open (e.g. HTTP > keep alive) or closed. > > > I would like to add a hook between step(1) and step (2). As I mentioned > > earlier, I want to dynamically control the Tomcat internal queue size > based > > on some application logic. Is there a way to do that? > > Do you want to control the acceptCount, maxConnections or something else? > > What problem are you trying to solve? > > The overhead of the accept process and the allocation of requests to > workers is fairly light weight. I wonder if the simplest solution would > be to allow every request through to the application and then decide > whether to process it or reject it. You could provide an HTTP error > response at that point that might be more informative for your users. > > Mark > > - > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
Re: Tomcat request hooks
On 09/05/2020 14:08, Vikas Kumar wrote: > I did not know about the images being dropped. > > Here are the steps I depicted in the image: > > Step 0: Request accepted by OS > Step 1: Request sent to Tomcat. Tomcat accepts the request if it's internal > queue is not full, else rejects the request > Step 2: Tomcat adds the request to the queue if all worker threads are > busy, else sends the request to one of the free worker threads > Step 3: Request is picked by the worker thread for processing The above sequence isn't quite right. Step 0 is the accept queue. See acceptCount in the docs. Note that not every OS honours this setting and may use a smaller or larger value. It the accept queue is full, the connection gets dropped. Step 1 is the Acceptor thread. This runs in a fairly tight loop. If the current connection count > maxConnections, the loop pauses. The Acceptor thread takes the next connection from the accept queue, accepts it (creates the connection) and then creates a task for the executor to process that connection. It then loops. If the connection isn't accepted in time, it will time out. The queue for the executor is essentially unlimited - although in practice it should never exceed maxConnections Step 2 Is the executor. If there are tasks in the queue and spare worker threads available, the task (== connection) is allocated a worker thread for processing. Step 3 is that the executor/worker thread processes the connection. When the request is completed, the connection is either kept open (e.g. HTTP keep alive) or closed. > I would like to add a hook between step(1) and step (2). As I mentioned > earlier, I want to dynamically control the Tomcat internal queue size based > on some application logic. Is there a way to do that? Do you want to control the acceptCount, maxConnections or something else? What problem are you trying to solve? The overhead of the accept process and the allocation of requests to workers is fairly light weight. I wonder if the simplest solution would be to allow every request through to the application and then decide whether to process it or reject it. You could provide an HTTP error response at that point that might be more informative for your users. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat request hooks
On Sat, 9 May 2020 at 9:09 PM, Vikas Kumar wrote: > I did not know about the images being dropped. > > Here are the steps I depicted in the image: > > Step 0: Request accepted by OS > Step 1: Request sent to Tomcat. Tomcat accepts the request if it's internal > queue is not full, else rejects the request > Step 2: Tomcat adds the request to the queue if all worker threads are > busy, else sends the request to one of the free worker threads > Step 3: Request is picked by the worker thread for processing > > I would like to add a hook between step(1) and step (2). As I mentioned > earlier, I want to dynamically control the Tomcat internal queue size based > on some application logic. Is there a way to do that? > The queue is managed here: https://github.com/apache/tomcat/blob/14406c0b49c29fd05dd8f707b62ece38429e16f8/java/org/apache/tomcat/util/net/AbstractEndpoint.java#L429 And there is an mbean here for the NIO connector: https://github.com/apache/tomcat/blob/7d66c652a3e4d64b9711712fa1409ba1261effd8/java/org/apache/tomcat/util/net/mbeans-descriptors.xml#L91 I don’t know if that makes the attribute query able as well as writable, though. You may want to experiment and check. What you are looking at writing, though, will be code that lives outside your warfile and as an extension to Tomcat ( is you want to enhance the AbstractEndpoint above), or at least JMX privileges (in case changing the limit is permitted by that mbean config) Alternatively, you could consider setting maxConnections and or acceptCount to higher numbers. See : https://tomcat.apache.org/tomcat-8.5-doc/config/http.html Ram > > > > > > > > > > > > > > > On Sat, May 9, 2020 at 6:16 PM calder wrote: > > > On Sat, May 9, 2020, 07:16 Vikas Kumar wrote: > > > > > As per my understanding (using a Spring Boot app with Tomcat server), > we > > > define: > > > > > >- Max no. of worker threads (maxThreads, default 200) > > >- Tomcat queue size (maxConnections, default 8192 for APR, 1 for > > >NIO). When all worker threads are busy, requests are placed into the > > queue. > > >As worker threads free up, queued requests are sent to them in FIFO > > order > > > > > > > > I'll assume the "steps" are shown in the "broken" image (inline/attached > > images are normally dropped) - you'll need the convert whatever text is > > contained into actual text and post back. > > > > > > [image: Untitled Diagram.png] > > > > > > I can add a hook at step (3) using a servlet filter or > > > > > [ snip ] > > >
Re: Tomcat request hooks
I did not know about the images being dropped. Here are the steps I depicted in the image: Step 0: Request accepted by OS Step 1: Request sent to Tomcat. Tomcat accepts the request if it's internal queue is not full, else rejects the request Step 2: Tomcat adds the request to the queue if all worker threads are busy, else sends the request to one of the free worker threads Step 3: Request is picked by the worker thread for processing I would like to add a hook between step(1) and step (2). As I mentioned earlier, I want to dynamically control the Tomcat internal queue size based on some application logic. Is there a way to do that? On Sat, May 9, 2020 at 6:16 PM calder wrote: > On Sat, May 9, 2020, 07:16 Vikas Kumar wrote: > > > As per my understanding (using a Spring Boot app with Tomcat server), we > > define: > > > >- Max no. of worker threads (maxThreads, default 200) > >- Tomcat queue size (maxConnections, default 8192 for APR, 1 for > >NIO). When all worker threads are busy, requests are placed into the > queue. > >As worker threads free up, queued requests are sent to them in FIFO > order > > > > > I'll assume the "steps" are shown in the "broken" image (inline/attached > images are normally dropped) - you'll need the convert whatever text is > contained into actual text and post back. > > > [image: Untitled Diagram.png] > > > > I can add a hook at step (3) using a servlet filter or > > > [ snip ] >
Re: Tomcat request hooks
On Sat, May 9, 2020, 07:16 Vikas Kumar wrote: > As per my understanding (using a Spring Boot app with Tomcat server), we > define: > >- Max no. of worker threads (maxThreads, default 200) >- Tomcat queue size (maxConnections, default 8192 for APR, 1 for >NIO). When all worker threads are busy, requests are placed into the queue. >As worker threads free up, queued requests are sent to them in FIFO order > > I'll assume the "steps" are shown in the "broken" image (inline/attached images are normally dropped) - you'll need the convert whatever text is contained into actual text and post back. [image: Untitled Diagram.png] > > I can add a hook at step (3) using a servlet filter or > [ snip ]
Tomcat request hooks
As per my understanding (using a Spring Boot app with Tomcat server), we define: - Max no. of worker threads (maxThreads, default 200) - Tomcat queue size (maxConnections, default 8192 for APR, 1 for NIO). When all worker threads are busy, requests are placed into the queue. As worker threads free up, queued requests are sent to them in FIFO order [image: Untitled Diagram.png] I can add a hook at step (3) using a servlet filter or request interceptor for pre and post-processing of the request. My question is: can I add a similar hook at step (2)? Basically I want to dynamically control the Tomcat queue size based on some application logic.
