Chris --
> You should also look at worker-thread availability. When you see these "high > latency" (which is usually a term reserved for I/O > characterization) events, do you have: > > 1. Available worker threads (from the executor thread pool) > 2. Any other shared/limited resource (e.g. DB connection pool) > Good thought. I should mention that the hosted application is canned, and is the same for all tomcat instances, with only minor variations in version between them. User workflow is also similar. Over the years we've developed a good feel for expected performance and resource utilization based on the user count per instance. So when one instance exhibits anomalous performance, we tend to go right to networking issues. > Also, are you seeing the otherwise unexpected slowness on each Tomcat > node, or are you seeing it at the load-balancer/multiplexer level? > We run multi-tenanted servers, with many instances of tomcat on each server. We've never seen issues at the load-balancer. Very occasionally, there might be a problem at the server level. When that happens, all instances on that server may become sluggish. What I'm talking about in this thread are cases where only one instance on a server is showing slowness in its jasper logs. Also, we typically do not see the same slowness when we test the application locally from the same network. I've had my eye on TCP retransmits as a possible culprit for a while, but I just didn't know for sure if my understanding of the tomcat processing timer is correct. --Eric > -----Original Message----- > From: Christopher Schultz <ch...@christopherschultz.net> > Sent: Thursday, September 10, 2020 8:24 AM > To: users@tomcat.apache.org > Subject: Re: Tomcat Processing Timer Question > > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA256 > > Eric, > > On 9/9/20 20:42, Eric Robinson wrote: > > Hi Chris -- > > > >> Are you have any specific problem you are trying to diagnose or fix? > >> Or are you just academically interested in what conditions might > cause "slow" > >> request processing? > > > > A little of both. We've been running about 1500 instances of tomcat > > for the past 15 years. We're not tomcat experts by any means, but > > we're always looking to refine our understanding of tomcat > > performance. Like many people, we have custom scripts (ours are in > > python) that parse the jasper logs and produce a report that > > summarizes responsiveness and helps us isolate underperforming tomcat > > instances and JSP calls. Occasionally, we see evidence of chronic high > > latency in processing time when there is no indication of bottlenecks > > or problems in the servers themselves or the database back-ends. We > > theorize that client connectivity could be responsible. > That is a reasonable conclusion. > > You should also look at worker-thread availability. When you see these "high > latency" (which is usually a term reserved for I/O > characterization) events, do you have: > > 1. Available worker threads (from the executor thread pool) 2. Any other > shared/limited resource (e.g. DB connection pool) > > Also, are you seeing the otherwise unexpected slowness on each Tomcat > node, or are you seeing it at the load-balancer/multiplexer level? > > - -chris > > >> -----Original Message----- From: Christopher Schultz > >> <ch...@christopherschultz.net> Sent: Wednesday, September 9, 2020 > >> 7:41 AM To: users@tomcat.apache.org Subject: Re: Tomcat Processing > >> Timer Question > >> > > Eric, > > > > On 9/8/20 17:29, Eric Robinson wrote: > >>>> Got it. So TCP retransmits can impact tomcat processing time under > >>>> certain conditions, more likely due to issues with receiving > >>>> requests from the client than sending responses. > > Well... buffering can happen either during the client-write phase or > > the server-read phase or both. > > > > Imagine a slow network like EDGE or something similar where the first > > byte arrives at Tomcat's poller and it handed-off to the > > request-processor (t=0 as far as Tomcat is concerned) and uploads a > > large image over that EDGE connection. The OS won't allocate an > > infinite input buffer, so at some point the Poller will get byte 0 > > when the client hasn't uploaded the complete request. It may still > > take several seconds to upload all those bytes. > > > > Imagine that the response is a transformed image so the response is > > also large. The OS won't allocate an infinite output buffer, so at > > some point the bytes will start streaming to the client (at a slow > > rate). When the output buffer fills, your request-processing thread > > will stall when calling ServletOutputStream.write() to write those > > image bytes. > > > > If your image transform is instantaneous, your access log will report > > that the request took "a long time" relative to the amount of time > > spent actually processing the request. Basically, you are just waiting > > on I/O the entire time. > > > > Are you have any specific problem you are trying to diagnose or fix? > > Or are you just academically interested in what conditions might cause > > "slow" request processing? > > > > Hope that helps, -chris > > > >>>>> -----Original Message----- From: Mark Thomas <ma...@apache.org> > >>>>> Sent: Tuesday, September 8, 2020 4:05 PM > >>>>> To: users@tomcat.apache.org Subject: Re: Tomcat Processing Timer > >>>>> Question > >>>>> > >>>>> On 08/09/2020 21:46, Eric Robinson wrote: > >>>>>> Hi Mark -- > >>>>>> > >>>>>> "If the request is split across multiple packets the timer starts > >>>>>> when Tomcat > >>>>> reads the first byte of the request from the first packet. > >>>>>> Tomcat stops the timer on a request after the last byte of the > >>>>>> response has > >>>>> been accepted by the network stack." > >>>>>> > >>>>>> Now we're getting somewhere. If tomcat starts its timer when it > >>>>>> reads the > >>>>> first byte of the client's request, and the request is split into > >>>>> multiple packets, doesn't it stand to reason that the timer would > >>>>> run longer when there are TCP retransmits? > >>>>> > >>>>> For the request, it depends. If the retransmit is for part of the > >>>>> request body and Tomcat hasn't read that far yet (or starting > >>>>> reading at all) then it probably won't impact the processing time. > >>>>> If Tomcat is performing a read and waiting for that packet then it > >>>>> will. > >>>>> > >>>>> For the response, not unless the response is sfficiently big and > >>>>> the retransmit sufficiently earlier in the response that the TCP > >>>>> buffers fill and Tomcat is blocked from further writes. > >>>>> > >>>>> Mark > >>>>> > >>>>> > >>>>>> > >>>>>> --Eric > >>>>>> > >>>>>>> -----Original Message----- From: Mark Thomas > <ma...@apache.org> > >>>>>>> Sent: Tuesday, September 8, 2020 > >>>>>>> 3:34 PM To: users@tomcat.apache.org Subject: Re: Tomcat > >>>>>>> Processing Timer Question > >>>>>>> > >>>>>>> On 08/09/2020 21:19, Eric Robinson wrote: > >>>>>>>> Hi Mark and Christopher, > >>>>>>>> > >>>>>>>> For clarification, suppose a client sends and HTTP POST request > >>>>>>>> which > >>>>>>> is bigger than the PMTU and has to be broken into multiple > >>>>>>> packets. It sounds like you're saying that the request is > >>>>>>> buffered by the network stack, and the stack does not send it up > >>>>>>> to tomcat until the full > >>>>> request is received. > >>>>>>> That would make sense if every HTTP request is encapsulated in > >>>>>>> its own separate TCP connection. Most of the time, that is not > >>>>>>> the case. A single connection is held open and used for multiple > >>>>>>> HTTP requests. The network stack has no understanding of > >>>>>>> anything above TCP, so it does not know when an HTTP request > >>>>>>> complete. > >>>>>>> It must therefore deliver whatever it has, and it would be up to > >>>>>>> tomcat to decide when the HTTP request is complete, wouldn't it? > >>>>>>>> > >>>>>>>> If that is the case, tomcat could receive a partial HTTP > >>>>>>>> request and > >>>>>>> would have to wait for the rest before processing it. > >>>>>>> So when does tomcat start its processing timer? > >>>>>>> > >>>>>>> Tomcat starts the processing timer as soon as Tomcat processes > >>>>>>> the first bytes of the request. In practice, this means the > >>>>>>> network stack has to deliver the data to Tomcat, the Poller > >>>>>>> fires a read event, a thread is allocated to process that read > >>>>>>> event, any TLS handshake has completed and Tomcat has read the > >>>>>>> first real byte of the request. > >>>>>>> > >>>>>>> If the request is split across multiple packets the timer starts > >>>>>>> when Tomcat reads the first byte of the request from the first > >>>>>>> packet. > >>>>>>> > >>>>>>> Tomcat stops the timer on a request after the last byte of the > >>>>>>> response has been accepted by the network stack. > >>>>>>> > >>>>>>> HTH, > >>>>>>> > >>>>>>> Mark > >>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>>> -----Original Message----- From: Christopher Schultz > >>>>>>>>> <ch...@christopherschultz.net> Sent: > >>>>>>>>> Tuesday, September 8, 2020 1:19 PM To: > >>>>>>>>> users@tomcat.apache.org Subject: Re: Tomcat Processing Timer > >>>>>>>>> Question > >>>>>>>>> > >>>>>>>> Eric, > >>>>>>>> > >>>>>>>> On 9/8/20 13:46, Eric Robinson wrote: > >>>>>>>>>>> It is my understanding that the AccessLogValve %D field > >>>>>>>>>>> records the time from when the last byte of the client's > >>>>>>>>>>> request is received to when the last byte of the server's > >>>>>>>>>>> response is placed on > >>>>> the wire. > >>>>>>>>>>> Is that correct? If so, would TCP retransmissions impact the > >>>>>>>>>>> timer? > >>>>>>>> > >>>>>>>> I'm not positive, but I believe Tomcat has zero visibility into > >>>>>>>> that level of detail. > >>>>>>>> > >>>>>>>>>>> If there are connectivity issues between the client and > >>>>>>>>>>> server, resulting in TCP retransmits, could that appear as > >>>>>>>>>>> higher response times in the localhost_access logs? > >>>>>>>> > >>>>>>>> This would only happen if the re-transmissions were to cause > >>>>>>>> network buffering in the OS such that the stream writes (at the > >>>>>>>> Java level) were to block (and therefore "take time" instead of > >>>>>>>> being essentially > >>>>>>> instantaneous). > >>>>>>>> > >>>>>>>> -chris > >>>>>>>>> > >>>>>>>>> -------------------------------------------------------------- > - --- > >> > >> > >>>>>>>>> > - --------------------------------------------------------------------- > >> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For > >> additional commands, e-mail: users-h...@tomcat.apache.org > > > > Disclaimer : This email and any files transmitted with it are > confidential and intended solely for intended recipients. If you are not the > named addressee you should not disseminate, distribute, copy or alter this > email. Any views or opinions presented in this email are solely those of the > author and might not represent those of Physician Select Management. > Warning: Although Physician Select Management has taken reasonable > precautions to ensure no viruses are present in this email, the company > cannot accept responsibility for any loss or damage arising from the use of > this email or attachments. > > > > --------------------------------------------------------------------- > > > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > > For additional commands, e-mail: users-h...@tomcat.apache.org > > > -----BEGIN PGP SIGNATURE----- > Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ > > iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAl9aKOoACgkQHPApP > 6U8 > pFjeSA/+K0/me8yZTGMcVfCvMrPLTlQENqXNGcTsF8ILgMpz3YVpmqSPjPGG > g1W0 > xdX8U5saN3T7dLqlLLxGUDoa+YYsIqzQnTYccxqIN4RdBtThem0QKZPrwgNYSS > m8 > ogzpSXhzG1/aQCMPs0RGFN8vdvCX7gAnqtYd5988b4RFgnMBfrA3Z87cc0kxcsJ > c > nr33L/aMpMf6HJIkX11TPTuQrF8TlplQx2y2RoNxRRdEslHNaxJyDsU03C237On8 > 7AjicqkRNxJuDbZOu6bCq2em1gi7u22syi50B2X5HRWNyK9PaVhST5ccx34Gref > 5 > y84DUh8Gqaae7AGdj7aY1LeE+enKM3gIjsOaqNDk2X4Q+B05VOhxFwSAo79D > NZY+ > WX6liPTuO6y6tFW0UmtREnxm6turfzRGfTPUVzGEyAMqP9rBxPuxS7KS39VL7 > N7n > /44zHi30R1beJ8I0fFh5+PRKm9eZsbDniLA5as2oL4Vu2as6e1XmjK+GRn4Pb9U9 > jdG+ySFsROeNFg2hkyAXpacLp0PyL1RuiUsACIy8vCTP7jRprrK3aAuV6d4W0pb > q > +llMh3NVpVzkVGKc5Nc2ydRb5OstQajUmr21u1jH3ze6Rk+I3SmJ2TwWYq4Un > pTI > z9NweMjRR/RzRsJKDGF7HtMMzva5ywNLiUdawNytE5mKT0buobs= > =67LH > -----END PGP SIGNATURE----- > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org Disclaimer : This email and any files transmitted with it are confidential and intended solely for intended recipients. If you are not the named addressee you should not disseminate, distribute, copy or alter this email. Any views or opinions presented in this email are solely those of the author and might not represent those of Physician Select Management. Warning: Although Physician Select Management has taken reasonable precautions to ensure no viruses are present in this email, the company cannot accept responsibility for any loss or damage arising from the use of this email or attachments.
