Re: Which Connector properties will be used in case of redirection?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Suvendu, On 1/18/18 2:55 AM, Suvendu Sekhar Mondal wrote: > Hi Christopher, > > On Wed, Jan 17, 2018 at 10:41 PM, Christopher Schultz >wrote: >> -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 >> >> Suvendu, >> >> Which version of Tomcat? >> > > Tomcat version is 7.0.55 > >> On 1/17/18 8:20 AM, Suvendu Sekhar Mondal wrote: >>> I am seeing one issue. Under high load sporadically one web >>> service call fails with: "java.net.SocketTimeoutException: >>> Read timed out" after 60 Sec mark. >>> >>> In our app, httpd routes requests to then it get >>> redirected to 8889. >> >> Can you be very explicit about what you mean by "redirect", >> here? > > I was talking about auto redirection from to 8889 by > Catalina. > >>> Connector which listens on has connectionTimeout=2 but >>> the one which listens to 8889 does not have any >>> connectionTimeout set explicitly. As per doc, default >>> connectionTimeout value 60 Sec will be set for connector which >>> listens on 8889 - and that is playing a role here. Do you think >>> this assumption is correct? >>> >>> Here are the connector properties: >> protocol="HTTP/1.1" connectionTimeout="2" >>> redirectPort="8889" /> >> >> So you are probably expecting that all HTTP traffic to : will >> be redirected to HTTPS over :8889? >> > > Yes. > >> How long does the redirect take? >> > > I do not know how to measure redirection time. Please let me know > if there any mechanism to trace it. > >> Under heavy load, you might be running out of threads depending >> upon a lot of factors, such as what kind of protocol and endpoint >> are actually being used. It looks like you are using the APR >> connector (due to SSLCertificateFile) which has some positive >> attributes with request to scalability while NIO would probably >> be the best choice if possible. >> >> It's got terrible performance, though, unless you use the >> OpenSSL provider (which requires Tomcat 8.5 or later). >> >>> >> SSLCertificateKeyFile="C:\mykey.key" SSLEnabled="true" >>> acceptCount="100" clientAuth="false" >>> disableUploadTimeout="true" enableLookups="false" >>> keystoreFile="conf/.keystore" maxHttpHeaderSize="8192" >>> maxSavePostSize="-1" maxThreads="200" minSpareThreads="20" >>> protocol="HTTP/1.1" scheme="https" secure="true" >>> sslProtocol="TLS1.2" >>> compressableMimeTypes="text/html,text/xml,text/plain,application/jso n" >>> >>> >> >>> compressionMinSize="2048" >>> compression="force" threadPriority="6" /> >> >> When you say "high load", what kind of load are you talking >> about, specifically? >> >> You may simply have too much traffic for your existing hardware >> to handle. No amount of configuration can fix that. - -chris > > We have 9 JVMs. As per current setting, per JVM max concurrency > can reach up-to 300(maxThreads="200"+acceptCount="100"). Issue was > reported when 2300 users on the system. So that's 2300 users across all 9 JVMs with the same configuration? Your cluster ought to be able to support 9 * 200 = 1800 simultaneous active users plus 9 * 100 = 900 queued in the OS's TCP wait queue. 2300 should be within your capabilities, but it might depend heavily upon how the load-balancer decided to allocate requests. If it sent all the users to the same node in the cluster, then you'll see bad performance even if you "should" be able to handle that much load. Where is the SocketTimeoutException being thrown? On the server, or on the client? If it's on the server, then it's the client's fault (or the load-balancer's fault) and it's not a problem at all with your server capacity. If it's happening on the client, it makes a lot more sense that the server might be overloaded. But that 60-second timeout would have to be on the client, not on the server side. So I'm a little confused as to what is going on, here. > So, I think it is not a case of thread exhaustion but something > else. Most possibly few JVMs were under stress due to uneven load > balancing(we have stickysession). This issue just came to me and > all resource usage data are gone. I have asked for re-run so that I > can analyze it properly. Let's see. :) It would be good to see what the session-load and the current-user (meaning in-process request) load was on each node in the cluster at the time, and which of them seems to be failing. > Back to my original question, in case of HTTP to HTTPS auto > redirection, Connector properties set for the later will be > finally applied to the request - is that a correct statement? The order of events goes like this: 1. Client connects to server via HTTP 2. Server redirects client to HTTPS port [1] 3. Client connects to server via HTTPS 2. Server responds appropriately[2] [1] here is the 20s connect timeout, which isn't actually a connection timeout... it's a timeout waiting for the client to present the HTTP request line before terminating the
Re: Which Connector properties will be used in case of redirection?
Hi Christopher, On Wed, Jan 17, 2018 at 10:41 PM, Christopher Schultzwrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA256 > > Suvendu, > > Which version of Tomcat? > Tomcat version is 7.0.55 > On 1/17/18 8:20 AM, Suvendu Sekhar Mondal wrote: >> I am seeing one issue. Under high load sporadically one web >> service call fails with: "java.net.SocketTimeoutException: Read >> timed out" after 60 Sec mark. >> >> In our app, httpd routes requests to then it get redirected >> to 8889. > > Can you be very explicit about what you mean by "redirect", here? I was talking about auto redirection from to 8889 by Catalina. >> Connector which listens on has connectionTimeout=2 but the >> one which listens to 8889 does not have any connectionTimeout set >> explicitly. As per doc, default connectionTimeout value 60 Sec will >> be set for connector which listens on 8889 - and that is playing a >> role here. Do you think this assumption is correct? >> >> Here are the connector properties: > protocol="HTTP/1.1" connectionTimeout="2" redirectPort="8889" >> /> > > So you are probably expecting that all HTTP traffic to : will be > redirected to HTTPS over :8889? > Yes. > How long does the redirect take? > I do not know how to measure redirection time. Please let me know if there any mechanism to trace it. > Under heavy load, you might be running out of threads depending upon a > lot of factors, such as what kind of protocol and endpoint are > actually being used. It looks like you are using the APR connector > (due to SSLCertificateFile) which has some positive attributes with > request to scalability while NIO would probably be the best choice if > possible. > > It's got terrible performance, though, unless you use the OpenSSL > provider (which requires Tomcat 8.5 or later). > >> > SSLCertificateKeyFile="C:\mykey.key" SSLEnabled="true" >> acceptCount="100" clientAuth="false" disableUploadTimeout="true" >> enableLookups="false" keystoreFile="conf/.keystore" >> maxHttpHeaderSize="8192" maxSavePostSize="-1" maxThreads="200" >> minSpareThreads="20" protocol="HTTP/1.1" scheme="https" >> secure="true" sslProtocol="TLS1.2" >> compressableMimeTypes="text/html,text/xml,text/plain,application/json" >> >> > compressionMinSize="2048" >> compression="force" threadPriority="6" /> > > When you say "high load", what kind of load are you talking about, > specifically? > > You may simply have too much traffic for your existing hardware to > handle. No amount of configuration can fix that. > - -chris We have 9 JVMs. As per current setting, per JVM max concurrency can reach up-to 300(maxThreads="200"+acceptCount="100"). Issue was reported when 2300 users on the system. So, I think it is not a case of thread exhaustion but something else. Most possibly few JVMs were under stress due to uneven load balancing(we have stickysession). This issue just came to me and all resource usage data are gone. I have asked for re-run so that I can analyze it properly. Let's see. :) Back to my original question, in case of HTTP to HTTPS auto redirection, Connector properties set for the later will be finally applied to the request - is that a correct statement? Thanks! Suvendu - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Which Connector properties will be used in case of redirection?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Suvendu, Which version of Tomcat? On 1/17/18 8:20 AM, Suvendu Sekhar Mondal wrote: > I am seeing one issue. Under high load sporadically one web > service call fails with: "java.net.SocketTimeoutException: Read > timed out" after 60 Sec mark. > > In our app, httpd routes requests to then it get redirected > to 8889. Can you be very explicit about what you mean by "redirect", here? > Connector which listens on has connectionTimeout=2 but the > one which listens to 8889 does not have any connectionTimeout set > explicitly. As per doc, default connectionTimeout value 60 Sec will > be set for connector which listens on 8889 - and that is playing a > role here. Do you think this assumption is correct? > > Here are the connector properties: protocol="HTTP/1.1" connectionTimeout="2" redirectPort="8889" > /> So you are probably expecting that all HTTP traffic to : will be redirected to HTTPS over :8889? How long does the redirect take? Under heavy load, you might be running out of threads depending upon a lot of factors, such as what kind of protocol and endpoint are actually being used. It looks like you are using the APR connector (due to SSLCertificateFile) which has some positive attributes with request to scalability while NIO would probably be the best choice if possible. It's got terrible performance, though, unless you use the OpenSSL provider (which requires Tomcat 8.5 or later). > SSLCertificateKeyFile="C:\mykey.key" SSLEnabled="true" > acceptCount="100" clientAuth="false" disableUploadTimeout="true" > enableLookups="false" keystoreFile="conf/.keystore" > maxHttpHeaderSize="8192" maxSavePostSize="-1" maxThreads="200" > minSpareThreads="20" protocol="HTTP/1.1" scheme="https" > secure="true" sslProtocol="TLS1.2" > compressableMimeTypes="text/html,text/xml,text/plain,application/json" > > compressionMinSize="2048" > compression="force" threadPriority="6" /> When you say "high load", what kind of load are you talking about, specifically? You may simply have too much traffic for your existing hardware to handle. No amount of configuration can fix that. - -chris -BEGIN PGP SIGNATURE- Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQJRBAEBCAA7FiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlpfg7sdHGNocmlzQGNo cmlzdG9waGVyc2NodWx0ei5uZXQACgkQHPApP6U8pFj12Q//WlJMd0JpQJVqFt1z d3aI52pVFQih3Bii8UOAZzZr6omPqqFlkKt2cnAg8lWA3KQaQLJRyzlJKthIzNsB 12nMQlbQzCJZqn1IzggxtEWNL8KDVFY3fo2heJLrEANrIcXx+C6MAb1FLTeZ2/XP V9R1u5q8mOdVgPyj4vITIsjn0oaa4pZ/ypkYwRATT9BRBy8rxkzVneDKPo8ILfdZ Cp1Says1JD4uIEJn9VUlITOC1/08XyeHtrZXR3j3w9A07x9mLdHjVi6yxyyGcMoz u0X/bt1E+Q6LkfTaQ64ubh2dbPXW4JgQy8I43Qzbxytwc2SdB691Un8lAhdnxaIk zBioeiR4Kp+ATOuwbSPqJZ8n15pJENp95KvtR7B+xKn2klKMhuXHMb98OwdL7XBu AeoGW1+RNpEwq1A8l4XmrmvFmrkqKTJVM8kxrPM3lTpTuQcvEtKk0OHOqlD10TGm ldxP61qBLPK1ekjF/8CWxCO011wIvCoWySI8FDLOeXAo8F0ueBA0okajB2+9usqK KpMckGQuRsoC6yVKMAz+ej3UCMiqQPSuuK06Pv8UrI7qTRGqRkBtVcSOIucXKC0S 6QrB4Mm6HajD7+d2/bOd4lJGRSSx3IoWNRx62Hthu9H2MTpp1/EWTwc5aesq2Zmo OVsym/pIs8akxDw3Fkux5Imwr40= =DLo7 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Which Connector properties will be used in case of redirection?
Hello Everyone, I am seeing one issue. Under high load sporadically one web service call fails with: "java.net.SocketTimeoutException: Read timed out" after 60 Sec mark. In our app, httpd routes requests to then it get redirected to 8889. Connector which listens on has connectionTimeout=2 but the one which listens to 8889 does not have any connectionTimeout set explicitly. As per doc, default connectionTimeout value 60 Sec will be set for connector which listens on 8889 - and that is playing a role here. Do you think this assumption is correct? Here are the connector properties: Thanks! Suvendu - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org