Re: VERY HIGH TRAFFIC TUNING
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 7/10/14, 5:40 PM, André Warnier wrote: Christopher Schultz wrote: ... Interesting... load average is a crude measure of activity; I suppose that having those timeouts means that there is activity on a thread even when there is no real work to be done. I do recommend leaving the timeouts set to their defaults (-1 = infinite). In general terms, I would definitely not put the connectionTimeout nor the keepAliveTimeout to infinite, if that is what you meant here. In fact, it is exactly what I meant. ConnectionTimeout infinite seems like a perfect setup for a DOS attack. Keep-alive timeout infinite seems like the perfect way to block a lot of threads doing nothing (and opening yourself to another kind of DOS attack). Anyone allowing outsiders to make AJP connections to their Tomcat backends deserves to be DOS'd. However, in this case, we are talking about the AJP Connector, which processes requests coming in via Apache httpd and mod_jk, so I guess that one can rely on the Apache front-end not to relay anything nasty to Tomcat. Right. Presumably, the Apache httpd configuration does not have infinite connection timeout nor keep-alive timeout. I certainly wouldn't set things up this way. Which in a way, raises the question of why these parameters are even available for setting on the AJP Connector. Should these not better be left to the discretion of Apache httpd and mod_jk in the first place ? These configuration directives help deal with firewalls that close connections without either party knowing that the connection is no longer valid. If you could not set a timeout, then the Tomcat side could have a thread waiting forever on a connection that would never have any data arrive ever again. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTwBftAAoJEBzwKT+lPKRYKj4P/2HY0TDBRKaij3bDcqbItWrg aDYE8glav8UD589/cGjzoFiIuOqPXa8k2Mp1vqn9jj1K3SrndcKZzUKZARC0sGSR 6L0dTRZhYsTCnxJ5SH77d6dukuElrf82c73DbtVUQU3ZBrsk9x7iKX2V/w978wrS jknziJ2xO5+oW+/n6Uri8zp57I2wzYLyCK3+MhyuEDqfDo3deBvsUefWiHqGJ/27 lVkhM+LLL1cgM8xddVXsbP9/Sj+bVP3k6pLdvmxx76n5KO1Og6Ib2Hg0cSH9vsJs ++Y+YVqKVzvDTGOHuUqINP6UT0eJPueaIDJzAMePDQpCcobB4iEOjHU7kqOuff5/ pWiXy0I15aPmtsQxdcGqA2ZXa0GjIKuuDH3B6QCxbsasXbt8RQ3IrfIsaB7uVW4Z pNSNBVxxW3Vdw8//8/YV5rOAf5UxdFPkrNeYfg2l8XeK+nDV0Ioly/KDRy5V6UbR EWZFJLLYgUsz9c39/uRpEVhbVrd7mrouswAcAPc6SQHobcymBmEsJycKab3h6HWU 2Wa+otuNVt6LGOGdfZRB7VfNwdU1ksUNd8dNmQ86ar/MtjRV5EzF4vEqGnL17l1N PuHlHL0UlCYJRfOZcRdUHbAeP/8qYN2uaPC/uoNxv5OM3cI0Sr3PIHspDF5uv0ne SL5SS/b2I/ursp1Ov61e =cHY9 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: VERY HIGH TRAFFIC TUNING
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Leon, On 7/10/14, 3:27 PM, Leon Rosenberg wrote: answering only to the one directed at me (or so I think): ;) On Thu, Jul 10, 2014 at 4:09 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Leon, If you have very fast connections, go for a smaller amount. If you have keepalive and slow connections, remember that every connection can hold 1-2 threads without doing anything at all. Hmm? If you have keepalive enabled, then every client will held an open connection for some period of time. Correct. Depending on browser brand and version, the browser will typically hold 1-2 connections to the server. These days, it's more like 8. http://www.stevesouders.com/blog/2008/03/20/roundup-on-parallel-connections/ http://www.browserscope.org/?category=network http://stackoverflow.com/questions/985431/max-parallel-http-connections-in-a-browser I think most browsers will hang up the phone, though, if they aren't making any requests. So for every logged in users that is clicking or issuing a request regularly (maybe once a minute or so), you can end up having one or two threads, dependending on how many connections the browsers holds. Those threads would be idle most of the time, but still not available to your threadpool. Usually they show up as Runnable somewhere in socket.read in the thread dump. So if you have 100 users with 150 connections and 151 threads in your threadpool you can end up with a completely blocked server with zero cpu usage. This is one of the reasons /not/ to use a BIO-based connector, even for AJP connections: while waiting for the next keepalive request over a connection, the BIO thread will be bound to the connection, sitting there waiting and not getting anything done. The NIO connector throws the connection into a poller queue and the thread is free to do more work. So, you can either get the same amount of work done with fewer threads, or handle more work with the same number of threads. This is important when you've got many frontends and many backends so the F x B product is high, but the effective load on any server really shouldn't be anywhere near F x B. If you use BIO, your maxThreads needs to be somewhere near F x B unless you want nasty hangs, etc. If you use NIO, then maxConnections should probably be exactly F x B, but maxThreads can be significantly less than that. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTwBmtAAoJEBzwKT+lPKRYmxUQAJRoZCYvmFZWHddPlUETaOt6 2kTX08L19V16W7I5iyEhT8Zrw5mYdGXnA9FzQO0xtl7PMvfY59iIjvN+rapf347+ oFJ73cqaA4oBt7UtaMcLV5+By6CebFp2r/xb9bD0rNZ5DOBhBUJgbGbwLGTeKtAj P3LpaoodwDvWli75VLj4sgHEaAPWybuaLqRpud8W9DkhCT6DVt+RkQj3j9aDxOmu gQRB9qRGKCVExu+j5Tictmd8FHuI2Dx6gC8jsNIoMr1RUePJq835CM2OW6WPP+pa SUzddehtVAET5k4HLbFzviavMf1kBmFxSH616KRtbbZ/vtdqccWEO10GiHuCzBgH 3NlfRAsmCJWhse93zWRwX9p8HGXN2vHFl0cUlkU8yjOmGGPwSBmfE3xXtOFgFio1 yisSkCPTF0Xieyo8UcsxZkHusr/hZ6o1rCaV46zFYVMSLU7j9FDhJS+GMdljGa6y jlWnyHPGte5jbT3LsHvIxC+TBSdrX8nVXjvgV21h5YJqGimokPftHKEkiI48eebW zx5xYAJik5riAruhf+CBVCdOfQfZgIJu0S1WJ4S91HBkTLieaHHnGYuDPcoALYlC mVbyUfCnSVWpUtYVIdxEoxJeuK0jvndA2Knj/Q9D1u9fbu2ZD5heNfM0ed+3taDj ocPfNolGUZE2gMMjbkOk =v25m -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: VERY HIGH TRAFFIC TUNING
doomito wrote: Thank you for the answer. I will begging experimenting on 1 box with this configuration: Connector port=8009 protocol=AJP/1.3 address=127.0.0.1 emptySessionPath=true redirectPort=8443 maxThreads=2048 minSpareThreads=32 connectionTimeout=2 keepAliveTimeout=1 enableLookups=false request.registerRequests=false / I a couple of questions raises from your reply: 1) a couple of years ago we did a BENCHMARK mod_proxy vs mod_jk and the difference was noticeable in favor of mod_jk. There was any improvement on mod_proxy connector in particular? 2) this is definitely an ignorant question, but I see a lot of connections made to 8080 port instead of 8009. That said, I don't fully understand WHY and also, do I need to also tuned this? I have nothing going directly to tomcat, everything goes trough Apache mod_jk. Connector executor=tomcatThreadPool port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / Well, obviously something is going to port 8080 of Tomcat, since you are seeing connections made there. You can configure an AccessLogValve, which will give you a log of the requests that are going there. Is that server directly on the Internet ? If yes, then you probably have (mostly nasty) people trying to connect to that port via HTTP. If you do not want these connections at all, just comment-out that connector above, and restart Tomcat. (But be aware then, that even *you* will not be able to access Tomcat directly with a browser). If you want these connections, but only from the local host e.g., you can also make this connector listen only to the localhost IP address. It will then reject any connections to port 8080 that are /not/ coming from localhost. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: VERY HIGH TRAFFIC TUNING
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Leon, On 7/9/14, 3:16 AM, Leon Rosenberg wrote: On Wed, Jul 9, 2014 at 4:47 AM, Hernán Marsili her...@cmsmedios.com wrote: For the past 4 years we has been working with a 'stable' configuration in which we put APACHE in front of TOMCAT7 (previously Tomcat6) with mod_jk connector. We usually serve high traffic sites with about 7000 to 10.000 concurrent users per box (8gb RAM / 4 vcpu) (50.000 active users total). We are OK with the performance, but sometimes we notice Tomcat stops responding normally while there are at least 2 full CPU left to be consumed (JAVA memory is fine). Hard to tell from here, but dropping performance while still having resources is often an indicator for synchronization issues. You should analyze your thread usage. You could do it with jstack (save multiple stacks in short slots, like every 10 seconds for 2-3 minutes). Check what threads are in what states: - do you have any threads in BLOCKED state? If yes, what they are waiting for? - what are you RUNNABLE threads doing? Are they waiting for something, even not blocked - for example reading the database or reading the incoming request. - is your amount of TIMED_WAITING threads sufficient? If you have non, your thread pool is probably out of threads. This is the configuration we use for the connector: Connector port=8009 protocol=AJP/1.3 address=127.0.0.1 emptySessionPath=true redirectPort=8443 maxThreads=1024 minSpareThreads=32 enableLookups=false request.registerRequests=false / Generally removing apache httpd can increase performance. I assume you have a hardware loadbalancer in front of things, so httpd doesn't do you any good. I would agree in general. Hernán, is there another reason to use httpd in front? I have a couple of questions: 1) should we set a particular connector or let Tomcat7 decide? I understand using protocol=AJP/1.3 the auto-switch kicks in. But, for non-SSL high concurrency sites maybe is best to fixed on APR? Warning, flame war is about to begin, but personally I always found that the plain old java connector is the best option for speed. And the simplest to use too. If you insist of having apache httpd in front, you may want to try mod_proxy (or was it mod_proxy_ajp?) instead. All other things being equal, the BIO (plain old java connector) connector offers the best performance. But in this setup, there are many simultaneous requests. It looks like 5 httpd instances (7-10k connections x 5 ~= 50k connections) and 5 backend Tomcat instances (please correct me if I'm wrong). If every httpd had to talk to every Tomcat, then you will need enough connections/threads on every Tomcat to handle all httpd instances contacting it at once. So, given the default prefork MPM of 250 simultaneous connections, each Tomcat would need to be able to handle 250 x 5 = 1250 connections. With only 1024 threads, stalls can indeed happen. If one uses the APR or NIO connector (still using AJP), reads on incoming connections (e.g. waiting for the next request to come across the wire) are done asynchronously, so you only need a number of threads on your Tomcat instance to support the number of *active requests*, not the number of connections over which active requests could come. The upshot is that you can get away with fewer threads on the Tomcat side and (likely) still serve the same amount of traffic and either eliminate or minimize stalling. 2) how many THREADS can we have? can we go beyond the 1024? Depends on your OS. On modern linuxes sky is the limit. However context switches can kill you too. Your thread stack size + your memory limit will probably limit you before context switches do. If you have very fast connections, got for a smaller amount. If you have keepalive and slow connections, remember that every connection can hold 1-2 threads without doing anything at all. Hmm? But you can go up to 4096 without second thought, if you are really out of threads. However, if your problem are blocked threads or a slow DB, you will make things much much much worse. +1 3) is there any advantage on using processorCache? I've never used that, but the documentation suggests that processorCache = max(maxThreads, max expected simultaneous connections) would be a good value to try. I would try other techniques to improve performance, first. It sounds like this comes down to a memory/GC issue, so it should incrementally improve performance rather than fix something like stalling. 4) We are not defining a CONNECTION TIMEOUT not a KEEP ALIVE. Any advice on this one? The average user session is 7 minutes. If playing with CONNECTION_TIMEOUT, than better on OS level. But again that depends on what is really happening within your application and with the connections. You could monitor it with netstat and see if you have too many CLOSE_WAITs or something. That it's easier to decide what to do.
Re: VERY HIGH TRAFFIC TUNING
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hernán, On 7/9/14, 11:50 AM, doomito wrote: Thank you for the answer. I will begging experimenting on 1 box with this configuration: Connector port=8009 protocol=AJP/1.3 address=127.0.0.1 emptySessionPath=true redirectPort=8443 maxThreads=2048 minSpareThreads=32 connectionTimeout=2 keepAliveTimeout=1 enableLookups=false request.registerRequests=false / Beware about setting timeouts on AJP connectors. I a couple of questions raises from your reply: 1) a couple of years ago we did a BENCHMARK mod_proxy vs mod_jk and the difference was noticeable in favor of mod_jk. There was any improvement on mod_proxy connector in particular? According to recent tests (I just noticed that I hadn't posted the performance slides and data from ApacheConNA2014... I'm working to get the final version of the slides updated at which point I can give you a reference), single-threaded mod_jk and mod_proxy_ajp perform similarly for requests that produce less than 2MiB of response data. Higher than 2MiB of response data sees mod_proxy_ajp's performance go down quite a bit for some reason. For many threads (248) the results are similar except that we did not observe any mod_proxy_ajp performance drop higher than the 2MiB response size. mod_proxy_http is still faster than both up to a point, where the throughput of the connection (i.e. network speed) becomes the limiting factor). If you want to switch connectors, switch from mod_jk/mod_proxy_ajp to mod_proxy_http: the performance of that connector seems to beat them both by a noticeable margin. Note that using mod_proxy_http with SSL on the front-end requires more setup than what you get by default with the AJP-protocol connectors who automatically forward all the SSL information back to Tomcat. 2) this is definitely an ignorant question, but I see a lot of connections made to 8080 port instead of 8009. That said, I don't fully understand WHY and also, do I need to also tuned this? I have nothing going directly to tomcat, everything goes trough Apache mod_jk. Connector executor=tomcatThreadPool port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / That's an HTTP connector, not an AJP one. Perhaps you have some requests using AJP and others proxied using HTTP? What does your httpd configuration look like? You might want to just disable the HTTP connector and see what breaks: if you are going to proxy requests from httpd - Tomcat, you probably only want to use a single protocol. Finally, you should use a shared thread pool (executor) across all your connectors. This will allow you to use a single pool instead of multiple pools which can reduce your resource usage. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTvqVdAAoJEBzwKT+lPKRY1gUQAJPfKu5R/yRfU97lj/rvY7dO AS5P27U4qrc7Zs8HiCVSVgG5GNYKpxhi3qwHINT+c/WBBU5bgLWR5vHqWJ9pF5q8 /chdqdJ7JK7WICbmhLUJA1V5yaSU7kI27nCBaD2QnlC0YhXFj+c3yA6QjEvzg/YY vRGYP3IuKoIP35txm3yILMSunmWw+vbHsTm1idT9G0pCiRIwMDdVOIjTfTZoMB3R yhaSiyrMX4dosCQEXlTUWlhJhN8TFgVYukUJ1TAbAAHDwKZ3ekHTrniLuv4/WMvb 0wagQnxVq17PQ4Q3oC0cWfmLTf2Su7LbI0Z/pOEMvIzfOjsCJTg+pqMv6U1DgnXX Rww2a0foBUJ2unLaUZ7BCQWz9mHWJjjkX/tgWB2B2QBXcPHq5cjuN/yJykDDBNf1 AligGnhbgbLs6mMhjG9MDD/I+NKeFJ/KgGaWAOJ4FpT1CTenqualxe6NPnz3dT7e VG7p8qrTFVqaxd6dKhNT+dGj0voCdCdGORVOK1wC0QKASzIlw6oX4tr6MppayZNF z8JwtyuYkCfKP65UoskuX4u5RaEJJqiLNPs7cEU/qQfnsfHtJmjjrMb0BlaiLYrL 84aY6AayXgNw2mwoz4pHR6bsnmdJkeT8hMAp9jiG3ngxDqvQGlnqTmpvEE7A2psg 1YpYtIl36zAsWF/1gQ3C =ls3u -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: VERY HIGH TRAFFIC TUNING
Hi! I already did some testing. First of all, setting the connectiontimeout was not a good idea :) While the servers seems to be able to handle the same load, the load average went up. It is very very interesting what you mention regarding mod_proxy being faster than mod_jk / mod_proxyAJP. So if I understood correctly, mod_proxy should be a much better option for a high traffic site? even with APACHE in front of it? I came across with article which is interesting. http://www.blakerobertson.com/devlog/2012/2/7/performance-comparison-mod_jk-vs-mod_proxy_ajp-for-apache-22.html Regardind http, we use APACHE since we have quite a lot of rules. We also use mod_deflate, mod_expires and other apache modules. -- View this message in context: http://tomcat.10.x6.nabble.com/VERY-HIGH-TRAFFIC-TUNING-tp5019838p5019923.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: VERY HIGH TRAFFIC TUNING
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hernán, On 7/10/14, 11:19 AM, doomito wrote: Hi! I already did some testing. First of all, setting the connectiontimeout was not a good idea :) While the servers seems to be able to handle the same load, the load average went up. Interesting... load average is a crude measure of activity; I suppose that having those timeouts means that there is activity on a thread even when there is no real work to be done. I do recommend leaving the timeouts set to their defaults (-1 = infinite). It is very very interesting what you mention regarding mod_proxy being faster than mod_jk / mod_proxyAJP. So if I understood correctly, mod_proxy should be a much better option for a high traffic site? even with APACHE in front of it? If you are going to use Apache httpd out in front, our benchmarking suggests that mod_proxy_http is the highest-performing of the 3 connectors (mod_proxy_ajp, mod_proxy_http, and mod_jk). For these tests, we did not configure with a production-quality SSL configuration, which would allow Tomcat to pick-up the client's certificate and all other SSL-related information (e.g. cipher, key size, ssl session id, etc.). You should do your own benchmarking with such a configuration, using your own load profiles. Our testing only performed static file fetches. I came across with article which is interesting. http://www.blakerobertson.com/devlog/2012/2/7/performance-comparison-mod_jk-vs-mod_proxy_ajp-for-apache-22.html That /is/ interesting... the performance testing we did was all on RedHat Linux, and I've found that Linux and Windows have wildly differing performance characteristics, especially when it comes to network-related stuff. It's strange that, in Blake's testing, he saw the JVM use more resources when using mod_proxy_ajp than mod_jk since it's the same data, the same protocol, and the same Java code running in both cases. I think his data smells a little suspicious due to that particular observation. Also, mod_jk is *highly* configurable, while mod_proxy offers fewer options. I believe it's possible to tune mod_jk more finely if necessary. I have always used mod_jk *except* when using httpd on Windows and Mac because of build problems. In those cases (always test rigs, since we are an all-Linux deployment in production), we use mod_proxy_ajp for simplicity of setup. All I can tell you is that you should benchmark your own application in your own environment, and not rely on anyone else's benchmarks to solely guide your decisions. Regarding http, we use APACHE since we have quite a lot of rules. What kind of rules? We also use mod_deflate, mod_expires and other apache modules. If you are getting a benefit from Apache httpd, then use it. If you are concerned about performance, you might want to look at options for removing it. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1 Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQIcBAEBCAAGBQJTvuQvAAoJEBzwKT+lPKRYksQP/1SN9eDDrAVBXjvL85iU68Dd 2lLK1hh170lAFMmcv1EU91oQp+Au8O8U1tI4yV5ujAR5m9yag8u5uluPAaaSrmyS TFpC7H4tXuQH+PhAz8HDNKIFTrDRInfCTMTugnZi/JpaK3GcNxB/aNvd3KAPqkTd LXpURvjmP0SelBLe7gaLCrxhCK3i6fmOCjiBxTOpMkzZJKibdOXGp+U80liHLvOj iaRHdOLIwYv7ORbGQRV5q3eu9KPxKcPg0O3YvtHiNEFZzz78kuINOXLocnV8FKWH KRzD7cn3Wfhd59+yyIph8ZXsbgMjnbx77koHvJZxU5OusRXCL4QtqsADN78uruW6 CxlCHymf33VkxoUsCdIPQCmCk4DnqxvpLggGFuR7wnJfVnHnVvzsw7EIKrt/rokW 7EZxUOunPvph8wyV1809TZUfM7fzIAQQZU6ODeBdT34tdwWl/gpdXji2pohsgthk v5atPhDFXj1/NAKUZ/8xaefF3MwejectbaMng5zaFlrXff+ioXvwXzV7Z5BNybzS SpisBKRjVc6BslE3xaMcw77TbaZWMgPFs4YX1jwVDorQwAIjGk2P1vYq1gBUB3Ta f/UMqcd1p8okeEW+XB3wP+YRM816TKKDZ0odrJdg95l68Qwde4340KGG4eTvYXdi 70Xodbrsl+2auGKuVALX =okxr -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: VERY HIGH TRAFFIC TUNING
answering only to the one directed at me (or so I think): On Thu, Jul 10, 2014 at 4:09 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Leon, If you have very fast connections, go for a smaller amount. If you have keepalive and slow connections, remember that every connection can hold 1-2 threads without doing anything at all. Hmm? If you have keepalive enabled, then every client will held an open connection for some period of time. Depending on browser brand and version, the browser will typically hold 1-2 connections to the server. So for every logged in users that is clicking or issuing a request regularly (maybe once a minute or so), you can end up having one or two threads, dependending on how many connections the browsers holds. Those threads would be idle most of the time, but still not available to your threadpool. Usually they show up as Runnable somewhere in socket.read in the thread dump. So if you have 100 users with 150 connections and 151 threads in your threadpool you can end up with a completely blocked server with zero cpu usage. regards Leon
Re: VERY HIGH TRAFFIC TUNING
Christopher Schultz wrote: ... Interesting... load average is a crude measure of activity; I suppose that having those timeouts means that there is activity on a thread even when there is no real work to be done. I do recommend leaving the timeouts set to their defaults (-1 = infinite). In general terms, I would definitely not put the connectionTimeout nor the keepAliveTimeout to infinite, if that is what you meant here. ConnectionTimeout infinite seems like a perfect setup for a DOS attack. Keep-alive timeout infinite seems like the perfect way to block a lot of threads doing nothing (and opening yourself to another kind of DOS attack). However, in this case, we are talking about the AJP Connector, which processes requests coming in via Apache httpd and mod_jk, so I guess that one can rely on the Apache front-end not to relay anything nasty to Tomcat. Presumably, the Apache httpd configuration does not have infinite connection timeout nor keep-alive timeout. Which in a way, raises the question of why these parameters are even available for setting on the AJP Connector. Should these not better be left to the discretion of Apache httpd and mod_jk in the first place ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: VERY HIGH TRAFFIC TUNING
On Wed, Jul 9, 2014 at 4:47 AM, Hernán Marsili her...@cmsmedios.com wrote: Hi, Hello Hernán, For the past 4 years we has been working with a 'stable' configuration in which we put APACHE in front of TOMCAT7 (previously Tomcat6) with mod_jk connector. We usually serve high traffic sites with about 7000 to 10.000 concurrent users per box (8gb RAM / 4 vcpu) (50.000 active users total). We are OK with the performance, but sometimes we notice Tomcat stops responding normally while there are at least 2 full CPU left to be consumed (JAVA memory is fine). Hard to tell from here, but dropping performance while still having resources is often an indicator for synchronization issues. You should analyze your thread usage. You could do it with jstack (save multiple stacks in short slots, like every 10 seconds for 2-3 minutes). Check what threads are in what states: - do you have any threads in BLOCKED state? If yes, what they are waiting for? - what are you RUNNABLE threads doing? Are they waiting for something, even not blocked - for example reading the database or reading the incoming request. - is your amount of TIMED_WAITING threads sufficient? If you have non, your thread pool is probably out of threads. This is the configuration we use for the connector: Connector port=8009 protocol=AJP/1.3 address=127.0.0.1 emptySessionPath=true redirectPort=8443 maxThreads=1024 minSpareThreads=32 enableLookups=false request.registerRequests=false / Generally removing apache httpd can increase performance. I assume you have a hardware loadbalancer in front of things, so httpd doesn't do you any good. I have a couple of questions: 1) should we set a particular connector or let Tomcat7 decide? I understand using protocol=AJP/1.3 the auto-switch kicks in. But, for non-SSL high concurrency sites maybe is best to fixed on APR? Warning, flame war is about to begin, but personally I always found that the plain old java connector is the best option for speed. And the simplest to use too. If you insist of having apache httpd in front, you may want to try mod_proxy (or was it mod_proxy_ajp?) instead. 2) how many THREADS can we have? can we go beyond the 1024? Depends on your OS. On modern linuxes sky is the limit. However context switches can kill you too. If you have very fast connections, got for a smaller amount. If you have keepalive and slow connections, remember that every connection can hold 1-2 threads without doing anything at all. But you can go up to 4096 without second thought, if you are really out of threads. However, if your problem are blocked threads or a slow DB, you will make things much much much worse. 3) is there any advantage on using processorCache? 4) We are not defining a CONNECTION TIMEOUT not a KEEP ALIVE. Any advice on this one? The average user session is 7 minutes. If playing with CONNECTION_TIMEOUT, than better on OS level. But again that depends on what is really happening within your application and with the connections. You could monitor it with netstat and see if you have too many CLOSE_WAITs or something. That it's easier to decide what to do. Thanks for the help! Hernán. Leon. P.S. needless to say, but having a monitoring tool like http://www.moskito.org will help either ;-)
Re: VERY HIGH TRAFFIC TUNING
Hi. This kind of question is very difficult to answer reasonably, for anyone that is not you and does not have direct access to your system, to see what happens, when it happens. The general and reasonable answer would be that you need to use some monitoring tools, to find out where exactly the bottleneck is, and then, very carefully, start tuning your system one parameter at a time, to try to improve the situation. The worst thing to do is to start changing multiple things without really knowing what they are doing, because then you will get very confused very quickly. Ah, one more thing : the default parameters of both Apache httpd and Tomcat, are chosen by people who know what they are doing, to cover a majority of reasonable cases. So, changing these parameters without knowing exactly what each parameters does, and how different parameters interact with one another, is always risky. (This being said, the mere fact that you are asking on this list before doing that, is a bonus point for you). All this being said, a couple of notes below : Hernán Marsili wrote: Hi, For the past 4 years we has been working with a 'stable' configuration in which we put APACHE in front of TOMCAT7 (previously Tomcat6) with mod_jk connector. We usually serve high traffic sites with about 7000 to 10.000 concurrent users per box (8gb RAM / 4 vcpu) (50.000 active users total). So, basically, your configuration is fine and has been running reliably and in a stable way for 4 years, including a Tomcat version change. Good. One more reason for only changing things carefully. We are OK with the performance, but sometimes we notice Tomcat stops responding normally while there are at least 2 full CPU left to be consumed (JAVA memory is fine). You may want to indicate a bit more precisely what you mean by sometimes and by stops responding normally. This is the configuration we use for the connector: Connector port=8009 protocol=AJP/1.3 address=127.0.0.1 emptySessionPath=true redirectPort=8443 maxThreads=1024 minSpareThreads=32 enableLookups=false request.registerRequests=false / Note # 1 : you say that you have up to 10,000 concurrent users. Yet, there are only 1024 Threads in Tomcat. Users is not necessarily equal to requests, but let's assume for a moment that they are. Basically, Tomcat will use one Thread to process one HTTP request, from the time the request is received, to the time when the response to that request has been sent back to the user. So, maybe, there are times when your Tomcat is running out of available Threads to process all the requests that come in at some moments ? If that is the case, what will happen is : - The TCP/IP stack on Tomcat will accept the client connection - but this connection will be put in a queue, waiting for a thread to become available (iow : when it finishes the current request that it is processing). If all 1024 threads are currently busy processing requests (or waiting for additional requests from the same client, because of the keep-alive timeout, see below), then it will appear for the client as if Tomcat is not responding normally. I have a couple of questions: 1) should we set a particular connector or let Tomcat7 decide? I understand using protocol=AJP/1.3 the auto-switch kicks in. But, for non-SSL high concurrency sites maybe is best to fixed on APR? I cannot answer that and will wait for someone else more qualified to do that. 2) how many THREADS can we have? can we go beyond the 1024? Yes, there is no limit other than the available memory and the general performance of the machine. That is also a very easy parameter to change, and one that does not have a lot of obscure side-effects. Apart from everything else, I would suggest raising it to e.g. 4096, and see what happens. (As someone else commented however : if the problem is not really in Tomcat, but in some back-end database server, then this will make things worse). 3) is there any advantage on using processorCache? don't know. 4) We are not defining a CONNECTION TIMEOUT not a KEEP ALIVE. Any advice on this one? The average user session is 7 minutes. I do not rememember what the default value is for keep-alive. But this may also be something easy to adjust, and with potentially big effects. If it is currently anywhere larger than some 5 seconds, change it to 5 seconds maximum. The keep-alive logic was introduced at a time when networks were slower, and when setting up a new TCP/IP connection between a client and a server for each new request, was quite expensive. So the idea was : after a first request by a client, let's leave the connection open, to see if this client has more requests to send, over that same connection, within just a few seconds. This way, we avoid closing the connection each time, and have to re-create a connection for each request. (Example : a HTML page, with a lot of img tags in it.). Unfortunately, what happens in a
Re: VERY HIGH TRAFFIC TUNING
Thank you for the answer. I will begging experimenting on 1 box with this configuration: Connector port=8009 protocol=AJP/1.3 address=127.0.0.1 emptySessionPath=true redirectPort=8443 maxThreads=2048 minSpareThreads=32 connectionTimeout=2 keepAliveTimeout=1 enableLookups=false request.registerRequests=false / I a couple of questions raises from your reply: 1) a couple of years ago we did a BENCHMARK mod_proxy vs mod_jk and the difference was noticeable in favor of mod_jk. There was any improvement on mod_proxy connector in particular? 2) this is definitely an ignorant question, but I see a lot of connections made to 8080 port instead of 8009. That said, I don't fully understand WHY and also, do I need to also tuned this? I have nothing going directly to tomcat, everything goes trough Apache mod_jk. Connector executor=tomcatThreadPool port=8080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 / -- View this message in context: http://tomcat.10.x6.nabble.com/VERY-HIGH-TRAFFIC-TUNING-tp5019838p5019875.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
VERY HIGH TRAFFIC TUNING
Hi, For the past 4 years we has been working with a 'stable' configuration in which we put APACHE in front of TOMCAT7 (previously Tomcat6) with mod_jk connector. We usually serve high traffic sites with about 7000 to 10.000 concurrent users per box (8gb RAM / 4 vcpu) (50.000 active users total). We are OK with the performance, but sometimes we notice Tomcat stops responding normally while there are at least 2 full CPU left to be consumed (JAVA memory is fine). This is the configuration we use for the connector: Connector port=8009 protocol=AJP/1.3 address=127.0.0.1 emptySessionPath=true redirectPort=8443 maxThreads=1024 minSpareThreads=32 enableLookups=false request.registerRequests=false / I have a couple of questions: 1) should we set a particular connector or let Tomcat7 decide? I understand using protocol=AJP/1.3 the auto-switch kicks in. But, for non-SSL high concurrency sites maybe is best to fixed on APR? 2) how many THREADS can we have? can we go beyond the 1024? 3) is there any advantage on using processorCache? 4) We are not defining a CONNECTION TIMEOUT not a KEEP ALIVE. Any advice on this one? The average user session is 7 minutes. Thanks for the help! Hernán.
