On Wed, Jul 9, 2014 at 4:47 AM, Hernán Marsili <her...@cmsmedios.com> wrote:
> Hi, > Hello Hernán, > > For the past 4 years we has been working with a 'stable' configuration in > which we put APACHE in front of TOMCAT7 (previously Tomcat6) with mod_jk > connector. We usually serve high traffic sites with about 7000 to 10.000 > concurrent users per box (8gb RAM / 4 vcpu) (50.000 active users total). > > We are OK with the performance, but sometimes we notice Tomcat stops > responding normally while there are at least 2 full CPU left to be consumed > (JAVA memory is fine). > Hard to tell from here, but dropping performance while still having resources is often an indicator for synchronization issues. You should analyze your thread usage. You could do it with jstack (save multiple stacks in short slots, like every 10 seconds for 2-3 minutes). Check what threads are in what states: - do you have any threads in BLOCKED state? If yes, what they are waiting for? - what are you RUNNABLE threads doing? Are they waiting for something, even not blocked - for example reading the database or reading the incoming request. - is your amount of TIMED_WAITING threads sufficient? If you have non, your thread pool is probably out of threads. > > This is the configuration we use for the connector: > > <Connector port="8009" protocol="AJP/1.3" address="127.0.0.1" > emptySessionPath="true" redirectPort="8443" maxThreads="1024" > minSpareThreads="32" enableLookups="false" request.registerRequests="false" > /> > Generally removing apache httpd can increase performance. I assume you have a hardware loadbalancer in front of things, so httpd doesn't do you any good. > > I have a couple of questions: > 1) should we set a particular connector or let Tomcat7 decide? I understand > using protocol="AJP/1.3" the auto-switch kicks in. But, for non-SSL high > concurrency sites maybe is best to fixed on APR? > Warning, flame war is about to begin, but personally I always found that the plain old java connector is the best option for speed. And the simplest to use too. If you insist of having apache httpd in front, you may want to try mod_proxy (or was it mod_proxy_ajp?) instead. > > 2) how many THREADS can we have? can we go beyond the 1024? > Depends on your OS. On modern linuxes sky is the limit. However context switches can kill you too. If you have very fast connections, got for a smaller amount. If you have keepalive and slow connections, remember that every connection can hold 1-2 threads without doing anything at all. But you can go up to 4096 without second thought, if you are really out of threads. However, if your problem are blocked threads or a slow DB, you will make things much much much worse. > > 3) is there any advantage on using processorCache? > > 4) We are not defining a CONNECTION TIMEOUT not a KEEP ALIVE. Any advice on > this one? The average user session is 7 minutes. > If playing with CONNECTION_TIMEOUT, than better on OS level. But again that depends on what is really happening within your application and with the connections. You could monitor it with netstat and see if you have too many CLOSE_WAITs or something. That it's easier to decide what to do. > > Thanks for the help! > Hernán. > Leon. P.S. needless to say, but having a monitoring tool like http://www.moskito.org will help either ;-)
