On 04/10/17 13:51, TurboChargedDad . wrote: > Hello all.. > I am going to do my best to describe my problem. Hopefully someone will > have some sort of insight. > > Tomcat 7.0.41 (working on updating that) > Java 1.6 (Working on getting this updated to the latest minor release) > RHEL Linux > > I inherited an opti-tenant setup. Individual user accounts on the system > each have their own Tomcat instance, each is started using sysinit. This > is done to keep each website in its own permissible world so one website > can't interfere with a others data. > > There are two load balanced apache proxies at the edge that point to one > Tomcat server (I know I know but again I inherited this) > > Apache lays over the top of tomcat to terminate SSL and uses AJP to > proxypass to each tomcat instance based on the users assigned port. > > Things have run fine for years (so I am being told anyway) until recently. > Let me give an example of an outage. > > User1, user2 and user3 all use unique databases on a shared database > server, SQL server 10. > > User 4 runs on a windows jboss server and also has a database on shared > database server 10. > > Users 5-50 all run in the mentioned Linux server using tomcat and have > databases on *other* various shared databases servers but have nothing to > do with database server 10. > > User 4 had a stored proc go wild on database server 10 basically knocking > it offline. > > Now one would expect sites 1-4 to experience interruption of service > because they use a shared DBMS platform. However. > > Every single site goes down. I monitor the connections for each site with a > custom tool. When this happens, the connections start stacking up across > all the components. (Proxies all the way through the stack) > Looking at the AJP connection pool threads for user 9 shows that user has > exhausted their AJP connection pool threads. They are maxed out at 300 yet > that user doesn't have high activity at all. The CPU load, memory usage and > traffic for everything except SQL server 10 is stable during this outrage. > The proxies start consuming more and more memory the longer the outrage > occurs but that's expected as the connection counts stack up into the > thousands. After a short time all the sites apache / ssl termination later > start throwing AJP timeout errors. Shortly after that the edge proxies > will naturally also starting throwing timeout errors of their own. > > I am only watching user 9 using a tool that allows me to have insight into > what's going on using JMX metrics but I suspect that once I get all the > others instrumented that I will see the same thing. Maxed out AJP > connection pools. > > Aren't those supposed to be unique per user/ JVM? Am I missing something in > the docs? > > Any assistance from the tomcat gods is much appreciated.
TL;DR - Try switching to the NIO AJP connector on Tomcat. Take a look at this session I just uploaded from TomcatCon London last week. You probably want to start around 35:00 and the topic of thread exhaustion. HTH, Mark P.S. The other sessions we have are on the way. I plan to update the site and post links once I have them all uploaded. --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
