On Jan 27, 2008 4:32 PM, Clay Collier <[EMAIL PROTECTED]> wrote: > Hi all- > > I'm running into a recurring problem with an application that uses > DBCP pooling to access a database. The application initially seems to > run fine; in the test environment, it has no problem dealing with a > large volume of connections and responds to everything the way that > you would expect. However, on my production server exposed to users, > the application runs fine for an indeterminate period of time, and > then begins locking up. Users send messages to the server, but never > get any response back- not even an exception. The manager's server > status show lots of long-running threads sitting around for long > periods of time. Load on the Tomcat server and the database server > are not a factor. Looking at the thread dump seems to give this as > the relevant bits (this from a dump with two stuck threads): > > "http-8080-Processor25" daemon prio=10 tid=0x08a1c800 nid=0x4cd5 in > Object.wait() [0xb4d3b000..0xb4d3c030] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x8cf5c6c8> (a > org.apache.tomcat.dbcp.pool.impl.GenericObjectPool) > at java.lang.Object.wait(Object.java:485) > at > org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:810) > - locked <0x8cf5c6c8> (a > org.apache.tomcat.dbcp.pool.impl.GenericObjectPool) > at > org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96) > at > org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880) > at > edu.stanford.hpn.servlet.NetPlumberServlet.doPost(NetPlumberServlet.java:76) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:710) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874) > at > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) > at > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) > at > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) > at java.lang.Thread.run(Thread.java:619) > > "http-8080-Processor24" daemon prio=10 tid=0x08a1b800 nid=0x4cd4 in > Object.wait() [0xb4d8c000..0xb4d8d0b0] > java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > - waiting on <0x8cf5c6c8> (a > org.apache.tomcat.dbcp.pool.impl.GenericObjectPool) > at java.lang.Object.wait(Object.java:485) > at > org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:810) > - locked <0x8cf5c6c8> (a > org.apache.tomcat.dbcp.pool.impl.GenericObjectPool) > at > org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:96) > at > org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection(BasicDataSource.java:880) > at > edu.stanford.hpn.servlet.TopologyActivator.performAction(TopologyActivator.java:85) > at > edu.stanford.hpn.servlet.NetPlumberServlet.doPost(NetPlumberServlet.java:99) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:710) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) > at > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213) > at > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174) > at > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) > at > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874) > at > org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) > at > org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) > at > org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) > at > org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689) > at java.lang.Thread.run(Thread.java:619) > > All of the threads that become non-responsive are waiting on > getConnection(), and these two threads are the only ones waiting on or > locking 0x8cf5c6c8. So I took a look at my pooling setup: > <Resource driverClassName="com.mysql.jdbc.Driver" name="jdbc/ProdDB" > password="(removed)" type="javax.sql.DataSource" > url="jdbc:mysql://vns-app-3.stanfor > d.edu:3306/netplumber" username="tomcat" maxActive="99" maxIdle="15" > maxWait="10000" removeAbandoned="true" logAbandoned="true" > removeAbandonedTimeout="60" minIdle="5" initialSize="5" > timeBetweenEvictionRuns="20000" validationQuery="SELECT id FROM > account WHERE id = 1" testOnBorrow="true" testOnReturn="true" test > WhileIdle="true"/> > > We don't seem to be reaching the maxActive cap- I've tried setting > this to -1, which makes no difference, and tried intentionally > exhausting the thread pool in the test environment, which results in > the expected timeout exception rather than things being stuck. I've > looked for connection leaks but can't locate any- plus, it seems like > the removeAbandoned would take care of that, and I'm not seeing any > abandoned connections logged by Tomcat. The MySQL server is not > running out of connections, nor are the pool connections being idled > out- the app isn't idle nearly long enough for MySQL's idle disconnect > to come into play. > > As far as the servlet code goes, most of my servlets inherit from a > common ancestor called NetplumberServlet that sets up the Datasource > during the init() method. NetplumberServlet implements doPost, which > gets a connection, does a quick lookup, and then closes the > connection, and then calls performAction, which is implemented by the > child classes. performAction gets its own connection from the pool, > and returns it when processing is done. > > I've tried a lot of different things here regarding pool options, > etc., and nothing seems to make a difference. Any ideas as to what > I'm missing here?
What version of Tomcat are you running? If (as is likely currently) it is one that bundles Commons Pool 1.3, there is something funny going on because the line number where wait on borrowObject is occurring in the thread dumps would indicate that a) the pool is exhausted and b) the maxWait parameter is set at the default, which is to wait indefinitely Is it possible that your production config is missing maxWait? Phil --------------------------------------------------------------------- To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
