I've some similar experiences with c3p0 (although only in test environment), somewhere theres some extra settings about wait time and pool size, you can google it.. Im in the midst of upgrading to ubuntu so I do not have my source available..
You could also try to switch to a more maintained connection pool... 2009/4/15 Serkan Camurcuoglu <[email protected]>: > if all your threads are waiting for connections with no thread using a db > connection, I suspect that you either leak connections (i.e. forget to free > some connections) or each thread uses (requires) multiple connections during > a transaction and at some point all threads need at least one more db > connection to continue.. > > > > J wrote: >> >> Not just one thread, but that thread dump part (from my first post) is >> repeated for 149 other threads have the same situation. ( because of >> tomcats >> 150 maxthreads). >> >> But when looking at the db during a freeze, there is no lock on any table >> at >> db level. MySQL Administrator shows that there are 15 threads during the >> freeze (equal to c3p0's maxPoolSize), all sleeping. >> >> I have now added some C3P0 debug and workaround options to see if I can >> get >> some extra info out of it. >> >> http://www.mchange.com/projects/c3p0/index.html#configuring_to_debug_and_wor >> karound_broken_clients >> (debugUnreturnedConnectionStackTraces true and unreturnedConnectionTimeout >> 180) >> >> >> -----Oorspronkelijk bericht----- >> Van: John Krasnay [mailto:[email protected]] Verzonden: woensdag 15 april >> 2009 15:57 >> Aan: [email protected] >> Onderwerp: Re: WebApp Freezes >> >> Sounds like you have a thread holding a lock on a critical table and >> subsequent threads are lining up behind it waiting for it to finish. You >> should check your MySQL to try and figure out who's holding the lock and >> why. >> >> Note that the culprit thread need not be hung up in the database. Locks >> are held until the transaction commits, so your thread could be hung up >> in app code after making a database update but before the transaction is >> committed. >> >> jk >> >> On Wed, Apr 15, 2009 at 03:41:07PM +0200, J wrote: >> >>> >>> (Sorry for the empty message. First I tried Gmail, but that doesn't work >>> with this mailing list. Then I tried GMX webclient, but that client >>> always >>> sends as HTML, which probably caused the message to be stripped to >>> empty.) >>> >>> >>> I'm experiencing freezes on a production website. >>> >>> Server specs: >>> -OS: Linux CentOS >>> -Webserver: Tomcat 6 >>> -MySQL 5 with default settings >>> -Frameworks: Wicket 1.4-rc2 (webframework), Spring 2.5.6, Hibernate >>> 3.3.1.GA, C3P0 db pooling 0.9.1.2, URLRewriteFilter 3.1.0, Spring >>> OpenSessionInViewFilter >>> Problem: >>> At some point (about 5 to 24 hours) after a boot, Tomcat seems to stop >>> serving requests, although it does create new threads for new incoming >>> connections. But since it does not serve those connections, the new >>> connections will show and keep showing 0kb transfer. Shortly after that, >>> max-thread is reached, so then I'm unable to access any webapp (the main >>> website + tomcat manager) running on that Tomcat server. I'm not sure if >>> >> >> it >> >>> >>> is able to serve static resources in the short time window where it still >>> has some threads free, because the time window is too short to notice >>> when >>> it happens. >>> >>> Observations: >>> -No exceptions or errors in the catalina logs. So no memory problems, >>> >> >> since >> >>> >>> no error occurs in the logs >>> -Java Thread dump using command: kill -QUIT (see output below), shows the >>> text "locked" and "WAITING". >>> -Adding c3p0 db pooling idle checks, tests and timeout settings (see >>> >> >> below) >> >>> >>> did not help. >>> -Using Mysql Administrator GUI shows that after a freeze, there will be >>> 15 >>> threads, all sleeping. Normally this would return to a minPoolSize of 5, >>> >> >> if >> >>> >>> I'm correct, but thats not the case. >>> -The freeze continues for hours, and does not recover. I have to restart >>> Tomcat. >>> >>> >>> I'm suspecting that it has something to do with DB pooling. I'm not sure >>> >> >> if >> >>> >>> "locked" and "WAITING" are normal behaviour. But if there is something >>> wrong, why doesn't c3p0 recover from it? >>> >>> Can somebody shed some light on this? :) >>> >>> >>> >>> >>> >>> Configuration: >>> >>> === DB pooling in Spring applicationContext.xml ==== >>> <bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource"> >>> <property >>> name="driverClass"><value>com.mysql.jdbc.Driver</value></property> >>> <property name="jdbcUrl" value="jdbc:mysql://localhost/xxxxxx" /> >>> <property name="user"><value>xxxxxxxx</value></property> >>> <property name="password"><value>xxxxxxxxx</value></property> >>> >>> <property name="maxConnectionAge"><value>600</value></property> >>> <property >>> name="idleConnectionTestPeriod"><value>180</value></property> >>> <property >>> name="testConnectionOnCheckin"><value>true</value></property> >>> >>> <property name="acquireIncrement"><value>5</value></property> >>> <property name="maxIdleTime"><value>180</value></property> >>> <property name="maxPoolSize"><value>15</value></property> >>> <property name="maxStatements"><value>100</value></property> >>> <property name="minPoolSize"><value>5</value></property> >>> </bean> >>> =========================== >>> >>> >>> ===== Thread dump part =========== >>> "http-8080-28" daemon prio=10 tid=0xb4fbb400 nid=0x4ae2 in Object.wait() >>> [0xb4316000..0xb4318130] >>> java.lang.Thread.State: WAITING (on object monitor) >>> at java.lang.Object.wait(Native Method) >>> at >>> >>> >> >> >> com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePo >> >>> >>> ol.java:1315) >>> at >>> >>> >> >> >> com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicRe >> >>> >>> sourcePool.java:557) >>> - locked <0x72c74b10> (a >>> com.mchange.v2.resourcepool.BasicResourcePool) >>> at >>> >>> >> >> >> com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResource >> >>> >>> Pool.java:477) >>> at >>> >>> >> >> >> com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C >> >>> >>> 3P0PooledConnectionPool.java:525) >>> at >>> >>> >> >> >> com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(Abstract >> >>> >>> PoolBackedDataSource.java:128) >>> at >>> >>> >> >> >> org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConn >> >>> >>> ection(LocalDataSourceConnectionProvider.java:81) >>> at >>> >>> >> >> >> org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:4 >> >>> >>> 23) >>> at >>> >>> >> >> >> org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:14 >> >>> >>> 4) >>> at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119) >>> at >>> org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57) >>> at >>> org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326) >>> at >>> >>> >> >> >> org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(Hiber >> >>> >>> nateTransactionManager.java:510) >>> at >>> >>> >> >> >> org.springframework.transaction.support.AbstractPlatformTransactionManager.g >> >>> >>> etTransaction(AbstractPlatformTransactionManager.java:350) >>> at >>> >>> >> >> >> org.springframework.transaction.interceptor.TransactionAspectSupport.createT >> >>> >>> ransactionIfNecessary(TransactionAspectSupport.java:262) >>> at >>> >>> >> >> >> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(Tr >> >>> >>> ansactionInterceptor.java:101) >>> at >>> >>> >> >> >> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect >> >>> >>> iveMethodInvocation.java:171) >>> at >>> >>> >> >> >> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro >> >>> >>> xy.java:204) >>> at $Proxy37.findById(Unknown Source) >>> at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source) >>> at >>> >>> >> >> >> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl >> >>> >>> .java:25) >>> at java.lang.reflect.Method.invoke(Method.java:597) >>> at >>> >>> >> >> >> org.apache.wicket.proxy.LazyInitProxyFactory$JdkHandler.invoke(LazyInitProxy >> >>> >>> Factory.java:416) >>> at org.apache.wicket.proxy.$Proxy2.findById(Unknown Source) >>> at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> at >>> >>> >> >> >> org.apache.wicket.model.LoadableDetachableModel.getObject(LoadableDetachable >> >>> >>> Model.java:119) >>> at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx >>> at sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown >>> Source) >>> at >>> >>> >> >> >> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc >> >>> >>> torAccessorImpl.java:27) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:513) >>> at >>> >>> >> >> >> org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java >> >>> >>> :151) >>> at >>> >>> >> >> >> org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java >> >>> >>> :89) >>> at >>> >>> >> >> >> org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.new >> >>> >>> Page(BookmarkablePageRequestTarget.java:306) >>> at >>> >>> >> >> >> org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.get >> >>> >>> Page(BookmarkablePageRequestTarget.java:321) >>> at >>> >>> >> >> >> org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.pro >> >>> >>> cessEvents(BookmarkablePageRequestTarget.java:234) >>> at >>> >>> >> >> >> org.apache.wicket.request.AbstractRequestCycleProcessor.processEvents(Abstra >> >>> >>> ctRequestCycleProcessor.java:92) >>> at >>> >>> >> >> >> org.apache.wicket.RequestCycle.processEventsAndRespond(RequestCycle.java:119 >> >>> >>> 2) >>> at org.apache.wicket.RequestCycle.step(RequestCycle.java:1271) >>> at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1370) >>> at org.apache.wicket.RequestCycle.request(RequestCycle.java:501) >>> at >>> org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:455) >>> at >>> >>> >> >> >> org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:288) >> >>> >>> at >>> >>> >> >> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application >> >>> >>> FilterChain.java:235) >>> at >>> >>> >> >> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh >> >>> >>> ain.java:206) >>> at >>> >>> >> >> >> org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterI >> >>> >>> nternal(OpenSessionInViewFilter.java:198) >>> at >>> >>> >> >> >> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF >> >>> >>> ilter.java:76) >>> at >>> >>> >> >> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application >> >>> >>> FilterChain.java:235) >>> at >>> >>> >> >> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh >> >>> >>> ain.java:206) >>> at >>> >>> >> >> >> org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:164 >> >>> >>> ) >>> at >>> org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:141) >>> at >>> >>> >> >> >> org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.jav >> >>> >>> a:90) >>> at >>> >>> >> >> >> org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter >> >>> >>> .java:417) >>> at >>> >>> >> >> >> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application >> >>> >>> FilterChain.java:235) >>> at >>> >>> >> >> >> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh >> >>> >>> ain.java:206) >>> at >>> >>> >> >> >> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja >> >>> >>> va:233) >>> at >>> >>> >> >> >> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja >> >>> >>> va:191) >>> at >>> >>> >> >> >> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128 >> >>> >>> ) >>> at >>> >>> >> >> >> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102 >> >>> >>> ) >>> at >>> >>> >> >> >> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java >> >>> >>> :109) >>> at >>> >>> >> >> >> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286) >> >>> >>> at >>> >>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845) >>> at >>> >>> >> >> >> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http >> >>> >>> 11Protocol.java:583) >>> at >>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447) >>> at java.lang.Thread.run(Thread.java:619) >>> ======================================= >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
