(Sorry for the empty message. First I tried Gmail, but that doesn't work
with this mailing list. Then I tried GMX webclient, but that client always
sends as HTML, which probably caused the message to be stripped to empty.)
I'm experiencing freezes on a production website.
Server specs:
-OS: Linux CentOS
-Webserver: Tomcat 6
-MySQL 5 with default settings
-Frameworks: Wicket 1.4-rc2 (webframework), Spring 2.5.6, Hibernate
3.3.1.GA, C3P0 db pooling 0.9.1.2, URLRewriteFilter 3.1.0, Spring
OpenSessionInViewFilter
Problem:
At some point (about 5 to 24 hours) after a boot, Tomcat seems to stop
serving requests, although it does create new threads for new incoming
connections. But since it does not serve those connections, the new
connections will show and keep showing 0kb transfer. Shortly after that,
max-thread is reached, so then I'm unable to access any webapp (the main
website + tomcat manager) running on that Tomcat server. I'm not sure if it
is able to serve static resources in the short time window where it still
has some threads free, because the time window is too short to notice when
it happens.
Observations:
-No exceptions or errors in the catalina logs. So no memory problems, since
no error occurs in the logs
-Java Thread dump using command: kill -QUIT (see output below), shows the
text "locked" and "WAITING".
-Adding c3p0 db pooling idle checks, tests and timeout settings (see below)
did not help.
-Using Mysql Administrator GUI shows that after a freeze, there will be 15
threads, all sleeping. Normally this would return to a minPoolSize of 5, if
I'm correct, but thats not the case.
-The freeze continues for hours, and does not recover. I have to restart
Tomcat.
I'm suspecting that it has something to do with DB pooling. I'm not sure if
"locked" and "WAITING" are normal behaviour. But if there is something
wrong, why doesn't c3p0 recover from it?
Can somebody shed some light on this? :)
Configuration:
=== DB pooling in Spring applicationContext.xml ====
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property
name="driverClass"><value>com.mysql.jdbc.Driver</value></property>
<property name="jdbcUrl" value="jdbc:mysql://localhost/xxxxxx" />
<property name="user"><value>xxxxxxxx</value></property>
<property name="password"><value>xxxxxxxxx</value></property>
<property name="maxConnectionAge"><value>600</value></property>
<property
name="idleConnectionTestPeriod"><value>180</value></property>
<property
name="testConnectionOnCheckin"><value>true</value></property>
<property name="acquireIncrement"><value>5</value></property>
<property name="maxIdleTime"><value>180</value></property>
<property name="maxPoolSize"><value>15</value></property>
<property name="maxStatements"><value>100</value></property>
<property name="minPoolSize"><value>5</value></property>
</bean>
===========================
===== Thread dump part ===========
"http-8080-28" daemon prio=10 tid=0xb4fbb400 nid=0x4ae2 in Object.wait()
[0xb4316000..0xb4318130]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePo
ol.java:1315)
at
com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicRe
sourcePool.java:557)
- locked <0x72c74b10> (a
com.mchange.v2.resourcepool.BasicResourcePool)
at
com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResource
Pool.java:477)
at
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C
3P0PooledConnectionPool.java:525)
at
com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(Abstract
PoolBackedDataSource.java:128)
at
org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConn
ection(LocalDataSourceConnectionProvider.java:81)
at
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:4
23)
at
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:14
4)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
at
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
at
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
at
org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(Hiber
nateTransactionManager.java:510)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.g
etTransaction(AbstractPlatformTransactionManager.java:350)
at
org.springframework.transaction.interceptor.TransactionAspectSupport.createT
ransactionIfNecessary(TransactionAspectSupport.java:262)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(Tr
ansactionInterceptor.java:101)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:171)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)
at $Proxy37.findById(Unknown Source)
at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.wicket.proxy.LazyInitProxyFactory$JdkHandler.invoke(LazyInitProxy
Factory.java:416)
at org.apache.wicket.proxy.$Proxy2.findById(Unknown Source)
at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
at
org.apache.wicket.model.LoadableDetachableModel.getObject(LoadableDetachable
Model.java:119)
at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
at sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
torAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java
:151)
at
org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java
:89)
at
org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.new
Page(BookmarkablePageRequestTarget.java:306)
at
org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.get
Page(BookmarkablePageRequestTarget.java:321)
at
org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.pro
cessEvents(BookmarkablePageRequestTarget.java:234)
at
org.apache.wicket.request.AbstractRequestCycleProcessor.processEvents(Abstra
ctRequestCycleProcessor.java:92)
at
org.apache.wicket.RequestCycle.processEventsAndRespond(RequestCycle.java:119
2)
at org.apache.wicket.RequestCycle.step(RequestCycle.java:1271)
at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1370)
at org.apache.wicket.RequestCycle.request(RequestCycle.java:501)
at
org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:455)
at
org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:288)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
at
org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterI
nternal(OpenSessionInViewFilter.java:198)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:76)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
at
org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:164
)
at
org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:141)
at
org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.jav
a:90)
at
org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter
.java:417)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
=======================================
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
