Not just one thread, but that thread dump part (from my first post) is
repeated for 149 other threads have the same situation. ( because of tomcats
150 maxthreads).
But when looking at the db during a freeze, there is no lock on any table at
db level. MySQL Administrator shows that there are 15 threads during the
freeze (equal to c3p0's maxPoolSize), all sleeping.
I have now added some C3P0 debug and workaround options to see if I can get
some extra info out of it.
http://www.mchange.com/projects/c3p0/index.html#configuring_to_debug_and_wor
karound_broken_clients
(debugUnreturnedConnectionStackTraces true and unreturnedConnectionTimeout
180)
-----Oorspronkelijk bericht-----
Van: John Krasnay [mailto:j...@krasnay.ca]
Verzonden: woensdag 15 april 2009 15:57
Aan: users@wicket.apache.org
Onderwerp: Re: WebApp Freezes
Sounds like you have a thread holding a lock on a critical table and
subsequent threads are lining up behind it waiting for it to finish. You
should check your MySQL to try and figure out who's holding the lock and
why.
Note that the culprit thread need not be hung up in the database. Locks
are held until the transaction commits, so your thread could be hung up
in app code after making a database update but before the transaction is
committed.
jk
On Wed, Apr 15, 2009 at 03:41:07PM +0200, J wrote:
(Sorry for the empty message. First I tried Gmail, but that doesn't work
with this mailing list. Then I tried GMX webclient, but that client always
sends as HTML, which probably caused the message to be stripped to empty.)
I'm experiencing freezes on a production website.
Server specs:
-OS: Linux CentOS
-Webserver: Tomcat 6
-MySQL 5 with default settings
-Frameworks: Wicket 1.4-rc2 (webframework), Spring 2.5.6, Hibernate
3.3.1.GA, C3P0 db pooling 0.9.1.2, URLRewriteFilter 3.1.0, Spring
OpenSessionInViewFilter
Problem:
At some point (about 5 to 24 hours) after a boot, Tomcat seems to stop
serving requests, although it does create new threads for new incoming
connections. But since it does not serve those connections, the new
connections will show and keep showing 0kb transfer. Shortly after that,
max-thread is reached, so then I'm unable to access any webapp (the main
website + tomcat manager) running on that Tomcat server. I'm not sure if
it
is able to serve static resources in the short time window where it still
has some threads free, because the time window is too short to notice when
it happens.
Observations:
-No exceptions or errors in the catalina logs. So no memory problems,
since
no error occurs in the logs
-Java Thread dump using command: kill -QUIT (see output below), shows the
text "locked" and "WAITING".
-Adding c3p0 db pooling idle checks, tests and timeout settings (see
below)
did not help.
-Using Mysql Administrator GUI shows that after a freeze, there will be 15
threads, all sleeping. Normally this would return to a minPoolSize of 5,
if
I'm correct, but thats not the case.
-The freeze continues for hours, and does not recover. I have to restart
Tomcat.
I'm suspecting that it has something to do with DB pooling. I'm not sure
if
"locked" and "WAITING" are normal behaviour. But if there is something
wrong, why doesn't c3p0 recover from it?
Can somebody shed some light on this? :)
Configuration:
=== DB pooling in Spring applicationContext.xml ====
<bean id="dataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource">
<property
name="driverClass"><value>com.mysql.jdbc.Driver</value></property>
<property name="jdbcUrl" value="jdbc:mysql://localhost/xxxxxx" />
<property name="user"><value>xxxxxxxx</value></property>
<property name="password"><value>xxxxxxxxx</value></property>
<property name="maxConnectionAge"><value>600</value></property>
<property
name="idleConnectionTestPeriod"><value>180</value></property>
<property
name="testConnectionOnCheckin"><value>true</value></property>
<property name="acquireIncrement"><value>5</value></property>
<property name="maxIdleTime"><value>180</value></property>
<property name="maxPoolSize"><value>15</value></property>
<property name="maxStatements"><value>100</value></property>
<property name="minPoolSize"><value>5</value></property>
</bean>
===========================
===== Thread dump part ===========
"http-8080-28" daemon prio=10 tid=0xb4fbb400 nid=0x4ae2 in Object.wait()
[0xb4316000..0xb4318130]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at
com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePo
ol.java:1315)
at
com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicRe
sourcePool.java:557)
- locked <0x72c74b10> (a
com.mchange.v2.resourcepool.BasicResourcePool)
at
com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResource
Pool.java:477)
at
com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C
3P0PooledConnectionPool.java:525)
at
com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(Abstract
PoolBackedDataSource.java:128)
at
org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConn
ection(LocalDataSourceConnectionProvider.java:81)
at
org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:4
23)
at
org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:14
4)
at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:119)
at
org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:57)
at
org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1326)
at
org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(Hiber
nateTransactionManager.java:510)
at
org.springframework.transaction.support.AbstractPlatformTransactionManager.g
etTransaction(AbstractPlatformTransactionManager.java:350)
at
org.springframework.transaction.interceptor.TransactionAspectSupport.createT
ransactionIfNecessary(TransactionAspectSupport.java:262)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(Tr
ansactionInterceptor.java:101)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Reflect
iveMethodInvocation.java:171)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopPro
xy.java:204)
at $Proxy37.findById(Unknown Source)
at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.wicket.proxy.LazyInitProxyFactory$JdkHandler.invoke(LazyInitProxy
Factory.java:416)
at org.apache.wicket.proxy.$Proxy2.findById(Unknown Source)
at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
at
org.apache.wicket.model.LoadableDetachableModel.getObject(LoadableDetachable
Model.java:119)
at xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
at sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstruc
torAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java
:151)
at
org.apache.wicket.session.DefaultPageFactory.newPage(DefaultPageFactory.java
:89)
at
org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.new
Page(BookmarkablePageRequestTarget.java:306)
at
org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.get
Page(BookmarkablePageRequestTarget.java:321)
at
org.apache.wicket.request.target.component.BookmarkablePageRequestTarget.pro
cessEvents(BookmarkablePageRequestTarget.java:234)
at
org.apache.wicket.request.AbstractRequestCycleProcessor.processEvents(Abstra
ctRequestCycleProcessor.java:92)
at
org.apache.wicket.RequestCycle.processEventsAndRespond(RequestCycle.java:119
2)
at org.apache.wicket.RequestCycle.step(RequestCycle.java:1271)
at org.apache.wicket.RequestCycle.steps(RequestCycle.java:1370)
at org.apache.wicket.RequestCycle.request(RequestCycle.java:501)
at
org.apache.wicket.protocol.http.WicketFilter.doGet(WicketFilter.java:455)
at
org.apache.wicket.protocol.http.WicketFilter.doFilter(WicketFilter.java:288)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
at
org.springframework.orm.hibernate3.support.OpenSessionInViewFilter.doFilterI
nternal(OpenSessionInViewFilter.java:198)
at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestF
ilter.java:76)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
at
org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:164
)
at
org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:141)
at
org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.jav
a:90)
at
org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter
.java:417)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Application
FilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterCh
ain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.ja
va:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.ja
va:191)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128
)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102
)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java
:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http
11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
=======================================
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
For additional commands, e-mail: users-h...@wicket.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
For additional commands, e-mail: users-h...@wicket.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscr...@wicket.apache.org
For additional commands, e-mail: users-h...@wicket.apache.org
