I am running JBoss 3.2.2 in production. Our system processes hundreds of thousands
transactions a day. We are using clustering with cache invalidation with 3 instances
across 3 machines. In general, everything works great. However, every other week or
so, I will see the system "hang", with errors like these thrown:
2004-06-15 05:46:41,257 WARN [org.jboss.tm.TransactionImpl] Transaction
TransactionImpl:XidImpl [FormatId=257, GlobalId=www7.mateso.com//55944547,
BranchQual=] timed out. status=STATUS_ACTIVE
2004-06-15 05:46:41,266 WARN [org.jboss.ejb.plugins.AbstractInstanceCache] Unable to
passivate due to ctx lock, id=5084257
2004-06-15 05:46:41,354 WARN [org.jboss.tm.TransactionImpl] Transaction
TransactionImpl:XidImpl [FormatId=257, GlobalId=www7.mateso.com//55944555,
BranchQual=] timed out. status=STATUS_ACTIVE
2004-06-15 05:47:18,543 WARN [org.jboss.tm.TransactionImpl] Transaction
TransactionImpl:XidImpl [FormatId=257, GlobalId=www7.mateso.com//55944825,
BranchQual=] timed out. status=STATUS_ACTIVE
2004-06-15 05:47:52,166 WARN [org.jboss.tm.TransactionImpl] Transaction
TransactionImpl:XidImpl [FormatId=257, GlobalId=www7.mateso.com//55945145,
BranchQual=] timed out. status=STATUS_ACTIVE
2004-06-15 05:47:52,168 ERROR [org.jboss.ejb.BeanLock] PoolThread-7Saw rolled back
tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=www7.mateso.com//55945145,
BranchQual=] waiting for txLock
2004-06-15 05:47:52,168 WARN [org.jboss.tm.TransactionImpl] Lock contention,
tx=TransactionImpl:XidImpl [FormatId=257, GlobalId=www7.mateso.com//55945145,
BranchQual=]
2004-06-15 05:47:52,168 ERROR [org.jboss.ejb.plugins.LogInterceptor] RuntimeException:
java.lang.RuntimeException: Transaction marked for rollback, possibly a timeout
at
org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.waitForTx(QueuedPessimisticEJBLock.java(Compiled
Code))
at
org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.doSchedule(QueuedPessimisticEJBLock.java(Compiled
Code))
at
org.jboss.ejb.plugins.lock.QueuedPessimisticEJBLock.schedule(QueuedPessimisticEJBLock.java(Compiled
Code))
at
org.jboss.ejb.plugins.EntityLockInterceptor.invoke(EntityLockInterceptor.java(Compiled
Code))
at
org.jboss.ejb.plugins.EntityCreationInterceptor.invoke(EntityCreationInterceptor.java(Compiled
Code))
at
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java(Compiled
Code))
at
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java(Compiled
Code))
at
org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java(Compiled Code))
at
org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java(Compiled
Code))
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java(Compiled
Code))
at
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java(Compiled
Code))
at org.jboss.ejb.EntityContainer.internalInvoke(EntityContainer.java(Compiled
Code))
at org.jboss.ejb.Container.invoke(Container.java(Compiled Code))
at
org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java(Compiled
Code))
at org.jboss.ejb.plugins.local.EntityProxy.invoke(EntityProxy.java(Compiled
Code))
at $Proxy219.getConsumerValue(Unknown Source)
at
mts.psp.consumer.ConsumerManagerEJB.getConsumerByEmail(ConsumerManagerEJB.java:248)
at sun.reflect.GeneratedMethodAccessor745.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java(Compiled
Code))
at java.lang.reflect.Method.invoke(Method.java(Compiled Code))
at
org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor.invoke(StatelessSessionContainer.java(Compiled
Code))
at
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java(Compiled
Code))
at
org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.invoke(StatelessSessionInstanceInterceptor.java(Compiled
Code))
at
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java(Compiled
Code))
at
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java(Compiled
Code))
at
org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java(Compiled Code))
at
org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java(Compiled
Code))
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java(Compiled
Code))
at
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java(Compiled
Code))
at
org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java(Compiled
Code))
at org.jboss.ejb.Container.invoke(Container.java(Compiled Code))
at
org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java(Compiled
Code))
at
org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java(Compiled
Code))
at $Proxy579.getConsumerByEmail(Unknown Source)
at
mts.psp.consumer.ConsumerDelegate.getConsumerByEmail(ConsumerDelegate.java:290)
This occurs across all 3 instances at the same time, and the site becomes
unaccessible. 9 times out of 10, the servers will snap out of this state within a few
minutes, and everything will work fine again. However, sometimes it stays in this
state for a longer period of time and I need to bounce the instances. I'm am pretty
sure that there is no external program locking our database (which would cause similar
behavior), especially since just an application server restart fixes the issue.
Another oddity (which may or may not be related), is I will see
-------------------------------------------------------
GMS: address is www7:32832
-------------------------------------------------------
Thrown out all of a sudden (www7 is one of the server names). Usually, you only see
this on startup. After seeing this output, the other two members of the cluster will
detect a dead member, and then detect a new member.
Any help or clues as to what is going on would be greatly appreciated.
A stack trace from one of the other servers looks pretty similar but is a little
different:
2004-06-15 07:18:42,089 WARN [org.jboss.tm.TransactionImpl] Transaction
TransactionImpl:XidImpl [FormatId=257, GlobalId=www4.mateso.com//50385479,
BranchQual=] timed out. status=STATUS_ACTIVE
2004-06-15 07:18:42,120 ERROR [org.jboss.ejb.plugins.LogInterceptor]
TransactionRolledbackLocalException in method: public abstract void
mts.psp.consumer.ConsumerManagerLocal.updateSiteLoginDate(java.lang.Integer,java.lang.Integer),
causedBy:
org.jboss.tm.JBossRollbackException: Unable to commit, tx=TransactionImpl:XidImpl
[FormatId=257, GlobalId=www4.mateso.com//50385479, BranchQual=]
status=STATUS_NO_TRANSACTION
at org.jboss.tm.TransactionImpl.commit(TransactionImpl.java(Compiled Code))
at
org.jboss.ejb.plugins.TxInterceptorCMT.endTransaction(TxInterceptorCMT.java(Compiled
Code))
at
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions(TxInterceptorCMT.java(Compiled
Code))
at
org.jboss.ejb.plugins.TxInterceptorCMT.invoke(TxInterceptorCMT.java(Compiled Code))
at
org.jboss.ejb.plugins.SecurityInterceptor.invoke(SecurityInterceptor.java(Compiled
Code))
at org.jboss.ejb.plugins.LogInterceptor.invoke(LogInterceptor.java(Compiled
Code))
at
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke(ProxyFactoryFinderInterceptor.java(Compiled
Code))
at
org.jboss.ejb.StatelessSessionContainer.internalInvoke(StatelessSessionContainer.java(Compiled
Code))
at org.jboss.ejb.Container.invoke(Container.java(Compiled Code))
at
org.jboss.ejb.plugins.local.BaseLocalProxyFactory.invoke(BaseLocalProxyFactory.java(Compiled
Code))
at
org.jboss.ejb.plugins.local.StatelessSessionProxy.invoke(StatelessSessionProxy.java(Compiled
Code))
at
mts.psp.consumer.ConsumerDelegate.updateSiteLoginDate(ConsumerDelegate.java(Compiled
Code))
at
mts.psp.web.struts.util.WebSessionUtil.setLoggedIn(WebSessionUtil.java(Compiled Code))
at
mts.psp.web.struts.auth.SessionFilter.handleCookiesAndMungeCode(SessionFilter.java(Compiled
Code))
at mts.psp.web.struts.auth.SessionFilter.doFilter(SessionFilter.java(Compiled
Code))
at
org.mortbay.jetty.servlet.WebApplicationHandler$Chain.doFilter(WebApplicationHandler.java(Compiled
Code))
at
org.mortbay.jetty.servlet.WebApplicationHandler.dispatch(WebApplicationHandler.java(Compiled
Code))
at
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java(Compiled Code))
at org.mortbay.http.HttpContext.handle(HttpContext.java(Compiled Code))
at
org.mortbay.jetty.servlet.WebApplicationContext.handle(WebApplicationContext.java(Compiled
Code))
at org.mortbay.http.HttpContext.handle(HttpContext.java(Compiled Code))
at org.mortbay.http.HttpServer.service(HttpServer.java(Compiled Code))
at org.jboss.jetty.Jetty.service(Jetty.java(Compiled Code))
at org.mortbay.http.HttpConnection.service(HttpConnection.java(Inlined
Compiled Code))
at
org.mortbay.http.ajp.AJP13Connection.handleNext(AJP13Connection.java(Compiled Code))
at org.mortbay.http.HttpConnection.handle(HttpConnection.java(Compiled Code))
at
org.mortbay.http.ajp.AJP13Listener.handleConnection(AJP13Listener.java(Compiled Code))
at org.mortbay.util.ThreadedServer.handle(ThreadedServer.java(Compiled Code))
at org.mortbay.util.ThreadPool$PoolThread.run(ThreadPool.java(Compiled Code))
View the original post :
http://www.jboss.org/index.html?module=bb&op=viewtopic&p=3838794#3838794
Reply to the post :
http://www.jboss.org/index.html?module=bb&op=posting&mode=reply&p=3838794
-------------------------------------------------------
This SF.Net email is sponsored by The 2004 JavaOne(SM) Conference
Learn from the experts at JavaOne(SM), Sun's Worldwide Java Developer
Conference, June 28 - July 1 at the Moscone Center in San Francisco, CA
REGISTER AND SAVE! http://java.sun.com/javaone/sf Priority Code NWMGYKND
_______________________________________________
JBoss-user mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-user
