Bugs item #614116, was opened at 2002-09-24 19:56
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=376685&aid=614116&group_id=22866
Category: JBossTX
Group: v3.0 Rabbit Hole
Status: Open
Resolution: None
Priority: 5
Submitted By: Elias Ross (genman)
Assigned to: Nobody/Anonymous (nobody)
Summary: Oracle XA pooling/repeated rollback
Initial Comment:
I've been trying to isolate some problems I've been
having with transaction roll-backs and message-driven
beans. Basically, when a transaction is rolled-back
a number of times in the onMessage() body
(by calling MessageDrivenContext.setRollbackOnly()
5-10 times in succession),
the message is re-delivered to the MDB, but the Oracle
Connection is no longer in a good state.
This problem manifests itself when using a very small
connection pool (say 1-5) connections.
Example code snippet:
public void onMessage(javax.jms.Message jmsMessage)
{
Connection c = null;
try {
TextMessage tm = (TextMessage)jmsMessage;
String text = tm.getText();
c = ds.getConnection();
PreparedStatement s = c.prepareStatement("insert into test values (?)");
s.setString(1, text);
s.executeUpdate();
mdc.setRollbackOnly();
} catch (Exception e) {
throw new EJBException(e);
} finally {
if (c != null) {
try { c.close(); }
catch (Exception e) {
e.printStackTrace();
}
}
First I see it works for a few times. It rolls-back successfully
about 5 or 6 times. Then, for no reason, I see:
15:50:44,922 WARN [TxCapsule] XAException: tx=XidImpl [FormatId=257,
GlobalId=eross.m-qube.com//4,
BranchQual=] errorCode=XAER_RMERR
javax.transaction.xa.XAException
at oracle.jdbc.xa.OracleXAResource.allowGlobalTxnModeOnly(OracleXAResource.java:1069)
at oracle.jdbc.xa.OracleXAResource.suspendStacked(OracleXAResource.java:296)
at oracle.jdbc.xa.client.OracleXAResource.end(OracleXAResource.java:381)
at org.jboss.tm.TxCapsule.endResource(TxCapsule.java:1237)
at org.jboss.tm.TxCapsule.delistResource(TxCapsule.java:579)
at org.jboss.tm.TransactionImpl.delistResource(TransactionImpl.java:92)
at
org.jboss.resource.connectionmanager.XATxConnectionManager$XAConnectionEventListener.delist(XATxConnectionManager.java:284)
at
org.jboss.resource.connectionmanager.XATxConnectionManager$XAConnectionEventListener.connectionClosed(XATxConnectionManager.java:331)
at
org.jboss.resource.adapter.jdbc.BaseManagedConnection.fireConnectionEvent(BaseManagedConnection.java:152)
at
org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.fireConnectionEvent(XAManagedConnection.java:215)
at
org.jboss.resource.adapter.jdbc.xa.XAManagedConnection$1.connectionClosed(XAManagedConnection.java:127)
at
oracle.jdbc.pool.OraclePooledConnection.callListener(OraclePooledConnection.java:482)
at
oracle.jdbc.pool.OraclePooledConnection.logicalClose(OraclePooledConnection.java:445)
at oracle.jdbc.driver.OracleConnection.logicalClose(OracleConnection.java:2900)
at oracle.jdbc.driver.OracleConnection.close(OracleConnection.java:1418)
at com.proteusmobile.smx.AMDB.onMessage(AMDB.java:140)
Later:
15:50:44,929 WARN [TxCapsule] XAException: tx=XidImpl [FormatId=257,
GlobalId=eross.m-qube.com//4,
BranchQual=] errorCode=XAER_RMERR
javax.transaction.xa.XAException
at oracle.jdbc.xa.OracleXAResource.allowGlobalTxnModeOnly(OracleXAResource.java:1069)
at oracle.jdbc.xa.OracleXAResource.suspendStacked(OracleXAResource.java:296)
at oracle.jdbc.xa.client.OracleXAResource.end(OracleXAResource.java:381)
at org.jboss.tm.TxCapsule.endResource(TxCapsule.java:1237)
at org.jboss.tm.TxCapsule.endResources(TxCapsule.java:1312)
at org.jboss.tm.TxCapsule.rollback(TxCapsule.java:440)
at org.jboss.tm.TransactionImpl.rollback(TransactionImpl.java:83)
at org.jboss.jms.asf.StdServerSession.onMessage(StdServerSession.java:301)
at
org.jboss.mq.SpyMessageConsumer.sessionConsumerProcessMessage(SpyMessageConsumer.java:603)
at org.jboss.mq.SpyMessageConsumer.addMessage(SpyMessageConsumer.java:417)
at org.jboss.mq.SpySession.run(SpySession.java:259)
at org.jboss.jms.asf.StdServerSession.run(StdServerSession.java:177)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:655)
at java.lang.Thread.run(Thread.java:479)
And then, I can no longer get a connection:
15:50:44,971 WARN [TxCapsule] XAException: tx=XidImpl [FormatId=257,
GlobalId=eross.m-qube.com//5,
BranchQual=] errorCode=XAER_PROTO
javax.transaction.xa.XAException
at oracle.jdbc.xa.OracleXAResource.disallowLocalTxnMode(OracleXAResource.java:1045)
at oracle.jdbc.xa.client.OracleXAResource.start(OracleXAResource.java:153)
at org.jboss.tm.TxCapsule.startResource(TxCapsule.java:1180)
at org.jboss.tm.TxCapsule.enlistResource(TxCapsule.java:679)
at org.jboss.tm.TransactionImpl.enlistResource(TransactionImpl.java:102)
at
org.jboss.resource.connectionmanager.XATxConnectionManager$XAConnectionEventListener.enlist(XATxConnectionManager.java:262)
at
org.jboss.resource.connectionmanager.XATxConnectionManager.managedConnectionReconnected(XATxConnectionManager.java:202)
at
org.jboss.resource.connectionmanager.BaseConnectionManager2.allocateConnection(BaseConnectionManager2.java:534)
at
org.jboss.resource.connectionmanager.BaseConnectionManager2$ConnectionManagerProxy.allocateConnection(BaseConnectionManager2.java:812)
at
org.jboss.resource.adapter.jdbc.JDBCDataSource.getConnection(JDBCDataSource.java:110)
at com.proteusmobile.smx.AMDB.onMessage(AMDB.java:91)
In oracle.jdbc.xa.OracleXAResource (decompiled snippets):
protected void allowGlobalTxnModeOnly(int i)
throws XAException
{
if(((OracleConnection)m_conn).m_txn_mode != 2)
throw new XAException(i);
else
return;
}
//...
protected void disallowLocalTxnMode(int i)
throws XAException
{
if(((OracleConnection)m_conn).m_txn_mode == 1)
throw new XAException(i);
else
return;
}
I can no longer get a connection at all, as the
DataSource.getConnection() will no longer work.
I assume it's due to some funky state changes that have
come along with the repeated roll-backs.
----------------------------------------------------------------------
Comment By: Igor Fedorenko (igorfie)
Date: 2002-10-07 11:12
Message:
Logged In: YES
user_id=232950
Does it make any difference? Regardless of "TMSUSPEND"
fixed or not you still need to use TrackConnectionByTx to
overcome concurrency problems of oracle drivers.
----------------------------------------------------------------------
Comment By: Matt Cleveland (groovesoftware)
Date: 2002-10-07 08:42
Message:
Logged In: YES
user_id=85088
Thanks for your help. Changing TxConnectionManager to
use TMSUCCESS also fixes my problem. Right now I will
stick with the TrackConnectionByTx workaround I described
below since it doesn't require changing any code. Can I look
forward to having a fix for the TMSUSPEND problem in 3.2
final?
----------------------------------------------------------------------
Comment By: Igor Fedorenko (igorfie)
Date: 2002-10-03 18:55
Message:
Logged In: YES
user_id=232950
There is another problem with 3.2/40 which did not exist in
3.0 and which could be a cause of your problems.
3.2/4.0 calls xares.end(xid, TMSUSPEND) to delist a
connection from tx. This is done to be able to re-enlist the
connection should the tx need it again. Oracle does not like
TMSUSPEND and throws an exception.
To check if this is your problem change
org.jboss.resource.connectionmanager.TxConnectionManager
to use TMSUCCESS instead of TMSUSPEND and rerun
your tests. If this does not help (i.e. you still get an exception
with TMSUCCESS) could you please provide simple testcase
that demonstrates the problem?
----------------------------------------------------------------------
Comment By: Matt Cleveland (groovesoftware)
Date: 2002-10-03 08:49
Message:
Logged In: YES
user_id=85088
I have been watching this bug for several days because I
believe I am seeing the same error in a much different
situation. At least the oracle part of the stack trace I am
seeing is exactly the same (stack trace is pasted below).
Let me see if I can shed any light or at least add some
information to the discussion.
First of all, the problem I am seeing does not occur in 3.0.2
but it does occur in 3.2.0beta and 3.2.0 current CVS and
CVS HEAD (4.0.0alpha). However the workaround
suggested by igorfie fixes it in 3.2.0 and 4.0.0 (more on this
below).
The problem I am seeing does not in involve an MDB. I am
simply making Oracle updates from a stateless session
bean. I do not believe there is any concurrency involved
because I am the only one executing tests against the JBoss
installation I am testing with. However I could be wrong
about the concurrency since the servlet that makes the
stateless session bean call is part of a frameset.
Also, the problem I am seeing does not (as far as I can tell)
involve a rollback except for the one caused by this error. In
other words there is none of my code requesting a rollback
and if it wasn't for this Oracle XA error I believe the
transaction would commit succesfully.
The other notable factor is that the servlet involves a mixture
of non-tx and xa operations on Oracle. Let me explain.
Before calling the bean the servlet makes some Oracle calls
on a connection it gets from an
org.jboss.resource.connectionmanager.NoTxConnectionMana
ger. Then it calls the stateless session bean. The stateless
session bean does some more oracle work on a connection
it gets from an
org.jboss.resource.connectionmanager.XATxConnectionMana
ger.
I can work around the problem by setting
TrackConnectionByTx=true on the XA connection manager
as suggested by igorfie. As igorfie says this will, "configure
JBOSS not to use same xa resource in two
concurrent transactions." But I'm not really sure why this
fixes it since as far as I can tell there is no concurrency
problems involved in my case (although again I should say I
might be wrong on that). I am also unsure why I do not have
this problem in 3.0.2.
It seems like this solution is reasonable and will not have any
performance impact on our application as long as the
connection pool is sized appropriately. Does anyone see
any performance issues with this workaround?
Based on my information I would like to suggest that the
problem is not one of concurrency, but comes from some
pattern of reuse of the Oracle connections which can be
reproduced in concurrent and non-concurrent cases but that
TrackConnectionByTx alters enough to fix.
Finally, here's my stack trace so that you can confirm that
this is actually the same problem. Note that the first 3 lines
(the Oracle Part) are exactly the same as that reported by
genman.
Hope this helps.
Matt
2002-10-03 12:00:43,793 WARN
[org.jboss.tm.TransactionImpl] XAException:
tx=TransactionImpl:XidImpl [FormatId=257,
GlobalId=malt//11, BranchQual=] errorCode=XAER_RMERR
javax.transaction.xa.XAException
at
oracle.jdbc.xa.OracleXAResource.allowGlobalTxnModeOnly
(OracleXAResource.java:1069)
at oracle.jdbc.xa.OracleXAResource.suspendStacked
(OracleXAResource.java:296)
at oracle.jdbc.xa.client.OracleXAResource.end
(OracleXAResource.java:381)
at org.jboss.tm.TransactionImpl.endResource
(TransactionImpl.java:1056)
at org.jboss.tm.TransactionImpl.endResources
(TransactionImpl.java:1118)
at org.jboss.tm.TransactionImpl.commit
(TransactionImpl.java:332)
at org.jboss.ejb.plugins.TxInterceptorCMT.endTransaction
(TxInterceptorCMT.java:308)
at
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions
(TxInterceptorCMT.java:194)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke
(TxInterceptorCMT.java:66)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke
(SecurityInterceptor.java:130)
at org.jboss.ejb.plugins.LogInterceptor.invoke
(LogInterceptor.java:205)
at org.jboss.ejb.plugins.CleanShutdownInterceptor.invoke
(CleanShutdownInterceptor.java:255)
at
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke
(ProxyFactoryFinderInterceptor.java:154)
at org.jboss.ejb.StatelessSessionContainer.invoke
(StatelessSessionContainer.java:303)
at org.jboss.ejb.Container.invoke(Container.java:671)
at org.jboss.mx.server.MBeanServerImpl.invoke
(MBeanServerImpl.java:549)
at org.jboss.invocation.local.LocalInvoker.invoke
(LocalInvoker.java:100)
at org.jboss.invocation.InvokerInterceptor.invoke
(InvokerInterceptor.java:88)
at org.jboss.proxy.TransactionInterceptor.invoke
(TransactionInterceptor.java:77)
at org.jboss.proxy.SecurityInterceptor.invoke
(SecurityInterceptor.java:80)
at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke
(StatelessSessionInterceptor.java:109)
at org.jboss.proxy.ClientContainer.invoke
(ClientContainer.java:82)
at $Proxy99.getAllBasicGuestInformation(Unknown
Source)
at
com.synxis.srms.ejbs.dm.SrmsGuestInformationServiceTxHa
ndlerBean.getAllBasicGuestInformation
(SrmsGuestInformationServiceTxHandlerBean.java:726)
at java.lang.reflect.Method.invoke(Native Method)
at
org.jboss.ejb.StatelessSessionContainer$ContainerInterceptor
.invoke(StatelessSessionContainer.java:606)
at
org.jboss.resource.connectionmanager.CachedConnectionInte
rceptor.invoke(CachedConnectionInterceptor.java:186)
at
org.jboss.ejb.plugins.StatelessSessionInstanceInterceptor.inv
oke(StatelessSessionInstanceInterceptor.java:77)
at org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext
(AbstractTxInterceptor.java:108)
at
org.jboss.ejb.plugins.TxInterceptorCMT.runWithTransactions
(TxInterceptorCMT.java:214)
at org.jboss.ejb.plugins.TxInterceptorCMT.invoke
(TxInterceptorCMT.java:66)
at org.jboss.ejb.plugins.SecurityInterceptor.invoke
(SecurityInterceptor.java:130)
at org.jboss.ejb.plugins.LogInterceptor.invoke
(LogInterceptor.java:205)
at org.jboss.ejb.plugins.CleanShutdownInterceptor.invoke
(CleanShutdownInterceptor.java:255)
at
org.jboss.ejb.plugins.ProxyFactoryFinderInterceptor.invoke
(ProxyFactoryFinderInterceptor.java:154)
at org.jboss.ejb.StatelessSessionContainer.invoke
(StatelessSessionContainer.java:303)
at org.jboss.ejb.Container.invoke(Container.java:671)
at org.jboss.mx.server.MBeanServerImpl.invoke
(MBeanServerImpl.java:549)
at org.jboss.invocation.local.LocalInvoker.invoke
(LocalInvoker.java:100)
at org.jboss.invocation.InvokerInterceptor.invoke
(InvokerInterceptor.java:88)
at org.jboss.proxy.TransactionInterceptor.invoke
(TransactionInterceptor.java:77)
at org.jboss.proxy.SecurityInterceptor.invoke
(SecurityInterceptor.java:80)
at org.jboss.proxy.ejb.StatelessSessionInterceptor.invoke
(StatelessSessionInterceptor.java:109)
at org.jboss.proxy.ClientContainer.invoke
(ClientContainer.java:82)
at $Proxy99.getAllBasicGuestInformation(Unknown
Source)
at
com.synxis.srms.servlet.va.GuestInfo.GuestInfoName.lookup
Name(GuestInfoName.java:101)
at
com.synxis.srms.servlet.va.GuestInfo.GuestInfoFrameset.doR
ealPost(GuestInfoFrameset.java:76)
at com.synxis.srms.servlet.SrmsServletBase.doPost
(SrmsServletBase.java:115)
at com.synxis.srms.servlet.SrmsServletBase.doGet
(SrmsServletBase.java:144)
at javax.servlet.http.HttpServlet.service
(HttpServlet.java:740)
at javax.servlet.http.HttpServlet.service
(HttpServlet.java:853)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilt
er(ApplicationFilterChain.java:247)
at org.apache.catalina.core.ApplicationFilterChain.doFilter
(ApplicationFilterChain.java:193)
at org.apache.catalina.core.StandardWrapperValve.invoke
(StandardWrapperValve.java:243)
at org.apache.catalina.core.StandardPipeline.invokeNext
(StandardPipeline.java:566)
at org.apache.catalina.core.StandardPipeline.invoke
(StandardPipeline.java:472)
at org.apache.catalina.core.ContainerBase.invoke
(ContainerBase.java:943)
at org.apache.catalina.core.StandardContextValve.invoke
(StandardContextValve.java:190)
at org.apache.catalina.core.StandardPipeline.invokeNext
(StandardPipeline.java:566)
at org.apache.catalina.valves.CertificatesValve.invoke
(CertificatesValve.java:246)
at org.apache.catalina.core.StandardPipeline.invokeNext
(StandardPipeline.java:564)
at org.apache.catalina.core.StandardPipeline.invoke
(StandardPipeline.java:472)
at org.apache.catalina.core.ContainerBase.invoke
(ContainerBase.java:943)
at org.apache.catalina.core.StandardContext.invoke
(StandardContext.java:2347)
at org.apache.catalina.core.StandardHostValve.invoke
(StandardHostValve.java:180)
at org.apache.catalina.core.StandardPipeline.invokeNext
(StandardPipeline.java:566)
at org.apache.catalina.valves.ErrorDispatcherValve.invoke
(ErrorDispatcherValve.java:170)
at org.apache.catalina.core.StandardPipeline.invokeNext
(StandardPipeline.java:564)
at org.apache.catalina.valves.ErrorReportValve.invoke
(ErrorReportValve.java:170)
at org.apache.catalina.core.StandardPipeline.invokeNext
(StandardPipeline.java:564)
at org.apache.catalina.valves.AccessLogValve.invoke
(AccessLogValve.java:468)
at org.apache.catalina.core.StandardPipeline.invokeNext
(StandardPipeline.java:564)
at org.apache.catalina.core.StandardPipeline.invoke
(StandardPipeline.java:472)
at org.apache.catalina.core.ContainerBase.invoke
(ContainerBase.java:943)
at org.apache.catalina.core.StandardEngineValve.invoke
(StandardEngineValve.java:174)
at org.apache.catalina.core.StandardPipeline.invokeNext
(StandardPipeline.java:566)
at org.apache.catalina.core.StandardPipeline.invoke
(StandardPipeline.java:472)
at org.apache.catalina.core.ContainerBase.invoke
(ContainerBase.java:943)
at
org.apache.catalina.connector.http.HttpProcessor.process
(HttpProcessor.java:1027)
at org.apache.catalina.connector.http.HttpProcessor.run
(HttpProcessor.java:1125)
at java.lang.Thread.run(Thread.java:479)
----------------------------------------------------------------------
Comment By: Elias Ross (genman)
Date: 2002-10-01 23:51
Message:
Logged In: YES
user_id=556458
>From what I have seen in the bug database, Oracle hasn't
had a very good track record.
Thanks for taking a good look at this, and I'll let my team
know about this problem. In the meantime, I'll wait with
bated breath for version 3.2.
----------------------------------------------------------------------
Comment By: Igor Fedorenko (igorfie)
Date: 2002-10-01 21:57
Message:
Logged In: YES
user_id=232950
I took a closer look on this problem and think that I know
what is going on. Basically, we have two transactions
TX1 and TX2 both using same xa connection and
executed concurrently on two different threads
tx1: start jms
tx1: start oracle
tx1: get connection
tx1: prepare statement
tx1: execute update
tx1: end oracle
tx1: rollback jms => tx2: start jms
tx2: start oracle
tx2: get connection
tx1: rollback oracle ** tx2: prepare statement **
** this is when first exception is thrown. note that
sometimes second transaction gets enough time to
execute update.
I am not sure if you can see the table above ;-) but the
idea is that when jms session rolls back jms message is
redelivered and cunsumed in a new transaction that
works in parallel with original tx. Notice that TX2
executes prepareStatement at the same time TX1 calls
rollback on oracle xa resource. And this is where oracle
does not shine and all :-(
I wrote few tests to exercise this scenario and ran them
with oracle 8173 and jdbc drivers version 9201. These
tests showed that oracle xa drivers are not thread safe.
There are few possible options to fix this problem. First,
it is possible that problem will disappear with oracle
9201 and introduction of minor sinchronization between
calls to shared xa connection. Second, we could
configure JBOSS not to use same xa resource in two
concurrent transactions (this option is available in JBOSS
3.2 and 4.0). Third, we could introduce heavy
synchronization of xa connection calls; I did some
prototyping here and it looks like we need to
synchronize all calls to xa connection and its derived
objects (xa resource, connection, statements and result
sets). I consider last option as a very last resort and
would not try to implement until all other possibilities are
eliminated.
PS: if somebody wants to take a look on my tests (or/and
discuss them with oracle?) I'll be glad to make them
available.
PSS: I still hope to get oracle 9i later this week.
----------------------------------------------------------------------
Comment By: Elias Ross (genman)
Date: 2002-09-26 13:11
Message:
Logged In: YES
user_id=556458
Yes it is version 9201 on Linux.
Perhaps the problem has resolved itself in a later version of
JBoss?
I appreciate you looking into this.
----------------------------------------------------------------------
Comment By: Igor Fedorenko (igorfie)
Date: 2002-09-25 14:59
Message:
Logged In: YES
user_id=232950
I'll try to reproduce this problem when I get oracle 9201
around (which is expected to happen next week but I do not
have control over it). Am I right in an assumption that your
server version is 9201?
----------------------------------------------------------------------
Comment By: Elias Ross (genman)
Date: 2002-09-25 14:08
Message:
Logged In: YES
user_id=556458
Jboss is version 3.0.1 -- using Oracle 9.2.0.1 thin
drivers. (Specifically jboss-3.0.1_tomcat-4.0.4).
Also, when the connection is being removed from the connection
pool, I see:
15:54:45,197 INFO [JBossManagedConnectionPool] Exception destroying ManagedConnection
javax.resource.ResourceException: Unable to close DB connection:
java.sql.SQLException: ORA-02089: COMMIT is
not allowed in a subordinate session
at
org.jboss.resource.adapter.jdbc.xa.XAManagedConnection.destroy(XAManagedConnection.java:193)
at
org.jboss.resource.connectionmanager.InternalManagedConnectionPool.doDestroy(InternalManagedConnectionPool.java:268)
at
org.jboss.resource.connectionmanager.InternalManagedConnectionPool.removeTimedOut(InternalManagedConnectionPool.java:186)
at org.jboss.resource.connectionmanager.IdleRemover$1.run(IdleRemover.java:70)
at java.lang.Thread.run(Thread.java:479)
I suspect it's due to some sort of Oracle (!) or JBoss bug
which changes the connection transaction type
from local to global...
Thanks for considering the problem.
----------------------------------------------------------------------
Comment By: Igor Fedorenko (igorfie)
Date: 2002-09-25 09:24
Message:
Logged In: YES
user_id=232950
Could you provide information about oracle server version,
oracle jdbc driver version and driver type (thin/oci) and JBOSS
version?
----------------------------------------------------------------------
You can respond by visiting:
https://sourceforge.net/tracker/?func=detail&atid=376685&aid=614116&group_id=22866
-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
Jboss-development mailing list
[EMAIL PROTECTED]
https://lists.sourceforge.net/lists/listinfo/jboss-development
