Hi,
I don't think the main issue is DB2. Using "With UR" may solve the issue of
locking on table but the main problem is exception occured in thread 1
caused tranasaction in thread 2 to rollback.
I am suspecting the following.
14.00 The Async schedule job which is BeanManaged started Transaction1
using userTransaction.begin().
14:09 The REST call tried to fetch data from Batch table(Synchronous call)
(Containter managed txn), but end up with LockAcquisitionException which
marked setRollbackOnly to true.
14:21 The Async schedule job tried to commit after creating the Batch file
but eventually got rollbacked by the container with the exception
org.apache.openejb.core.transaction.TransactionRolledbackException:
Transaction was rolled back, presumably because setRollbackOnly
was called during a synchronization
Both method calls which started at 14.00 and 14.09 executed in 2 different
independent threads first call is BeanManaged and Rest call is
ContainerManaged.
I am suspecting that the exception occured at 14.09 is the root cause of
the rollback of the schedule job transaction.
But if it is true then I think its wrong as both calls are executed in 2
independent threads and an exception occured in one thread should not cause
other thread transaction to rollback.
Hope my analysis make sense.
Regards
Kalyan.
On Fri, Dec 19, 2014 at 1:01 PM, Andy Gumbrecht <[email protected]>
wrote:
>
> Hi Radhakrishna,
>
> This is something to do with a deadlock on the DB2 table that your query
> is trying to execute.
>
> You will need to use the DB2 tools find the root cause of the deadlock at
> the time this error occurs - Just google 'DB2 Deadlock' and see if you can
> tune the table (flagging it as volatile may help).
>
> Things like this happen when several processes are trying to get a lock on
> the table at the same time - Check to see if you have other processes that
> may be using this table.
>
> Andy.
>
>
> On 19/12/2014 11:35, Radhakrishna Kalyan wrote:
>
>> Hi,
>>
>> We have deployed our application for the first time in our Production env
>> yesterday, and we got the following exception.
>>
>> We have Asynchronous schedule job running every day at a given time say
>> 14.00.
>> The method performs a query operation on a Transaction table and create a
>> text file with those transactions found for a given criteria.
>> After creating the text file the method also create an entry in another
>> table called Batch table identifying which batch files have been created.
>>
>> Also a rest interface is exposed to search the Batch table and list out
>> the
>> entries and find out which batch files are created.
>>
>>
>> Here is the exception log stacktrace
>>
>>
>> dec 18, 2014 2:20:13 EM
>> org.apache.openejb.core.transaction.EjbTransactionUtil
>> handleSystemException
>> SEVERE: EjbTransactionUtil.handleSystemException: The transaction has
>> been marked rollback only because the bean encountered a
>> non-application exception
>> :org.hibernate.exception.LockAcquisitionException : could not execute
>> query
>> javax.ejb.EJBTransactionRolledbackException: The transaction has been
>> marked rollback only because the bean encountered a non-application
>> exception :org.hibernate.exception.LockAcquisitionException : could
>> not execute query
>> at org.apache.openejb.core.ivm.BaseEjbProxyHandler.
>> convertException(BaseEjbProxyHandler.java:381)
>> at org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(
>> BaseEjbProxyHandler.java:312)
>> at com.netgiro.centralizedbatch.db.dao.BatchOrderDao$$
>> LocalBeanProxy.getCount(com/netgiro/centralizedbatch/db/
>> dao/BatchOrderDao.java)
>> at com.netgiro.centralizedbatch.rest.resources.
>> BatchOrdersResource.searchBatchOrders(BatchOrdersResource.java:163)
>> at sun.reflect.GeneratedMethodAccessor46.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.openejb.core.interceptor.
>> ReflectionInvocationContext$Invocation.invoke(
>> ReflectionInvocationContext.java:192)
>> at org.apache.openejb.core.interceptor.
>> ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:173)
>> at com.netgiro.centralizedbatch.interceptors.
>> AuditLogInterceptor.aroundInvoke(AuditLogInterceptor.java:22)
>> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.openejb.core.interceptor.
>> ReflectionInvocationContext$Invocation.invoke(
>> ReflectionInvocationContext.java:192)
>> at org.apache.openejb.core.interceptor.
>> ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:173)
>> at org.apache.openejb.monitoring.StatsInterceptor.record(
>> StatsInterceptor.java:181)
>> at org.apache.openejb.monitoring.StatsInterceptor.invoke(
>> StatsInterceptor.java:100)
>> at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.openejb.core.interceptor.
>> ReflectionInvocationContext$Invocation.invoke(
>> ReflectionInvocationContext.java:192)
>> at org.apache.openejb.core.interceptor.
>> ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:173)
>> at org.apache.openejb.core.interceptor.InterceptorStack.
>> invoke(InterceptorStack.java:85)
>> at org.apache.openejb.core.stateless.StatelessContainer._
>> invoke(StatelessContainer.java:227)
>> at org.apache.openejb.core.stateless.StatelessContainer.
>> invoke(StatelessContainer.java:194)
>> at org.apache.openejb.util.proxy.ProxyEJB$Handler.invoke(
>> ProxyEJB.java:74)
>> at com.netgiro.centralizedbatch.rest.resources.
>> BatchOrdersResource$$LocalBeanProxy.searchBatchOrders(com/netgiro/
>> centralizedbatch/rest/resources/BatchOrdersResource.java)
>> at sun.reflect.GeneratedMethodAccessor45.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.openejb.server.cxf.rs.OpenEJBEJBInvoker.
>> performInvocation(OpenEJBEJBInvoker.java:89)
>> at org.apache.cxf.service.invoker.AbstractInvoker.
>> invoke(AbstractInvoker.java:104)
>> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(
>> JAXRSInvoker.java:205)
>> at org.apache.openejb.server.cxf.rs.OpenEJBEJBInvoker.invoke(
>> OpenEJBEJBInvoker.java:62)
>> at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(
>> JAXRSInvoker.java:102)
>> at org.apache.openejb.server.cxf.rs.AutoJAXRSInvoker.invoke(
>> AutoJAXRSInvoker.java:64)
>> at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.
>> run(ServiceInvokerInterceptor.java:58)
>> at org.apache.cxf.interceptor.ServiceInvokerInterceptor.
>> handleMessage(ServiceInvokerInterceptor.java:94)
>> at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(
>> PhaseInterceptorChain.java:272)
>> at org.apache.cxf.transport.ChainInitiationObserver.onMessage(
>> ChainInitiationObserver.java:121)
>> at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(
>> AbstractHTTPDestination.java:239)
>> at org.apache.openejb.server.cxf.rs.CxfRsHttpListener.
>> onMessage(CxfRsHttpListener.java:187)
>> at org.apache.openejb.server.httpd.OpenEJBHttpRegistry$
>> ClassLoaderHttpListener.onMessage(OpenEJBHttpRegistry.java:111)
>> at org.apache.openejb.server.httpd.HttpListenerRegistry.
>> onMessage(HttpListenerRegistry.java:71)
>> at org.apache.openejb.server.httpd.OpenEJBHttpServer.
>> process(OpenEJBHttpServer.java:261)
>> at org.apache.openejb.server.httpd.OpenEJBHttpServer.
>> processRequest(OpenEJBHttpServer.java:194)
>> at org.apache.openejb.server.httpd.OpenEJBHttpServer.
>> service(OpenEJBHttpServer.java:109)
>> at org.apache.openejb.server.httpd.HttpEjbServer.service(
>> HttpEjbServer.java:63)
>> at org.apache.openejb.server.ServerServiceFilter.service(
>> ServerServiceFilter.java:65)
>> at org.apache.openejb.server.ServerServiceFilter.service(
>> ServerServiceFilter.java:65)
>> at org.apache.openejb.server.ServiceStats.service(
>> ServiceStats.java:54)
>> at org.apache.openejb.server.ServerServiceFilter.service(
>> ServerServiceFilter.java:65)
>> at org.apache.openejb.server.ServiceLogger.service(
>> ServiceLogger.java:93)
>> at org.apache.openejb.server.ServerServiceFilter.service(
>> ServerServiceFilter.java:65)
>> at org.apache.openejb.server.ServicePool.access$201(
>> ServicePool.java:39)
>> at org.apache.openejb.server.ServicePool$3.run(ServicePool.
>> java:198)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> Caused by: org.hibernate.exception.LockAcquisitionException: could not
>> execute query
>> at org.hibernate.exception.internal.SQLExceptionTypeDelegate.
>> convert(SQLExceptionTypeDelegate.java:94)
>> at org.hibernate.exception.internal.
>> StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.
>> java:49)
>> at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(
>> SqlExceptionHelper.java:125)
>> at org.hibernate.loader.Loader.doList(Loader.java:2551)
>> at org.hibernate.loader.Loader.doList(Loader.java:2534)
>> at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.
>> java:2364)
>> at org.hibernate.loader.Loader.list(Loader.java:2359)
>> at org.hibernate.loader.criteria.CriteriaLoader.list(
>> CriteriaLoader.java:124)
>> at org.hibernate.internal.SessionImpl.list(SessionImpl.
>> java:1587)
>> at org.hibernate.internal.CriteriaImpl.list(
>> CriteriaImpl.java:374)
>> at org.hibernate.internal.CriteriaImpl.uniqueResult(
>> CriteriaImpl.java:396)
>> at com.netgiro.centralizedbatch.db.dao.BatchOrderDao.getCount(
>> BatchOrderDao.java:223)
>> at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.openejb.core.interceptor.
>> ReflectionInvocationContext$Invocation.invoke(
>> ReflectionInvocationContext.java:192)
>> at org.apache.openejb.core.interceptor.
>> ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:173)
>> at com.netgiro.centralizedbatch.interceptors.
>> AuditLogInterceptor.aroundInvoke(AuditLogInterceptor.java:22)
>> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.openejb.core.interceptor.
>> ReflectionInvocationContext$Invocation.invoke(
>> ReflectionInvocationContext.java:192)
>> at org.apache.openejb.core.interceptor.
>> ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:173)
>> at org.apache.openejb.monitoring.StatsInterceptor.record(
>> StatsInterceptor.java:181)
>> at org.apache.openejb.monitoring.StatsInterceptor.invoke(
>> StatsInterceptor.java:100)
>> at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:606)
>> at org.apache.openejb.core.interceptor.
>> ReflectionInvocationContext$Invocation.invoke(
>> ReflectionInvocationContext.java:192)
>> at org.apache.openejb.core.interceptor.
>> ReflectionInvocationContext.proceed(ReflectionInvocationContext.java:173)
>> at org.apache.openejb.core.interceptor.InterceptorStack.
>> invoke(InterceptorStack.java:85)
>> at org.apache.openejb.core.stateless.StatelessContainer._
>> invoke(StatelessContainer.java:227)
>> at org.apache.openejb.core.stateless.StatelessContainer.
>> invoke(StatelessContainer.java:194)
>> at org.apache.openejb.core.ivm.EjbObjectProxyHandler.
>> synchronizedBusinessMethod(EjbObjectProxyHandler.java:308)
>> at org.apache.openejb.core.ivm.EjbObjectProxyHandler.
>> businessMethod(EjbObjectProxyHandler.java:303)
>> at org.apache.openejb.core.ivm.EjbObjectProxyHandler._invoke(
>> EjbObjectProxyHandler.java:92)
>> at org.apache.openejb.core.ivm.BaseEjbProxyHandler.invoke(
>> BaseEjbProxyHandler.java:308)
>> ... 57 more
>> Caused by: com.ibm.db2.jcc.am.po: DB2 SQL Error: SQLCODE=-911,
>> SQLSTATE=40001, SQLERRMC=68, DRIVER=4.7.112
>> at com.ibm.db2.jcc.am.dd.a(dd.java:663)
>> at com.ibm.db2.jcc.am.dd.a(dd.java:60)
>> at com.ibm.db2.jcc.am.dd.a(dd.java:127)
>> at com.ibm.db2.jcc.am.dm.b(dm.java:3751)
>> at com.ibm.db2.jcc.t4.eb.h(eb.java:278)
>> at com.ibm.db2.jcc.t4.eb.a(eb.java:239)
>> at com.ibm.db2.jcc.t4.eb.c(eb.java:31)
>> at com.ibm.db2.jcc.t4.u.a(u.java:32)
>> at com.ibm.db2.jcc.t4.j.Zb(j.java:259)
>> at com.ibm.db2.jcc.am.dm.X(dm.java:3544)
>> at com.ibm.db2.jcc.t4.d.f(d.java:1891)
>> at com.ibm.db2.jcc.am.ic.a(ic.java:201)
>> at com.ibm.db2.jcc.t4.d.a(d.java:109)
>> at com.ibm.db2.jcc.am.dm.c(dm.java:367)
>> at com.ibm.db2.jcc.am.dm.next(dm.java:294)
>> at org.apache.commons.dbcp.DelegatingResultSet.next(
>> DelegatingResultSet.java:207)
>> at org.apache.commons.dbcp.DelegatingResultSet.next(
>> DelegatingResultSet.java:207)
>> at org.hibernate.loader.Loader.processResultSet(Loader.java:946)
>> at org.hibernate.loader.Loader.doQuery(Loader.java:917)
>> at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCol
>> lections(Loader.java:348)
>> at org.hibernate.loader.Loader.doList(Loader.java:2548)
>> ... 90 more
>>
>> dec 18, 2014 2:21:12 EM org.apache.openejb.core.ivm.EjbObjectProxyHandler$1
>> call
>> SEVERE: EjbObjectProxyHandler: Asynchronous call to
>> 'BatchJobSchedulerService' on 'scheduleBatchOrder' failed
>> org.apache.openejb.ApplicationException:
>> org.apache.openejb.core.transaction.TransactionRolledbackException:
>> Transaction was rolled back, presumably because setRollbackOnly was
>> called during a synchronization: Transaction was rolled back,
>> presumably because setRollbackOnly was called during a synchronization
>> at org.apache.openejb.core.transaction.JtaTransactionPolicy.
>> completeTransaction(JtaTransactionPolicy.java:339)
>> at org.apache.openejb.core.transaction.TxRequired.commit(
>> TxRequired.java:76)
>> at org.apache.openejb.core.transaction.EjbTransactionUtil.
>> afterInvoke(EjbTransactionUtil.java:77)
>> at org.apache.openejb.core.stateless.StatelessContainer._
>> invoke(StatelessContainer.java:246)
>> at org.apache.openejb.core.stateless.StatelessContainer.
>> invoke(StatelessContainer.java:194)
>> at org.apache.openejb.core.ivm.EjbObjectProxyHandler.
>> synchronizedBusinessMethod(EjbObjectProxyHandler.java:308)
>> at org.apache.openejb.core.ivm.EjbObjectProxyHandler$1.call(
>> EjbObjectProxyHandler.java:284)
>> at org.apache.openejb.async.AsynchronousPool$
>> AsynchronousCall.call(AsynchronousPool.java:111)
>> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
>> at java.util.concurrent.ThreadPoolExecutor.runWorker(
>> ThreadPoolExecutor.java:1145)
>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
>> ThreadPoolExecutor.java:615)
>> at java.lang.Thread.run(Thread.java:744)
>> Caused by: org.apache.openejb.core.transaction.
>> TransactionRolledbackException:
>> Transaction was rolled back, presumably because setRollbackOnly was
>> called during a synchronization
>> at org.apache.openejb.core.transaction.JtaTransactionPolicy.
>> completeTransaction(JtaTransactionPolicy.java:338)
>> ... 11 more
>> Caused by: javax.transaction.RollbackException: Unable to commit:
>> transaction marked for rollback
>> at org.apache.geronimo.transaction.manager.
>> TransactionImpl.commit(TransactionImpl.java:272)
>> at org.apache.geronimo.transaction.manager.
>> TransactionManagerImpl.commit(TransactionManagerImpl.java:252)
>> at org.apache.openejb.core.transaction.JtaTransactionPolicy.
>> completeTransaction(JtaTransactionPolicy.java:331)
>> ... 11 more
>> Caused by: java.lang.Exception: Transaction has timed out
>> at org.apache.geronimo.transaction.manager.
>> TransactionImpl.commit(TransactionImpl.java:266)
>> ... 13 more
>>
>>
> --
> Andy Gumbrecht
> https://twitter.com/AndyGeeDe
> http://www.tomitribe.com
>
>
--
Thanks and Regards
N Radhakrishna Kalyan
P: +46 733 312 584
http://about.me/nrkkalyan
<http://about.me/nrkkalyan>
