Yeah you're right the test failed again on the CI-Server as well. There're also couple of unresolved concurrency/deadlock related JIRA tickets by OpenJPA going into the exactly same direction:
https://issues.apache.org/jira/browse/OPENJPA-1716 Babak Christian Mueller wrote > It's still failing sometimes on my machine: :-( > > Results : > Failed tests: > > JpaProducerConcurrentTest.testConcurrentProducers:45->doSendMessages:66->CamelTestSupport.assertMockEndpointsSatisfied:682 > mock://result Received message count. Expected: <10> but was: <8> > > Tests run: 49, Failures: 1, Errors: 0, Skipped: 0 > > [INFO] > ------------------------------------------------------------------------ > [INFO] BUILD FAILURE > [INFO] > ------------------------------------------------------------------------ > > > Best, > Christian > ----------------- > > Software Integration Specialist > > Apache Member > V.P. Apache Camel | Apache Camel PMC Member | Apache Camel committer > Apache Incubator PMC Member > > https://www.linkedin.com/pub/christian-mueller/11/551/642 > > > On Sat, Oct 12, 2013 at 11:07 PM, Babak Vahdat > < > babak.vahdat@ > >wrote: > >> Yeah indeed a deadlock inside the OpenJPA code is the root cause of the >> problem (org.apache.openjpa.kernel.BrokerImpl#lock()). I just commited a >> fix >> regarding this, so let's keep an eye on this test by the next builds. >> Following a screenshot of the proven deadlock through jconsole: >> >> < >> http://camel.465427.n5.nabble.com/file/n5741448/Screen_Shot_2013-10-12_at_11.02.58_PM.png >> > >> >> You can enforce the deadlock if you would make use of a high log level >> (log4j.properties), e.g. TRACE instead of just INFO. >> >> Babak >> >> >> Claus Ibsen-2 wrote >> > It could be due an OpenJPA upgrade. These tests used to work better in >> the >> > past. >> > >> > On Sat, Oct 12, 2013 at 10:05 AM, Babak Vahdat >> > < >> >> > babak.vahdat@ >> >> > > wrote: >> >> The test failed again on the CI-Server. I will look into this in the >> next >> >> days to see if I can find the root cause as on my box I can't >> reproduce >> >> this. >> >> >> >> Babak >> >> >> >> >> >> Babak Vahdat wrote >> >>> Hi >> >>> >> >>> @Christian I believe I've fixed the JPA test now, could you please >> check >> >>> to see if you can still reproduce the same exception you mentioned >> >>> below? >> >>> >> >>> Babak >> >>> Christian Mueller wrote >> >>>> Today, it looks much better [1]. We are trying to fix the unit tests >> >>>> which >> >>>> are failing from time to time on slower boxes. But this isn't so >> easy >> >>>> for >> >>>> all of them... >> >>>> >> >>>> I had a look at >> >>>> >> org.apache.camel.processor.jpa.JpaProducerConcurrentTest.testConcurrentProducers. >> >>>> A few times I get the following exception which results in missing >> >>>> exchanges in the mock:result endpoint: >> >>>> >> >>>> 2013-10-11 14:01:10,623 [pool-2-thread-2] ERROR >> >>>> DefaultErrorHandler - Failed delivery for (MessageId: >> >>>> ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-4 on >> >>>> ExchangeId: >> >>>> ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-9). >> Exhausted >> >>>> after delivery attempt: 1 caught: >> >>>> org.springframework.dao.InvalidDataAccessApiUsageException: This >> >>>> operation >> >>>> cannot be performed while a Transaction is active.; nested exception >> is >> >>>> >> > > <openjpa-2.2.2-r422266:1468616 nonfatal user error> >> >>>> org.apache.openjpa.persistence.InvalidStateException: This operation >> >>>> cannot >> >>>> be performed while a Transaction is active. >> >>>> >> >>>> Message History >> >>>> >> --------------------------------------------------------------------------------------------------------------------------------------- >> >>>> RouteId ProcessorId >> >>>> Processor >> >>>> Elapsed (ms) >> >>>> [route1 ] [route1 ] >> >>>> [direct://start >> >>>> ] [ 68] >> >>>> [route1 ] [to1 ] >> >>>> [jpa://org.apache.camel.examples.SendEmail >> >>>> ] [ 67] >> >>>> >> >>>> Exchange >> >>>> >> --------------------------------------------------------------------------------------------------------------------------------------- >> >>>> Exchange[ >> >>>> Id >> >>>> ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-9 >> >>>> ExchangePattern InOnly >> >>>> Headers >> >>>> >> {breadcrumbId=ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-4, >> >>>> >> CamelEntityManager=org.apache.openjpa.persistence.EntityManagerImpl@6e453dd5 >> , >> >>>> CamelRedelivered=false, CamelRedeliveryCounter=0} >> >>>> BodyType org.apache.camel.examples.SendEmail >> >>>> Body SendEmail[id: 102 address: null] >> >>>> ] >> >>>> >> >>>> Stacktrace >> >>>> >> --------------------------------------------------------------------------------------------------------------------------------------- >> >>>> org.springframework.dao.InvalidDataAccessApiUsageException: This >> >>>> operation >> >>>> cannot be performed while a Transaction is active.; nested exception >> is >> >>>> >> > > <openjpa-2.2.2-r422266:1468616 nonfatal user error> >> >>>> org.apache.openjpa.persistence.InvalidStateException: This operation >> >>>> cannot >> >>>> be performed while a Transaction is active. >> >>>> at >> >>>> >> org.springframework.orm.jpa.EntityManagerFactoryUtils.convertJpaAccessExceptionIfPossible(EntityManagerFactoryUtils.java:293) >> >>>> at >> >>>> >> org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerSynchronization.convertException(ExtendedEntityManagerCreator.java:501) >> >>>> at >> >>>> >> org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerSynchronization.afterCommit(ExtendedEntityManagerCreator.java:482) >> >>>> at >> >>>> >> org.springframework.transaction.support.TransactionSynchronizationUtils.invokeAfterCommit(TransactionSynchronizationUtils.java:133) >> >>>> at >> >>>> >> org.springframework.transaction.support.TransactionSynchronizationUtils.triggerAfterCommit(TransactionSynchronizationUtils.java:121) >> >>>> at >> >>>> >> org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerAfterCommit(AbstractPlatformTransactionManager.java:951) >> >>>> at >> >>>> >> org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:797) >> >>>> at >> >>>> >> org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724) >> >>>> at >> >>>> >> org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:148) >> >>>> at >> >>>> >> org.apache.camel.component.jpa.JpaProducer.process(JpaProducer.java:53) >> >>>> at >> >>>> >> org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) >> >>>> at >> >>>> >> org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:132) >> >>>> at >> >>>> >> org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:307) >> >>>> at >> >>>> >> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:127) >> >>>> at >> >>>> >> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398) >> >>>> at >> >>>> >> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) >> >>>> at >> org.apache.camel.processor.Pipeline.process(Pipeline.java:118) >> >>>> at org.apache.camel.processor.Pipeline.process(Pipeline.java:80) >> >>>> at >> >>>> >> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) >> >>>> at >> >>>> >> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:56) >> >>>> at >> >>>> >> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) >> >>>> at >> >>>> >> org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:73) >> >>>> at >> >>>> >> org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:378) >> >>>> at >> >>>> >> org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:1) >> >>>> at >> >>>> >> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:242) >> >>>> at >> >>>> >> org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:346) >> >>>> at >> org.apache.camel.impl.ProducerCache.send(ProducerCache.java:184) >> >>>> at >> >>>> >> org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:124) >> >>>> at >> >>>> >> org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:137) >> >>>> at >> >>>> >> org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:144) >> >>>> at >> >>>> >> org.apache.camel.processor.jpa.JpaProducerConcurrentTest$1.call(JpaProducerConcurrentTest.java:58) >> >>>> at >> >>>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >> >>>> at >> >>>> >> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >> >>>> at >> >>>> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >> >>>> at java.lang.Thread.run(Thread.java:680) >> >>>> Caused by: >> >>>> >> > > <openjpa-2.2.2-r422266:1468616 nonfatal user error> >> >>>> org.apache.openjpa.persistence.InvalidStateException: This operation >> >>>> cannot >> >>>> be performed while a Transaction is active. >> >>>> at >> org.apache.openjpa.kernel.BrokerImpl.begin(BrokerImpl.java:1369) >> >>>> at >> >>>> >> org.apache.openjpa.kernel.DelegatingBroker.begin(DelegatingBroker.java:925) >> >>>> at >> >>>> >> org.apache.openjpa.persistence.EntityManagerImpl.begin(EntityManagerImpl.java:565) >> >>>> at >> >>>> >> org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.enlistInCurrentTransaction(ExtendedEntityManagerCreator.java:422) >> >>>> at >> >>>> >> org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.doJoinTransaction(ExtendedEntityManagerCreator.java:399) >> >>>> at >> >>>> >> org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke(ExtendedEntityManagerCreator.java:351) >> >>>> at com.sun.proxy.$Proxy18.joinTransaction(Unknown Source) >> >>>> at >> >>>> >> org.apache.camel.component.jpa.JpaProducer$1.doInTransaction(JpaProducer.java:55) >> >>>> at >> >>>> >> org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:131) >> >>>> ... 27 more >> >>>> >> >>>> which is throwing from the line 55 in >> >>>> org.apache.camel.component.jpa.JpaProducer >> >>>> entityManager.joinTransaction(); >> >>>> >> >>>> I think this is a concurrency issue we have to investigate (not sure >> >>>> whether the issue is in Camel, Spring, OpenJPA, ...). Will open a >> JIRA >> >>>> for >> >>>> it. >> >>>> >> >>>> [1] https://builds.apache.org/job/Camel.trunk.fulltest/1562/ >> >>>> >> >>>> Best, >> >>>> >> >>>> Christian >> >>>> ----------------- >> >>>> >> >>>> Software Integration Specialist >> >>>> >> >>>> Apache Member >> >>>> V.P. Apache Camel | Apache Camel PMC Member | Apache Camel committer >> >>>> Apache Incubator PMC Member >> >>>> >> >>>> https://www.linkedin.com/pub/christian-mueller/11/551/642 >> >>>> >> >>>> >> >>>> On Thu, Oct 10, 2013 at 12:28 PM, James Carman >> >>>> < >> >> >> >>>> james@ >> >> >> >>>> >wrote: >> >>>> >> >>>>> There are quite a few broken tests. Jetty seems to be hosed. JPA >> has >> >>>>> a >> >>>>> failed test. CDI has issues. It looks like JMS and FTP even >> decided >> >>>>> to >> >>>>> join the party. This has been going on for a while now. This >> >>>>> particular >> >>>>> job hasn't been "green" since 9/29. Should we stop working other >> >>>>> issues >> >>>>> and get ourselves whole again? >> >>>>> >> >>>>> https://builds.apache.org/job/Camel.trunk.fulltest/1561/ >> >>>>> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> View this message in context: >> >> >> http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741403.html >> >> Sent from the Camel Development mailing list archive at Nabble.com. >> > >> > >> > >> > -- >> > Claus Ibsen >> > ----------------- >> > Red Hat, Inc. >> > Email: >> >> > cibsen@ >> >> > Twitter: davsclaus >> > Blog: http://davsclaus.com >> > Author of Camel in Action: http://www.manning.com/ibsen >> >> >> >> >> >> -- >> View this message in context: >> http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741448.html >> Sent from the Camel Development mailing list archive at Nabble.com. >> -- View this message in context: http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741510.html Sent from the Camel Development mailing list archive at Nabble.com.
