It works now on my machine. Thanks for taking care of it. Best,
Christian ----------------- Software Integration Specialist Apache Member V.P. Apache Camel | Apache Camel PMC Member | Apache Camel committer Apache Incubator PMC Member https://www.linkedin.com/pub/christian-mueller/11/551/642 On Sat, Oct 19, 2013 at 2:30 AM, Babak Vahdat <[email protected]>wrote: > Christian, > > I commited fix for this both on the master as well as the 2.12.x branch. > Would you please give it another try to see if you can still reproduce the > test failures you mentioned by your previous posts? > > Babak > > > Christian Mueller wrote > > Really good catch! > > > > Best, > > > > Christian > > ----------------- > > > > Software Integration Specialist > > > > Apache Member > > V.P. Apache Camel | Apache Camel PMC Member | Apache Camel committer > > Apache Incubator PMC Member > > > > https://www.linkedin.com/pub/christian-mueller/11/551/642 > > > > > > On Fri, Oct 18, 2013 at 11:47 PM, Babak Vahdat > > < > > > babak.vahdat@ > > > >wrote: > > > >> The root cause of the problem for the failing JPA test is CAMEL-5932 > >> through > >> which we started sharing the *same* EntityManager object across > >> JpaEndpoint, > >> all it's created producers and consumers which is wrong! JPA 2.0 Spec, > >> section 7.2: > >> > >> An entity manager must not be shared among multiple concurrently > >> executing > >> threads, as the entity manager and persistence context are not required > >> to > >> be threadsafe. Entity managers must only be accessed in a > single-threaded > >> manner. > >> > >> That said using OpenJPA you could make use of a proprietary property to > >> make > >> OpenJPA EntityManager-impl thread safe: > >> > >> > > <property name="openjpa.Multithreaded" value="true"/> > >> > >> But even then, what about Camel users whose JPA provider is Eclipselink > >> or > >> Hibernate-EntityManager. > >> > >> I logged a ticket to get this fixed and hopfully the JPA test being > >> mentioned here as well: > >> > >> https://issues.apache.org/jira/browse/CAMEL-6874 > >> > >> Also there're also some aimless lines of code inside > >> JpaProducer#process() > >> method which I will clean up as well. > >> > >> Babak > >> > >> > >> Babak Vahdat wrote > >> > Yeah you're right the test failed again on the CI-Server as well. > >> There're > >> > also couple of unresolved concurrency/deadlock related JIRA tickets by > >> > OpenJPA going into the exactly same direction: > >> > > >> > https://issues.apache.org/jira/browse/OPENJPA-1716 > >> > > >> > Babak > >> > Christian Mueller wrote > >> >> It's still failing sometimes on my machine: :-( > >> >> > >> >> Results : > >> >> Failed tests: > >> >> > >> >> > >> > JpaProducerConcurrentTest.testConcurrentProducers:45->doSendMessages:66->CamelTestSupport.assertMockEndpointsSatisfied:682 > >> >> mock://result Received message count. Expected: <10> but was: <8> > >> >> > >> >> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0 > >> >> > >> >> [INFO] > >> >> > >> ------------------------------------------------------------------------ > >> >> [INFO] BUILD FAILURE > >> >> [INFO] > >> >> > >> ------------------------------------------------------------------------ > >> >> > >> >> > >> >> Best, > >> >> Christian > >> >> ----------------- > >> >> > >> >> Software Integration Specialist > >> >> > >> >> Apache Member > >> >> V.P. Apache Camel | Apache Camel PMC Member | Apache Camel committer > >> >> Apache Incubator PMC Member > >> >> > >> >> https://www.linkedin.com/pub/christian-mueller/11/551/642 > >> >> > >> >> > >> >> On Sat, Oct 12, 2013 at 11:07 PM, Babak Vahdat > >> >> < > >> > >> >> babak.vahdat@ > >> > >> >> >wrote: > >> >> > >> >>> Yeah indeed a deadlock inside the OpenJPA code is the root cause of > >> the > >> >>> problem (org.apache.openjpa.kernel.BrokerImpl#lock()). I just > >> commited > >> a > >> >>> fix > >> >>> regarding this, so let's keep an eye on this test by the next > builds. > >> >>> Following a screenshot of the proven deadlock through jconsole: > >> >>> > >> >>> < > >> >>> > >> > http://camel.465427.n5.nabble.com/file/n5741448/Screen_Shot_2013-10-12_at_11.02.58_PM.png > >> >>> > > >> >>> > >> >>> You can enforce the deadlock if you would make use of a high log > >> level > >> >>> (log4j.properties), e.g. TRACE instead of just INFO. > >> >>> > >> >>> Babak > >> >>> > >> >>> > >> >>> Claus Ibsen-2 wrote > >> >>> > It could be due an OpenJPA upgrade. These tests used to work > better > >> in > >> >>> the > >> >>> > past. > >> >>> > > >> >>> > On Sat, Oct 12, 2013 at 10:05 AM, Babak Vahdat > >> >>> > < > >> >>> > >> >>> > babak.vahdat@ > >> >>> > >> >>> > > wrote: > >> >>> >> The test failed again on the CI-Server. I will look into this in > >> the > >> >>> next > >> >>> >> days to see if I can find the root cause as on my box I can't > >> >>> reproduce > >> >>> >> this. > >> >>> >> > >> >>> >> Babak > >> >>> >> > >> >>> >> > >> >>> >> Babak Vahdat wrote > >> >>> >>> Hi > >> >>> >>> > >> >>> >>> @Christian I believe I've fixed the JPA test now, could you > >> please > >> >>> check > >> >>> >>> to see if you can still reproduce the same exception you > >> mentioned > >> >>> >>> below? > >> >>> >>> > >> >>> >>> Babak > >> >>> >>> Christian Mueller wrote > >> >>> >>>> Today, it looks much better [1]. We are trying to fix the unit > >> >>> tests > >> >>> >>>> which > >> >>> >>>> are failing from time to time on slower boxes. But this isn't > so > >> >>> easy > >> >>> >>>> for > >> >>> >>>> all of them... > >> >>> >>>> > >> >>> >>>> I had a look at > >> >>> >>>> > >> >>> > >> > org.apache.camel.processor.jpa.JpaProducerConcurrentTest.testConcurrentProducers. > >> >>> >>>> A few times I get the following exception which results in > >> missing > >> >>> >>>> exchanges in the mock:result endpoint: > >> >>> >>>> > >> >>> >>>> 2013-10-11 14:01:10,623 [pool-2-thread-2] ERROR > >> >>> >>>> DefaultErrorHandler - Failed delivery for > (MessageId: > >> >>> >>>> ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-4 on > >> >>> >>>> ExchangeId: > >> >>> >>>> ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-9). > >> >>> Exhausted > >> >>> >>>> after delivery attempt: 1 caught: > >> >>> >>>> org.springframework.dao.InvalidDataAccessApiUsageException: > This > >> >>> >>>> operation > >> >>> >>>> cannot be performed while a Transaction is active.; nested > >> >>> exception > >> >>> is > >> >>> >>>> > >> >>> > > >> >> > > <openjpa-2.2.2-r422266:1468616 nonfatal user error> > >> >>> >>>> org.apache.openjpa.persistence.InvalidStateException: This > >> >>> operation > >> >>> >>>> cannot > >> >>> >>>> be performed while a Transaction is active. > >> >>> >>>> > >> >>> >>>> Message History > >> >>> >>>> > >> >>> > >> > --------------------------------------------------------------------------------------------------------------------------------------- > >> >>> >>>> RouteId ProcessorId > >> >>> >>>> Processor > >> >>> >>>> Elapsed (ms) > >> >>> >>>> [route1 ] [route1 ] > >> >>> >>>> [direct://start > >> >>> >>>> ] [ 68] > >> >>> >>>> [route1 ] [to1 ] > >> >>> >>>> [jpa://org.apache.camel.examples.SendEmail > >> >>> >>>> ] [ 67] > >> >>> >>>> > >> >>> >>>> Exchange > >> >>> >>>> > >> >>> > >> > --------------------------------------------------------------------------------------------------------------------------------------- > >> >>> >>>> Exchange[ > >> >>> >>>> Id > >> >>> >>>> ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-9 > >> >>> >>>> ExchangePattern InOnly > >> >>> >>>> Headers > >> >>> >>>> > >> >>> > >> > {breadcrumbId=ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-4, > >> >>> >>>> > >> >>> > >> > CamelEntityManager=org.apache.openjpa.persistence.EntityManagerImpl@6e453dd5 > >> >>> , > >> >>> >>>> CamelRedelivered=false, CamelRedeliveryCounter=0} > >> >>> >>>> BodyType org.apache.camel.examples.SendEmail > >> >>> >>>> Body SendEmail[id: 102 address: null] > >> >>> >>>> ] > >> >>> >>>> > >> >>> >>>> Stacktrace > >> >>> >>>> > >> >>> > >> > --------------------------------------------------------------------------------------------------------------------------------------- > >> >>> >>>> org.springframework.dao.InvalidDataAccessApiUsageException: > This > >> >>> >>>> operation > >> >>> >>>> cannot be performed while a Transaction is active.; nested > >> >>> exception > >> >>> is > >> >>> >>>> > >> >>> > > >> >> > > <openjpa-2.2.2-r422266:1468616 nonfatal user error> > >> >>> >>>> org.apache.openjpa.persistence.InvalidStateException: This > >> >>> operation > >> >>> >>>> cannot > >> >>> >>>> be performed while a Transaction is active. > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.springframework.orm.jpa.EntityManagerFactoryUtils.convertJpaAccessExceptionIfPossible(EntityManagerFactoryUtils.java:293) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerSynchronization.convertException(ExtendedEntityManagerCreator.java:501) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerSynchronization.afterCommit(ExtendedEntityManagerCreator.java:482) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.springframework.transaction.support.TransactionSynchronizationUtils.invokeAfterCommit(TransactionSynchronizationUtils.java:133) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.springframework.transaction.support.TransactionSynchronizationUtils.triggerAfterCommit(TransactionSynchronizationUtils.java:121) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerAfterCommit(AbstractPlatformTransactionManager.java:951) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:797) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:148) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> org.apache.camel.component.jpa.JpaProducer.process(JpaProducer.java:53) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:132) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:307) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:127) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) > >> >>> >>>> at > >> >>> org.apache.camel.processor.Pipeline.process(Pipeline.java:118) > >> >>> >>>> at > >> >>> org.apache.camel.processor.Pipeline.process(Pipeline.java:80) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:56) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:73) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:378) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:1) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:242) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:346) > >> >>> >>>> at > >> >>> org.apache.camel.impl.ProducerCache.send(ProducerCache.java:184) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:124) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:137) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:144) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.processor.jpa.JpaProducerConcurrentTest$1.call(JpaProducerConcurrentTest.java:58) > >> >>> >>>> at > >> >>> >>>> > >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > >> >>> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > >> >>> >>>> at java.lang.Thread.run(Thread.java:680) > >> >>> >>>> Caused by: > >> >>> >>>> > >> >>> > > >> >> > > <openjpa-2.2.2-r422266:1468616 nonfatal user error> > >> >>> >>>> org.apache.openjpa.persistence.InvalidStateException: This > >> >>> operation > >> >>> >>>> cannot > >> >>> >>>> be performed while a Transaction is active. > >> >>> >>>> at > >> >>> org.apache.openjpa.kernel.BrokerImpl.begin(BrokerImpl.java:1369) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.openjpa.kernel.DelegatingBroker.begin(DelegatingBroker.java:925) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.openjpa.persistence.EntityManagerImpl.begin(EntityManagerImpl.java:565) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.enlistInCurrentTransaction(ExtendedEntityManagerCreator.java:422) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.doJoinTransaction(ExtendedEntityManagerCreator.java:399) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke(ExtendedEntityManagerCreator.java:351) > >> >>> >>>> at com.sun.proxy.$Proxy18.joinTransaction(Unknown Source) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.apache.camel.component.jpa.JpaProducer$1.doInTransaction(JpaProducer.java:55) > >> >>> >>>> at > >> >>> >>>> > >> >>> > >> > org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:131) > >> >>> >>>> ... 27 more > >> >>> >>>> > >> >>> >>>> which is throwing from the line 55 in > >> >>> >>>> org.apache.camel.component.jpa.JpaProducer > >> >>> >>>> entityManager.joinTransaction(); > >> >>> >>>> > >> >>> >>>> I think this is a concurrency issue we have to investigate (not > >> >>> sure > >> >>> >>>> whether the issue is in Camel, Spring, OpenJPA, ...). Will open > >> a > >> >>> JIRA > >> >>> >>>> for > >> >>> >>>> it. > >> >>> >>>> > >> >>> >>>> [1] https://builds.apache.org/job/Camel.trunk.fulltest/1562/ > >> >>> >>>> > >> >>> >>>> Best, > >> >>> >>>> > >> >>> >>>> Christian > >> >>> >>>> ----------------- > >> >>> >>>> > >> >>> >>>> Software Integration Specialist > >> >>> >>>> > >> >>> >>>> Apache Member > >> >>> >>>> V.P. Apache Camel | Apache Camel PMC Member | Apache Camel > >> >>> committer > >> >>> >>>> Apache Incubator PMC Member > >> >>> >>>> > >> >>> >>>> https://www.linkedin.com/pub/christian-mueller/11/551/642 > >> >>> >>>> > >> >>> >>>> > >> >>> >>>> On Thu, Oct 10, 2013 at 12:28 PM, James Carman > >> >>> >>>> < > >> >>> >> > >> >>> >>>> james@ > >> >>> >> > >> >>> >>>> >wrote: > >> >>> >>>> > >> >>> >>>>> There are quite a few broken tests. Jetty seems to be hosed. > >> JPA > >> >>> has > >> >>> >>>>> a > >> >>> >>>>> failed test. CDI has issues. It looks like JMS and FTP even > >> >>> decided > >> >>> >>>>> to > >> >>> >>>>> join the party. This has been going on for a while now. This > >> >>> >>>>> particular > >> >>> >>>>> job hasn't been "green" since 9/29. Should we stop working > >> other > >> >>> >>>>> issues > >> >>> >>>>> and get ourselves whole again? > >> >>> >>>>> > >> >>> >>>>> https://builds.apache.org/job/Camel.trunk.fulltest/1561/ > >> >>> >>>>> > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> >> > >> >>> >> -- > >> >>> >> View this message in context: > >> >>> >> > >> >>> > >> http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741403.html > >> >>> >> Sent from the Camel Development mailing list archive at > >> Nabble.com. > >> >>> > > >> >>> > > >> >>> > > >> >>> > -- > >> >>> > Claus Ibsen > >> >>> > ----------------- > >> >>> > Red Hat, Inc. > >> >>> > Email: > >> >>> > >> >>> > cibsen@ > >> >>> > >> >>> > Twitter: davsclaus > >> >>> > Blog: http://davsclaus.com > >> >>> > Author of Camel in Action: http://www.manning.com/ibsen > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> > >> >>> -- > >> >>> View this message in context: > >> >>> > >> http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741448.html > >> >>> Sent from the Camel Development mailing list archive at Nabble.com. > >> >>> > >> > >> > >> > >> > >> > >> -- > >> View this message in context: > >> http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741864.html > >> Sent from the Camel Development mailing list archive at Nabble.com. > >> > > > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741876.html > Sent from the Camel Development mailing list archive at Nabble.com. >
