CAMEL-5932 also broke one of our distribution examples, so logged another ticket to get that fixed as well:
https://issues.apache.org/jira/browse/CAMEL-6877 Babak Am 20.10.13 21:59 schrieb "Christian Müller" unter <christian.muel...@gmail.com>: >It works now on my machine. >Thanks for taking care of it. > >Best, > >Christian >----------------- > >Software Integration Specialist > >Apache Member >V.P. Apache Camel | Apache Camel PMC Member | Apache Camel committer >Apache Incubator PMC Member > >https://www.linkedin.com/pub/christian-mueller/11/551/642 > > >On Sat, Oct 19, 2013 at 2:30 AM, Babak Vahdat ><babak.vah...@swissonline.ch>wrote: > >> Christian, >> >> I commited fix for this both on the master as well as the 2.12.x branch. >> Would you please give it another try to see if you can still reproduce >>the >> test failures you mentioned by your previous posts? >> >> Babak >> >> >> Christian Mueller wrote >> > Really good catch! >> > >> > Best, >> > >> > Christian >> > ----------------- >> > >> > Software Integration Specialist >> > >> > Apache Member >> > V.P. Apache Camel | Apache Camel PMC Member | Apache Camel committer >> > Apache Incubator PMC Member >> > >> > https://www.linkedin.com/pub/christian-mueller/11/551/642 >> > >> > >> > On Fri, Oct 18, 2013 at 11:47 PM, Babak Vahdat >> > < >> >> > babak.vahdat@ >> >> > >wrote: >> > >> >> The root cause of the problem for the failing JPA test is CAMEL-5932 >> >> through >> >> which we started sharing the *same* EntityManager object across >> >> JpaEndpoint, >> >> all it's created producers and consumers which is wrong! JPA 2.0 >>Spec, >> >> section 7.2: >> >> >> >> An entity manager must not be shared among multiple concurrently >> >> executing >> >> threads, as the entity manager and persistence context are not >>required >> >> to >> >> be threadsafe. Entity managers must only be accessed in a >> single-threaded >> >> manner. >> >> >> >> That said using OpenJPA you could make use of a proprietary property >>to >> >> make >> >> OpenJPA EntityManager-impl thread safe: >> >> >> >> >> > <property name="openjpa.Multithreaded" value="true"/> >> >> >> >> But even then, what about Camel users whose JPA provider is >>Eclipselink >> >> or >> >> Hibernate-EntityManager. >> >> >> >> I logged a ticket to get this fixed and hopfully the JPA test being >> >> mentioned here as well: >> >> >> >> https://issues.apache.org/jira/browse/CAMEL-6874 >> >> >> >> Also there're also some aimless lines of code inside >> >> JpaProducer#process() >> >> method which I will clean up as well. >> >> >> >> Babak >> >> >> >> >> >> Babak Vahdat wrote >> >> > Yeah you're right the test failed again on the CI-Server as well. >> >> There're >> >> > also couple of unresolved concurrency/deadlock related JIRA >>tickets by >> >> > OpenJPA going into the exactly same direction: >> >> > >> >> > https://issues.apache.org/jira/browse/OPENJPA-1716 >> >> > >> >> > Babak >> >> > Christian Mueller wrote >> >> >> It's still failing sometimes on my machine: :-( >> >> >> >> >> >> Results : >> >> >> Failed tests: >> >> >> >> >> >> >> >> >> >>JpaProducerConcurrentTest.testConcurrentProducers:45->doSendMessages:66-> >>CamelTestSupport.assertMockEndpointsSatisfied:682 >> >> >> mock://result Received message count. Expected: <10> but was: <8> >> >> >> >> >> >> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0 >> >> >> >> >> >> [INFO] >> >> >> >> >> >>------------------------------------------------------------------------ >> >> >> [INFO] BUILD FAILURE >> >> >> [INFO] >> >> >> >> >> >>------------------------------------------------------------------------ >> >> >> >> >> >> >> >> >> Best, >> >> >> Christian >> >> >> ----------------- >> >> >> >> >> >> Software Integration Specialist >> >> >> >> >> >> Apache Member >> >> >> V.P. Apache Camel | Apache Camel PMC Member | Apache Camel >>committer >> >> >> Apache Incubator PMC Member >> >> >> >> >> >> https://www.linkedin.com/pub/christian-mueller/11/551/642 >> >> >> >> >> >> >> >> >> On Sat, Oct 12, 2013 at 11:07 PM, Babak Vahdat >> >> >> < >> >> >> >> >> babak.vahdat@ >> >> >> >> >> >wrote: >> >> >> >> >> >>> Yeah indeed a deadlock inside the OpenJPA code is the root cause >>of >> >> the >> >> >>> problem (org.apache.openjpa.kernel.BrokerImpl#lock()). I just >> >> commited >> >> a >> >> >>> fix >> >> >>> regarding this, so let's keep an eye on this test by the next >> builds. >> >> >>> Following a screenshot of the proven deadlock through jconsole: >> >> >>> >> >> >>> < >> >> >>> >> >> >> >>http://camel.465427.n5.nabble.com/file/n5741448/Screen_Shot_2013-10-12_at >>_11.02.58_PM.png >> >> >>> > >> >> >>> >> >> >>> You can enforce the deadlock if you would make use of a high log >> >> level >> >> >>> (log4j.properties), e.g. TRACE instead of just INFO. >> >> >>> >> >> >>> Babak >> >> >>> >> >> >>> >> >> >>> Claus Ibsen-2 wrote >> >> >>> > It could be due an OpenJPA upgrade. These tests used to work >> better >> >> in >> >> >>> the >> >> >>> > past. >> >> >>> > >> >> >>> > On Sat, Oct 12, 2013 at 10:05 AM, Babak Vahdat >> >> >>> > < >> >> >>> >> >> >>> > babak.vahdat@ >> >> >>> >> >> >>> > > wrote: >> >> >>> >> The test failed again on the CI-Server. I will look into this >>in >> >> the >> >> >>> next >> >> >>> >> days to see if I can find the root cause as on my box I can't >> >> >>> reproduce >> >> >>> >> this. >> >> >>> >> >> >> >>> >> Babak >> >> >>> >> >> >> >>> >> >> >> >>> >> Babak Vahdat wrote >> >> >>> >>> Hi >> >> >>> >>> >> >> >>> >>> @Christian I believe I've fixed the JPA test now, could you >> >> please >> >> >>> check >> >> >>> >>> to see if you can still reproduce the same exception you >> >> mentioned >> >> >>> >>> below? >> >> >>> >>> >> >> >>> >>> Babak >> >> >>> >>> Christian Mueller wrote >> >> >>> >>>> Today, it looks much better [1]. We are trying to fix the >>unit >> >> >>> tests >> >> >>> >>>> which >> >> >>> >>>> are failing from time to time on slower boxes. But this >>isn't >> so >> >> >>> easy >> >> >>> >>>> for >> >> >>> >>>> all of them... >> >> >>> >>>> >> >> >>> >>>> I had a look at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.processor.jpa.JpaProducerConcurrentTest.testConcurrentPr >>oducers. >> >> >>> >>>> A few times I get the following exception which results in >> >> missing >> >> >>> >>>> exchanges in the mock:result endpoint: >> >> >>> >>>> >> >> >>> >>>> 2013-10-11 14:01:10,623 [pool-2-thread-2] ERROR >> >> >>> >>>> DefaultErrorHandler - Failed delivery for >> (MessageId: >> >> >>> >>>> ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-4 >>on >> >> >>> >>>> ExchangeId: >> >> >>> >>>> >>ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-9). >> >> >>> Exhausted >> >> >>> >>>> after delivery attempt: 1 caught: >> >> >>> >>>> org.springframework.dao.InvalidDataAccessApiUsageException: >> This >> >> >>> >>>> operation >> >> >>> >>>> cannot be performed while a Transaction is active.; nested >> >> >>> exception >> >> >>> is >> >> >>> >>>> >> >> >>> > >> >> >> >> > <openjpa-2.2.2-r422266:1468616 nonfatal user error> >> >> >>> >>>> org.apache.openjpa.persistence.InvalidStateException: This >> >> >>> operation >> >> >>> >>>> cannot >> >> >>> >>>> be performed while a Transaction is active. >> >> >>> >>>> >> >> >>> >>>> Message History >> >> >>> >>>> >> >> >>> >> >> >> >>------------------------------------------------------------------------- >>-------------------------------------------------------------- >> >> >>> >>>> RouteId ProcessorId >> >> >>> >>>> Processor >> >> >>> >>>> Elapsed (ms) >> >> >>> >>>> [route1 ] [route1 ] >> >> >>> >>>> [direct://start >> >> >>> >>>> ] [ 68] >> >> >>> >>>> [route1 ] [to1 ] >> >> >>> >>>> [jpa://org.apache.camel.examples.SendEmail >> >> >>> >>>> ] [ 67] >> >> >>> >>>> >> >> >>> >>>> Exchange >> >> >>> >>>> >> >> >>> >> >> >> >>------------------------------------------------------------------------- >>-------------------------------------------------------------- >> >> >>> >>>> Exchange[ >> >> >>> >>>> Id >> >> >>> >>>> ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-9 >> >> >>> >>>> ExchangePattern InOnly >> >> >>> >>>> Headers >> >> >>> >>>> >> >> >>> >> >> >> >>{breadcrumbId=ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-4 >>, >> >> >>> >>>> >> >> >>> >> >> >> >>CamelEntityManager=org.apache.openjpa.persistence.EntityManagerImpl@6e453 >>dd5 >> >> >>> , >> >> >>> >>>> CamelRedelivered=false, CamelRedeliveryCounter=0} >> >> >>> >>>> BodyType org.apache.camel.examples.SendEmail >> >> >>> >>>> Body SendEmail[id: 102 address: null] >> >> >>> >>>> ] >> >> >>> >>>> >> >> >>> >>>> Stacktrace >> >> >>> >>>> >> >> >>> >> >> >> >>------------------------------------------------------------------------- >>-------------------------------------------------------------- >> >> >>> >>>> org.springframework.dao.InvalidDataAccessApiUsageException: >> This >> >> >>> >>>> operation >> >> >>> >>>> cannot be performed while a Transaction is active.; nested >> >> >>> exception >> >> >>> is >> >> >>> >>>> >> >> >>> > >> >> >> >> > <openjpa-2.2.2-r422266:1468616 nonfatal user error> >> >> >>> >>>> org.apache.openjpa.persistence.InvalidStateException: This >> >> >>> operation >> >> >>> >>>> cannot >> >> >>> >>>> be performed while a Transaction is active. >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.springframework.orm.jpa.EntityManagerFactoryUtils.convertJpaAccessExc >>eptionIfPossible(EntityManagerFactoryUtils.java:293) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityMa >>nagerSynchronization.convertException(ExtendedEntityManagerCreator.java:5 >>01) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityMa >>nagerSynchronization.afterCommit(ExtendedEntityManagerCreator.java:482) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.springframework.transaction.support.TransactionSynchronizationUtils.i >>nvokeAfterCommit(TransactionSynchronizationUtils.java:133) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.springframework.transaction.support.TransactionSynchronizationUtils.t >>riggerAfterCommit(TransactionSynchronizationUtils.java:121) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.springframework.transaction.support.AbstractPlatformTransactionManage >>r.triggerAfterCommit(AbstractPlatformTransactionManager.java:951) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.springframework.transaction.support.AbstractPlatformTransactionManage >>r.processCommit(AbstractPlatformTransactionManager.java:797) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.springframework.transaction.support.AbstractPlatformTransactionManage >>r.commit(AbstractPlatformTransactionManager.java:724) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.springframework.transaction.support.TransactionTemplate.execute(Trans >>actionTemplate.java:148) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >>org.apache.camel.component.jpa.JpaProducer.process(JpaProducer.java:53) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProce >>ssorBridge.process(AsyncProcessorConverterHelper.java:61) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcesso >>r.java:132) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java: >>307) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >>org.apache.camel.processor.SendProcessor.process(SendProcessor.java:127) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryError >>Handler.java:398) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalPr >>ocessor.java:191) >> >> >>> >>>> at >> >> >>> org.apache.camel.processor.Pipeline.process(Pipeline.java:118) >> >> >>> >>>> at >> >> >>> org.apache.camel.processor.Pipeline.process(Pipeline.java:80) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalPr >>ocessor.java:191) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.component.direct.DirectProducer.process(DirectProducer.j >>ava:56) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalPr >>ocessor.java:191) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer. >>java:73) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:378 >>) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >>org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:1) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >>org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:242) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >>org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:346) >> >> >>> >>>> at >> >> >>> org.apache.camel.impl.ProducerCache.send(ProducerCache.java:184) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplat >>e.java:124) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTem >>plate.java:137) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTem >>plate.java:144) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.processor.jpa.JpaProducerConcurrentTest$1.call(JpaProduc >>erConcurrentTest.java:58) >> >> >>> >>>> at >> >> >>> >>>> >> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >> >> >>> >>>> at >>java.util.concurrent.FutureTask.run(FutureTask.java:138) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor >>.java:895) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav >>a:918) >> >> >>> >>>> at java.lang.Thread.run(Thread.java:680) >> >> >>> >>>> Caused by: >> >> >>> >>>> >> >> >>> > >> >> >> >> > <openjpa-2.2.2-r422266:1468616 nonfatal user error> >> >> >>> >>>> org.apache.openjpa.persistence.InvalidStateException: This >> >> >>> operation >> >> >>> >>>> cannot >> >> >>> >>>> be performed while a Transaction is active. >> >> >>> >>>> at >> >> >>> org.apache.openjpa.kernel.BrokerImpl.begin(BrokerImpl.java:1369) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.openjpa.kernel.DelegatingBroker.begin(DelegatingBroker.java:92 >>5) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.openjpa.persistence.EntityManagerImpl.begin(EntityManagerImpl. >>java:565) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityMa >>nagerInvocationHandler.enlistInCurrentTransaction(ExtendedEntityManagerCr >>eator.java:422) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityMa >>nagerInvocationHandler.doJoinTransaction(ExtendedEntityManagerCreator.jav >>a:399) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityMa >>nagerInvocationHandler.invoke(ExtendedEntityManagerCreator.java:351) >> >> >>> >>>> at com.sun.proxy.$Proxy18.joinTransaction(Unknown >>Source) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.apache.camel.component.jpa.JpaProducer$1.doInTransaction(JpaProducer. >>java:55) >> >> >>> >>>> at >> >> >>> >>>> >> >> >>> >> >> >> >>org.springframework.transaction.support.TransactionTemplate.execute(Trans >>actionTemplate.java:131) >> >> >>> >>>> ... 27 more >> >> >>> >>>> >> >> >>> >>>> which is throwing from the line 55 in >> >> >>> >>>> org.apache.camel.component.jpa.JpaProducer >> >> >>> >>>> entityManager.joinTransaction(); >> >> >>> >>>> >> >> >>> >>>> I think this is a concurrency issue we have to investigate >>(not >> >> >>> sure >> >> >>> >>>> whether the issue is in Camel, Spring, OpenJPA, ...). Will >>open >> >> a >> >> >>> JIRA >> >> >>> >>>> for >> >> >>> >>>> it. >> >> >>> >>>> >> >> >>> >>>> [1] https://builds.apache.org/job/Camel.trunk.fulltest/1562/ >> >> >>> >>>> >> >> >>> >>>> Best, >> >> >>> >>>> >> >> >>> >>>> Christian >> >> >>> >>>> ----------------- >> >> >>> >>>> >> >> >>> >>>> Software Integration Specialist >> >> >>> >>>> >> >> >>> >>>> Apache Member >> >> >>> >>>> V.P. Apache Camel | Apache Camel PMC Member | Apache Camel >> >> >>> committer >> >> >>> >>>> Apache Incubator PMC Member >> >> >>> >>>> >> >> >>> >>>> https://www.linkedin.com/pub/christian-mueller/11/551/642 >> >> >>> >>>> >> >> >>> >>>> >> >> >>> >>>> On Thu, Oct 10, 2013 at 12:28 PM, James Carman >> >> >>> >>>> < >> >> >>> >> >> >> >>> >>>> james@ >> >> >>> >> >> >> >>> >>>> >wrote: >> >> >>> >>>> >> >> >>> >>>>> There are quite a few broken tests. Jetty seems to be >>hosed. >> >> JPA >> >> >>> has >> >> >>> >>>>> a >> >> >>> >>>>> failed test. CDI has issues. It looks like JMS and FTP >>even >> >> >>> decided >> >> >>> >>>>> to >> >> >>> >>>>> join the party. This has been going on for a while now. >>This >> >> >>> >>>>> particular >> >> >>> >>>>> job hasn't been "green" since 9/29. Should we stop working >> >> other >> >> >>> >>>>> issues >> >> >>> >>>>> and get ourselves whole again? >> >> >>> >>>>> >> >> >>> >>>>> https://builds.apache.org/job/Camel.trunk.fulltest/1561/ >> >> >>> >>>>> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> >> >> >>> >> -- >> >> >>> >> View this message in context: >> >> >>> >> >> >> >>> >> >> >>http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741403.html >> >> >>> >> Sent from the Camel Development mailing list archive at >> >> Nabble.com. >> >> >>> > >> >> >>> > >> >> >>> > >> >> >>> > -- >> >> >>> > Claus Ibsen >> >> >>> > ----------------- >> >> >>> > Red Hat, Inc. >> >> >>> > Email: >> >> >>> >> >> >>> > cibsen@ >> >> >>> >> >> >>> > Twitter: davsclaus >> >> >>> > Blog: http://davsclaus.com >> >> >>> > Author of Camel in Action: http://www.manning.com/ibsen >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> >> >> >>> -- >> >> >>> View this message in context: >> >> >>> >> >> >>http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741448.html >> >> >>> Sent from the Camel Development mailing list archive at >>Nabble.com. >> >> >>> >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> View this message in context: >> >> >>http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741864.html >> >> Sent from the Camel Development mailing list archive at Nabble.com. >> >> >> >> >> >> >> >> -- >> View this message in context: >> http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741876.html >> Sent from the Camel Development mailing list archive at Nabble.com. >>
