Nice digging Babak. On Sat, Oct 19, 2013 at 2:30 AM, Babak Vahdat <[email protected]> wrote: > Christian, > > I commited fix for this both on the master as well as the 2.12.x branch. > Would you please give it another try to see if you can still reproduce the > test failures you mentioned by your previous posts? > > Babak > > > Christian Mueller wrote >> Really good catch! >> >> Best, >> >> Christian >> ----------------- >> >> Software Integration Specialist >> >> Apache Member >> V.P. Apache Camel | Apache Camel PMC Member | Apache Camel committer >> Apache Incubator PMC Member >> >> https://www.linkedin.com/pub/christian-mueller/11/551/642 >> >> >> On Fri, Oct 18, 2013 at 11:47 PM, Babak Vahdat >> < > >> babak.vahdat@ > >> >wrote: >> >>> The root cause of the problem for the failing JPA test is CAMEL-5932 >>> through >>> which we started sharing the *same* EntityManager object across >>> JpaEndpoint, >>> all it's created producers and consumers which is wrong! JPA 2.0 Spec, >>> section 7.2: >>> >>> An entity manager must not be shared among multiple concurrently >>> executing >>> threads, as the entity manager and persistence context are not required >>> to >>> be threadsafe. Entity managers must only be accessed in a single-threaded >>> manner. >>> >>> That said using OpenJPA you could make use of a proprietary property to >>> make >>> OpenJPA EntityManager-impl thread safe: >>> >>> >> <property name="openjpa.Multithreaded" value="true"/> >>> >>> But even then, what about Camel users whose JPA provider is Eclipselink >>> or >>> Hibernate-EntityManager. >>> >>> I logged a ticket to get this fixed and hopfully the JPA test being >>> mentioned here as well: >>> >>> https://issues.apache.org/jira/browse/CAMEL-6874 >>> >>> Also there're also some aimless lines of code inside >>> JpaProducer#process() >>> method which I will clean up as well. >>> >>> Babak >>> >>> >>> Babak Vahdat wrote >>> > Yeah you're right the test failed again on the CI-Server as well. >>> There're >>> > also couple of unresolved concurrency/deadlock related JIRA tickets by >>> > OpenJPA going into the exactly same direction: >>> > >>> > https://issues.apache.org/jira/browse/OPENJPA-1716 >>> > >>> > Babak >>> > Christian Mueller wrote >>> >> It's still failing sometimes on my machine: :-( >>> >> >>> >> Results : >>> >> Failed tests: >>> >> >>> >> >>> JpaProducerConcurrentTest.testConcurrentProducers:45->doSendMessages:66->CamelTestSupport.assertMockEndpointsSatisfied:682 >>> >> mock://result Received message count. Expected: <10> but was: <8> >>> >> >>> >> Tests run: 49, Failures: 1, Errors: 0, Skipped: 0 >>> >> >>> >> [INFO] >>> >> >>> ------------------------------------------------------------------------ >>> >> [INFO] BUILD FAILURE >>> >> [INFO] >>> >> >>> ------------------------------------------------------------------------ >>> >> >>> >> >>> >> Best, >>> >> Christian >>> >> ----------------- >>> >> >>> >> Software Integration Specialist >>> >> >>> >> Apache Member >>> >> V.P. Apache Camel | Apache Camel PMC Member | Apache Camel committer >>> >> Apache Incubator PMC Member >>> >> >>> >> https://www.linkedin.com/pub/christian-mueller/11/551/642 >>> >> >>> >> >>> >> On Sat, Oct 12, 2013 at 11:07 PM, Babak Vahdat >>> >> < >>> >>> >> babak.vahdat@ >>> >>> >> >wrote: >>> >> >>> >>> Yeah indeed a deadlock inside the OpenJPA code is the root cause of >>> the >>> >>> problem (org.apache.openjpa.kernel.BrokerImpl#lock()). I just >>> commited >>> a >>> >>> fix >>> >>> regarding this, so let's keep an eye on this test by the next builds. >>> >>> Following a screenshot of the proven deadlock through jconsole: >>> >>> >>> >>> < >>> >>> >>> http://camel.465427.n5.nabble.com/file/n5741448/Screen_Shot_2013-10-12_at_11.02.58_PM.png >>> >>> > >>> >>> >>> >>> You can enforce the deadlock if you would make use of a high log >>> level >>> >>> (log4j.properties), e.g. TRACE instead of just INFO. >>> >>> >>> >>> Babak >>> >>> >>> >>> >>> >>> Claus Ibsen-2 wrote >>> >>> > It could be due an OpenJPA upgrade. These tests used to work better >>> in >>> >>> the >>> >>> > past. >>> >>> > >>> >>> > On Sat, Oct 12, 2013 at 10:05 AM, Babak Vahdat >>> >>> > < >>> >>> >>> >>> > babak.vahdat@ >>> >>> >>> >>> > > wrote: >>> >>> >> The test failed again on the CI-Server. I will look into this in >>> the >>> >>> next >>> >>> >> days to see if I can find the root cause as on my box I can't >>> >>> reproduce >>> >>> >> this. >>> >>> >> >>> >>> >> Babak >>> >>> >> >>> >>> >> >>> >>> >> Babak Vahdat wrote >>> >>> >>> Hi >>> >>> >>> >>> >>> >>> @Christian I believe I've fixed the JPA test now, could you >>> please >>> >>> check >>> >>> >>> to see if you can still reproduce the same exception you >>> mentioned >>> >>> >>> below? >>> >>> >>> >>> >>> >>> Babak >>> >>> >>> Christian Mueller wrote >>> >>> >>>> Today, it looks much better [1]. We are trying to fix the unit >>> >>> tests >>> >>> >>>> which >>> >>> >>>> are failing from time to time on slower boxes. But this isn't so >>> >>> easy >>> >>> >>>> for >>> >>> >>>> all of them... >>> >>> >>>> >>> >>> >>>> I had a look at >>> >>> >>>> >>> >>> >>> org.apache.camel.processor.jpa.JpaProducerConcurrentTest.testConcurrentProducers. >>> >>> >>>> A few times I get the following exception which results in >>> missing >>> >>> >>>> exchanges in the mock:result endpoint: >>> >>> >>>> >>> >>> >>>> 2013-10-11 14:01:10,623 [pool-2-thread-2] ERROR >>> >>> >>>> DefaultErrorHandler - Failed delivery for (MessageId: >>> >>> >>>> ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-4 on >>> >>> >>>> ExchangeId: >>> >>> >>>> ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-9). >>> >>> Exhausted >>> >>> >>>> after delivery attempt: 1 caught: >>> >>> >>>> org.springframework.dao.InvalidDataAccessApiUsageException: This >>> >>> >>>> operation >>> >>> >>>> cannot be performed while a Transaction is active.; nested >>> >>> exception >>> >>> is >>> >>> >>>> >>> >>> > >>> >> >> <openjpa-2.2.2-r422266:1468616 nonfatal user error> >>> >>> >>>> org.apache.openjpa.persistence.InvalidStateException: This >>> >>> operation >>> >>> >>>> cannot >>> >>> >>>> be performed while a Transaction is active. >>> >>> >>>> >>> >>> >>>> Message History >>> >>> >>>> >>> >>> >>> --------------------------------------------------------------------------------------------------------------------------------------- >>> >>> >>>> RouteId ProcessorId >>> >>> >>>> Processor >>> >>> >>>> Elapsed (ms) >>> >>> >>>> [route1 ] [route1 ] >>> >>> >>>> [direct://start >>> >>> >>>> ] [ 68] >>> >>> >>>> [route1 ] [to1 ] >>> >>> >>>> [jpa://org.apache.camel.examples.SendEmail >>> >>> >>>> ] [ 67] >>> >>> >>>> >>> >>> >>>> Exchange >>> >>> >>>> >>> >>> >>> --------------------------------------------------------------------------------------------------------------------------------------- >>> >>> >>>> Exchange[ >>> >>> >>>> Id >>> >>> >>>> ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-9 >>> >>> >>>> ExchangePattern InOnly >>> >>> >>>> Headers >>> >>> >>>> >>> >>> >>> {breadcrumbId=ID-christians-macbook-pro-fritz-box-54975-1381492869188-0-4, >>> >>> >>>> >>> >>> >>> CamelEntityManager=org.apache.openjpa.persistence.EntityManagerImpl@6e453dd5 >>> >>> , >>> >>> >>>> CamelRedelivered=false, CamelRedeliveryCounter=0} >>> >>> >>>> BodyType org.apache.camel.examples.SendEmail >>> >>> >>>> Body SendEmail[id: 102 address: null] >>> >>> >>>> ] >>> >>> >>>> >>> >>> >>>> Stacktrace >>> >>> >>>> >>> >>> >>> --------------------------------------------------------------------------------------------------------------------------------------- >>> >>> >>>> org.springframework.dao.InvalidDataAccessApiUsageException: This >>> >>> >>>> operation >>> >>> >>>> cannot be performed while a Transaction is active.; nested >>> >>> exception >>> >>> is >>> >>> >>>> >>> >>> > >>> >> >> <openjpa-2.2.2-r422266:1468616 nonfatal user error> >>> >>> >>>> org.apache.openjpa.persistence.InvalidStateException: This >>> >>> operation >>> >>> >>>> cannot >>> >>> >>>> be performed while a Transaction is active. >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.springframework.orm.jpa.EntityManagerFactoryUtils.convertJpaAccessExceptionIfPossible(EntityManagerFactoryUtils.java:293) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerSynchronization.convertException(ExtendedEntityManagerCreator.java:501) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerSynchronization.afterCommit(ExtendedEntityManagerCreator.java:482) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.springframework.transaction.support.TransactionSynchronizationUtils.invokeAfterCommit(TransactionSynchronizationUtils.java:133) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.springframework.transaction.support.TransactionSynchronizationUtils.triggerAfterCommit(TransactionSynchronizationUtils.java:121) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.springframework.transaction.support.AbstractPlatformTransactionManager.triggerAfterCommit(AbstractPlatformTransactionManager.java:951) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:797) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:724) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:148) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.component.jpa.JpaProducer.process(JpaProducer.java:53) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.util.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:61) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:132) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:307) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:127) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:398) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) >>> >>> >>>> at >>> >>> org.apache.camel.processor.Pipeline.process(Pipeline.java:118) >>> >>> >>>> at >>> >>> org.apache.camel.processor.Pipeline.process(Pipeline.java:80) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:56) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.processor.UnitOfWorkProducer.process(UnitOfWorkProducer.java:73) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:378) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.impl.ProducerCache$2.doInProducer(ProducerCache.java:1) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.impl.ProducerCache.doInProducer(ProducerCache.java:242) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.impl.ProducerCache.sendExchange(ProducerCache.java:346) >>> >>> >>>> at >>> >>> org.apache.camel.impl.ProducerCache.send(ProducerCache.java:184) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.impl.DefaultProducerTemplate.send(DefaultProducerTemplate.java:124) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:137) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.impl.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:144) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.processor.jpa.JpaProducerConcurrentTest$1.call(JpaProducerConcurrentTest.java:58) >>> >>> >>>> at >>> >>> >>>> >>> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) >>> >>> >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:138) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) >>> >>> >>>> at java.lang.Thread.run(Thread.java:680) >>> >>> >>>> Caused by: >>> >>> >>>> >>> >>> > >>> >> >> <openjpa-2.2.2-r422266:1468616 nonfatal user error> >>> >>> >>>> org.apache.openjpa.persistence.InvalidStateException: This >>> >>> operation >>> >>> >>>> cannot >>> >>> >>>> be performed while a Transaction is active. >>> >>> >>>> at >>> >>> org.apache.openjpa.kernel.BrokerImpl.begin(BrokerImpl.java:1369) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.openjpa.kernel.DelegatingBroker.begin(DelegatingBroker.java:925) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.openjpa.persistence.EntityManagerImpl.begin(EntityManagerImpl.java:565) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.enlistInCurrentTransaction(ExtendedEntityManagerCreator.java:422) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.doJoinTransaction(ExtendedEntityManagerCreator.java:399) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.springframework.orm.jpa.ExtendedEntityManagerCreator$ExtendedEntityManagerInvocationHandler.invoke(ExtendedEntityManagerCreator.java:351) >>> >>> >>>> at com.sun.proxy.$Proxy18.joinTransaction(Unknown Source) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.apache.camel.component.jpa.JpaProducer$1.doInTransaction(JpaProducer.java:55) >>> >>> >>>> at >>> >>> >>>> >>> >>> >>> org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:131) >>> >>> >>>> ... 27 more >>> >>> >>>> >>> >>> >>>> which is throwing from the line 55 in >>> >>> >>>> org.apache.camel.component.jpa.JpaProducer >>> >>> >>>> entityManager.joinTransaction(); >>> >>> >>>> >>> >>> >>>> I think this is a concurrency issue we have to investigate (not >>> >>> sure >>> >>> >>>> whether the issue is in Camel, Spring, OpenJPA, ...). Will open >>> a >>> >>> JIRA >>> >>> >>>> for >>> >>> >>>> it. >>> >>> >>>> >>> >>> >>>> [1] https://builds.apache.org/job/Camel.trunk.fulltest/1562/ >>> >>> >>>> >>> >>> >>>> Best, >>> >>> >>>> >>> >>> >>>> Christian >>> >>> >>>> ----------------- >>> >>> >>>> >>> >>> >>>> Software Integration Specialist >>> >>> >>>> >>> >>> >>>> Apache Member >>> >>> >>>> V.P. Apache Camel | Apache Camel PMC Member | Apache Camel >>> >>> committer >>> >>> >>>> Apache Incubator PMC Member >>> >>> >>>> >>> >>> >>>> https://www.linkedin.com/pub/christian-mueller/11/551/642 >>> >>> >>>> >>> >>> >>>> >>> >>> >>>> On Thu, Oct 10, 2013 at 12:28 PM, James Carman >>> >>> >>>> < >>> >>> >> >>> >>> >>>> james@ >>> >>> >> >>> >>> >>>> >wrote: >>> >>> >>>> >>> >>> >>>>> There are quite a few broken tests. Jetty seems to be hosed. >>> JPA >>> >>> has >>> >>> >>>>> a >>> >>> >>>>> failed test. CDI has issues. It looks like JMS and FTP even >>> >>> decided >>> >>> >>>>> to >>> >>> >>>>> join the party. This has been going on for a while now. This >>> >>> >>>>> particular >>> >>> >>>>> job hasn't been "green" since 9/29. Should we stop working >>> other >>> >>> >>>>> issues >>> >>> >>>>> and get ourselves whole again? >>> >>> >>>>> >>> >>> >>>>> https://builds.apache.org/job/Camel.trunk.fulltest/1561/ >>> >>> >>>>> >>> >>> >> >>> >>> >> >>> >>> >> >>> >>> >> >>> >>> >> >>> >>> >> -- >>> >>> >> View this message in context: >>> >>> >> >>> >>> >>> http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741403.html >>> >>> >> Sent from the Camel Development mailing list archive at >>> Nabble.com. >>> >>> > >>> >>> > >>> >>> > >>> >>> > -- >>> >>> > Claus Ibsen >>> >>> > ----------------- >>> >>> > Red Hat, Inc. >>> >>> > Email: >>> >>> >>> >>> > cibsen@ >>> >>> >>> >>> > Twitter: davsclaus >>> >>> > Blog: http://davsclaus.com >>> >>> > Author of Camel in Action: http://www.manning.com/ibsen >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> -- >>> >>> View this message in context: >>> >>> >>> http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741448.html >>> >>> Sent from the Camel Development mailing list archive at Nabble.com. >>> >>> >>> >>> >>> >>> >>> >>> -- >>> View this message in context: >>> http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741864.html >>> Sent from the Camel Development mailing list archive at Nabble.com. >>> > > > > > > -- > View this message in context: > http://camel.465427.n5.nabble.com/Build-failures-tp5741273p5741876.html > Sent from the Camel Development mailing list archive at Nabble.com.
-- Claus Ibsen ----------------- Red Hat, Inc. Email: [email protected] Twitter: davsclaus Blog: http://davsclaus.com Author of Camel in Action: http://www.manning.com/ibsen
