Stephan - I’ll get a broker running and try to match your version - I think I can get it from one of my customers whose running Fuse 6.2.
While I do that - have you considered trying to reproduce this using an embedded broker that the test could control? It would make it much easier to reproduce. I don’t think running the broker locally vs remotely should increase any probably of losing messages - we shouldn’t lose any as long as the configuration is correct. It may increase the probably of an issue, but we shouldn’t lose messages. Also, just to confirm - when you’re testing this you are stopping/starting the broker in the middle of the test, not killing and restarting the broker - correct? > On Feb 5, 2016, at 12:37 AM, Stephan Burkard <sburk...@gmail.com> wrote: > > Hi Quinn > > I just tested the POM changes you posted and the second run failed (without > failover-URL). I then tested with the failover-URL and the third attempt > failed. > > The latter is no big surprise since I discovered the problem during > failover tests in a master-slave-config. I then reduced the setup to a > single broker environment and it was still there. > > My test broker is apache-activemq-5.11.0.redhat-620133, a patched Redhat > version of AMQ 5.11. As you, I also don't change the AMQ version number in > the POM, I just use a newer broker than the library version. My broker runs > on another machine than the test. Perhaps this increases the probability of > losing a message? > > Regards > Stephan > > > > > On Thu, Feb 4, 2016 at 7:06 PM, Quinn Stevenson <qu...@pronoia-solutions.com > <mailto:qu...@pronoia-solutions.com> >> wrote: > >> I tested this with a 5.9.0 broker and I am seeing messages dropped with >> the TxText, but I still have to use the failover URL or the test just stops >> after the broker is restarted. >> >> I don’t have a 5.9.1 broker to test with, so I don’t know if that would >> help, but the next oldest broker I have is 5.10.1, and it seems to be >> working with that broker. >> >> NOTE: I’m not changing the activemq-version in the POM when I change the >> broker version - I’m just starting a different broker (locally) on the same >> port. >> >> >>> On Feb 4, 2016, at 10:41 AM, Quinn Stevenson < >> qu...@pronoia-solutions.com> wrote: >>> >>> I still can’t make either test drop messages between the input and the >> output queue with the POM changes I sent, but I did find one difference >> between what you’ve done and what I normally do that changes the output I’m >> seeing - I always use a failover URL >>> >>> <property name="brokerURL" >> value="failover:(tcp://localhost:61616?wireFormat.tightEncodingEnabled=false >> <tcp://localhost:61616?wireFormat.tightEncodingEnabled=false >> <tcp://localhost:61616?wireFormat.tightEncodingEnabled=false>>)"/> >>> >>> My test broker is v 5.10.1 as well - I’ll see if it makes any difference >> with 5.9.0 >>> >>> >>> >>>> On Feb 4, 2016, at 9:52 AM, Quinn Stevenson < >> qu...@pronoia-solutions.com <mailto:qu...@pronoia-solutions.com> >> <mailto:qu...@pronoia-solutions.com <mailto:qu...@pronoia-solutions.com>>> >> wrote: >>>> >>>> It is strange - I’m trying to compare what you have in the “standard” >> version to what I did before. We tested our configs pretty heavily under >> all sorts of strange conditions to verify we weren’t looking messages, but >> we were using newer versions of Camel and ActiveMQ. >>>> >>>> So we’re on the same page - can you try your tests again with POM >> dependencies that look something like this? >>>> >>>> <properties> >>>> <camel-version>2.12.5</camel-version> >>>> <activemq-version>5.9.0</activemq-version> >>>> </properties> >>>> >>>> <dependencies> >>>> <dependency> >>>> <groupId>org.apache.activemq</groupId> >>>> <artifactId>activemq-all</artifactId> >>>> <version>${activemq-version}</version> >>>> </dependency> >>>> <dependency> >>>> <groupId>org.apache.activemq</groupId> >>>> <artifactId>activemq-pool</artifactId> >>>> <version>${activemq-version}</version> >>>> </dependency> >>>> >>>> <dependency> >>>> <groupId>org.apache.camel</groupId> >>>> <artifactId>camel-spring</artifactId> >>>> <version>${camel-version}</version> >>>> </dependency> >>>> <dependency> >>>> <groupId>org.apache.camel</groupId> >>>> <artifactId>camel-jms</artifactId> >>>> <version>${camel-version}</version> >>>> </dependency> >>>> >>>> <dependency> >>>> <groupId>org.apache.camel</groupId> >>>> <artifactId>camel-test-spring</artifactId> >>>> <version>${camel-version}</version> >>>> <scope>test</scope> >>>> </dependency> >>>> >>>> <dependency> >>>> <groupId>commons-collections</groupId> >>>> <artifactId>commons-collections</artifactId> >>>> <version>3.2.1</version> >>>> <scope>test</scope> >>>> </dependency> >>>> <dependency> >>>> <groupId>org.hamcrest</groupId> >>>> <artifactId>hamcrest-integration</artifactId> >>>> <version>1.3</version> >>>> <scope>test</scope> >>>> </dependency> >>>> >>>> </dependencies> >>>> >>>> >>>> >>>>> On Feb 4, 2016, at 9:49 AM, Stephan Burkard <sburk...@gmail.com >>>>> <mailto:sburk...@gmail.com> >> <mailto:sburk...@gmail.com <mailto:sburk...@gmail.com>>> wrote: >>>>> >>>>> Hi Quinn >>>>> >>>>> The "standard" version is the big mystery. As I stated in my first >> post, a >>>>> Redhat engineer analysed a similar project (with less book-keeping and >>>>> logging stuff) and his conclusion was that as soon as a transaction >> manager >>>>> is explicitly defined, Spring JMS Template (that is used by Camel >> under the >>>>> hood) creates two of them by bug, by accident or just by strange >> behaviour. >>>>> >>>>> This conclusion was quite suprising since that meant that all our >> Camel-JMS >>>>> project are theoretically suffering from message loss. >>>>> >>>>> The "no-tx" version should definitely be OK, see also CAMEL-5055 for >> the " >>>>> lazyCreateTransactionManager" flag. The JMS transaction manager may >> not be >>>>> defined but it creates one implicitly because of "transacted = true". >>>>> >>>>> The two "flaws" you mentioned are perhaps an issue. It would be somehow >>>>> calming if it is my project who has a flaw. >>>>> >>>>> Regards >>>>> Stephan >>>>> >>>>> >>>>> >>>>> On Thu, Feb 4, 2016 at 4:44 PM, Quinn Stevenson < >> qu...@pronoia-solutions.com <mailto:qu...@pronoia-solutions.com> >> <mailto:qu...@pronoia-solutions.com <mailto:qu...@pronoia-solutions.com>> >>>>>> wrote: >>>>> >>>>>> I’m still going through the project, but the first couple of things >> that >>>>>> jump out at me are you have two Spring versions - the one you >> explicitly >>>>>> put in your POM (3.2.8.RELEASE) and the one pulled in by camel-spring >>>>>> (3.2.11.RELEASE). Also, camel-spring should be included in the POM >> since >>>>>> you’re using Spring routes. I’m not sure if that’s enough to cause >> issues >>>>>> or not. >>>>>> >>>>>> I believe what’s going on with the “no-tx” version is you’re actually >>>>>> using JMS transactions since you still have transacted set to true in >> the >>>>>> JmsConfiguration. >>>>>> >>>>>> I’m not sure what’s going in with the “standard” version - it looks >>>>>> similar to some XA stuff I’ve setup before (because I had multiple >> brokers >>>>>> involved) except I had to use XA Connection Factories. >>>>>> >>>>>> >>>>>> >>>>>>> On Feb 3, 2016, at 3:12 PM, Stephan Burkard <sburk...@gmail.com >>>>>>> <mailto:sburk...@gmail.com> >> <mailto:sburk...@gmail.com <mailto:sburk...@gmail.com>>> wrote: >>>>>>> >>>>>>> Yes, same broker. There is only one ActiveMQ connection config in the >>>>>>> project. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 3, 2016 at 8:00 PM, Quinn Stevenson < >>>>>> qu...@pronoia-solutions.com <mailto:qu...@pronoia-solutions.com> >>>>>> <mailto:qu...@pronoia-solutions.com <mailto:qu...@pronoia-solutions.com>> >>>>>>>> wrote: >>>>>>> >>>>>>>> Are both the source and destination queues hosted by the same >> ActiveMQ >>>>>>>> broker? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Feb 3, 2016, at 8:21 AM, Stephan Burkard <sburk...@gmail.com >>>>>>>>> <mailto:sburk...@gmail.com> >> <mailto:sburk...@gmail.com <mailto:sburk...@gmail.com>>> >>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi >>>>>>>>> >>>>>>>>> I have built a small Maven project (attached) to demonstrate a JMS >>>>>>>> transaction problem in Camel routes under certain load conditions. >> In >>>>>> fact >>>>>>>> I am losing messages between two queues. >>>>>>>>> >>>>>>>>> The project contains two different flavours of the same test. One >> of >>>>>>>> them suffers from the problem, the other (due to my tests) not. >>>>>>>>> >>>>>>>>> >>>>>>>>> *** What does the testcase? >>>>>>>>> 1. Produces 1000 messages (100/s) and sends them to an "input" >> queue. >>>>>>>>> 2. Sends the messages from the "input" queue to an "output" queue. >>>>>>>>> 3. Finally consumes the messages from the "output" queue to count >> them. >>>>>>>>> >>>>>>>>> >>>>>>>>> *** What is the difference between the two test flavours? >>>>>>>>> - There is a "standard" flavour that suffers from the problem >>>>>>>>> - And there is a "noTxManager" flavour that seems to not have the >>>>>> problem >>>>>>>>> - The "standard" flavour is kind of a well known Camel/ActiveMQ >>>>>>>> configuration >>>>>>>>> - with a Spring transaction manager >>>>>>>>> - with a Spring transaction policy >>>>>>>>> - With a "transacted" flag in Camel routes >>>>>>>>> - The "noTxManager" flavour is a "simple" configuration >>>>>>>>> - no Spring transaction manager >>>>>>>>> - no Spring transaction policy >>>>>>>>> - no "transacted" flag in Camel routes >>>>>>>>> - BUT: "lazyCreateTransactionManager" = false (so routes are >>>>>>>> transacted too) >>>>>>>>> >>>>>>>>> >>>>>>>>> *** How to run the testcases? >>>>>>>>> 1. Replace "[yourBrokerHost]" with the hostname of your ActiveMQ >> broker >>>>>>>>> 2. Run the testcase as JUnit test >>>>>>>>> 3. When you see lots of console messages that messages are sent, >> stop >>>>>>>> your ActiveMQ broker (do not kill-9 it, just shut it down normally) >>>>>>>>> 4. Exceptions are thrown on the console output >>>>>>>>> 5. After some seconds start your broker again >>>>>>>>> 6. The test finish normally and after some seconds dumps a book >> keeping >>>>>>>> on the console >>>>>>>>> >>>>>>>>> >>>>>>>>> *** How to interpret the results? >>>>>>>>> - When the test is successful, no message is lost. You can run the >> test >>>>>>>> without broker shutdown/startup and it will obviously always be >>>>>> successful. >>>>>>>>> - When the test fails, one or more messages are lost between queue >>>>>>>> "input" and "output". In my tests I was not able to run the >> "standard" >>>>>>>> flavour three times in a row successfully. About every second run >>>>>> failed. >>>>>>>> In contrast, the "noTxManager" flavour never failed in my tests. >>>>>>>>> >>>>>>>>> The book keeping for a failed test looks like the following. In >> this >>>>>>>> example Message number 281 is arrived at the input queue but not at >> the >>>>>>>> output queue. So it is lost. >>>>>>>>> >>>>>>>>> Messages created by Client: 1000 >>>>>>>>> Client Exceptions during send: 0 [] >>>>>>>>> >>>>>>>>> Messages received at input queue: 993 >>>>>>>>> Missing Messages at input queue: 7 >> [282,283,284,285,286,287,288] >>>>>>>>> Duplicate Messages at input queue: 0 [] >>>>>>>>> >>>>>>>>> Messages received at output queue: 992 >>>>>>>>> Missing Messages at output queue: 8 >>>>>> [281,282,283,284,285,286,287,288] >>>>>>>>> Duplicate Messages at output queue: 0 [] >>>>>>>>> >>>>>>>>> Lost Messages between Queues: 1 [281] >>>>>>>>> >>>>>>>>> >>>>>>>>> *** What is the problem? >>>>>>>>> A Redhat engineer tracked the problem down to a Spring JMS template >>>>>>>> behaviour that is kind of strange. If a Spring transaction manager >> is >>>>>>>> defined in the config, it will end up with two of them. Therefore >> the >>>>>> small >>>>>>>> time range where messages can get lost that arises only when you >> have a >>>>>>>> certain load. >>>>>>>>> >>>>>>>>> >>>>>>>>> *** So, what is my question? >>>>>>>>> - Does this really mean that it is unsafe to use the "standard" >> flavour >>>>>>>> of configuration? >>>>>>>>> - Is there another config with TxManager etc that works correctly? >>>>>>>>> - What are limits of the "noTxManager" config? When is it not >>>>>> sufficent? >>>>>>>>> >>>>>>>>> Regards >>>>>>>>> Stephan >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> <CamelAmqTxTest.zip>
