Would anyone object if test framework under systests + integration tests + perftests + 'Immediate/MandatoryMessageTest', were merged with Arnauds pure JMS changes, and put under:
/trunk/qpidtests ? This code will have no Qpid dependencies, except that there will need to be an interface created for an extension point for AMQP specific stuff (like immediate/mandatory flags). One nicety of this, is that the entire test framework will also be able to be run over other JMS products, for comparative testing. On 26/09/2007, Rupert Smith <[EMAIL PROTECTED]> wrote: > > This is exactly what the test framework behind ImmediateMessageTest (and > MandatoryMessageTest) does. It is not suitable for all tests, as it works > around the concept of a 'test circuit' described in the distributed testing > proposal. By default these tests pick up an in-vm broker, when run through > vanilla JUnit. FrameworkBaseCase is the equivalent of your 'BrokerTestCase'. > > > You can run them through a different test runner, i.e., the Coordinator, > passing in a test distribution engine (-e interop or fanout at the moment), > and the test is 'projected' onto a set of distributed test nodes. > > You can override the use of the in-vm broker, by passing in the > 'broker=tcp://localhost' option. They work on the same option set as the > perftests currently do, from PingPongProducer (very misleading name, but > that's code evolution for you). > > Also, there is a 'new style' perf test, QpidTestThroughputPerf, which is > built on top of this framework, instead of PPP. > > All this stuff is at the infamous 80% done stage, which is also why I am > very keen to stop it fragmenting amongst the different branches. > > On 26/09/2007, Martin Ritchie <[EMAIL PROTECTED]> wrote: > > > > On 26/09/2007, Rupert Smith <[EMAIL PROTECTED]> wrote: > > > Arnuad, > > > > > > There is also a README.txt in the integration tests directory to > > explain its > > > purpose. > > > The difference is: > > > > > > sys tests is for testing the Java client + broker together, > > as a > > > single system. > > > integration tests is for testing the Java client, as an AMQP > > component, > > > against any broker or other clients. > > > > > > 'sys' and 'integration' may not be the ideal names. However, please > > don't > > > suggest renaming them, as it will complicate merges. > > > > > > Sys tests are run as part of the Maven build, always using in-vm > > brokers. > > > > > > Integration tests require the independent starting/stopping of a > > broker to > > > run through, as well as possibly starting/stopping test clients in > > other > > > languages. They could be automated, but it is just a bit trickier to > > do. It > > > was my original intention to automate the whole interop test cycle > > between > > > all client languages and brokers in Qpid, and a scheme for doing so is > > given > > > in the interop test spec. > > > > > > Client tests, are supposed to be pure unit tests for the client code, > > but I > > > believe they also test client against an in-vm broker? As such, they > > should > > > not be run against a remote broker. > > > > > > Perftests could be run as part of a build too, although ideally nont > > against > > > an in-vm broker. One of the problems with running perftests to > > automatically > > > check performance changes on every build, is that the results of these > > tests > > > sometimes require 'interpretation'. It would be nice to do this > > > automatically, for example outputing latency/throughput graphs to a > > Wiki > > > page, but this is a whole project in itself. At the moment, I filter > > using > > > grep, and open them in a spread sheet. > > > > > > I have a macro... > > > > > > An interesting test for you to look at might be ImmediateMessageTest. > > This > > > one can be run in-vm, against a remote broker, or even distributed > > accross > > > many test nodes, all running the exact same test case. This is > > currently > > > where I am going with the tests, also with a view to being able to run > > large > > > pub/sub tests, and adding *lots* more interop tests, all with a common > > > > > framework. > > > > > > A situation I am very keen to avoid, is divergence of the test code > > between > > > different branches. The tests should be the same accross all, to show > > that > > > all work in the same way. Its the only sensible way I can think of, to > > > > > ensure that when we eventually move from 0.8 to 0.10 that we carry > > accross > > > the behaviour from the old to the new. > > > > > > These tests should work at the surface of the product, that is through > > the > > > JMS or Qpid APIs in the respective languages. In the Java case at > > least, > > > this should be easy because of JMS, and there should be a sub-class to > > do > > > Qpid/AMQP specific stuff (perhaps two one for M2/0.8 and one for > > trunk/0.10 > > > new client). > > > > > > Perhaps we could pull some of the test code (integration + perftests + > > > Immediate/MandatoryMessageTest) out of the current M2/M2.1/trunk > > branchfest, > > > into a separate top level project? That is something that I would like > > to > > > do. Arnaud, you have already ported perftests on trunk to work through > > pure > > > JMS, so that makes it possible to do this. Thoughts? > > > > I'm all for this. The tests should be independent of the AMQP version. > > Some tests may be version specific but they could stay with there > > version. > > > > Talking about the systest for a moment: > > I have converted some tests to use a static so it is easier to point a > > single test at a broker. But it would be nice to wrap up the test > > connection setup such that we have some form of a BrokerTestCase. > > This would take a broker URL and if required starts an inVM broker. > > From there we could then get maven to pass the InVM url for build > > tests but would also allow us to set the tests to run from maven > > against an external broker. > > > > I would say a first step. would be to separate the tests that use an > > invm broker and those that don't. I'm _not_ saying that all invm and > > all non_invm test should live together only that they should be > > distinguishable. Example: in the systest module AckTest doesn't use an > > inVM broker but MandatoryMessageTest does. This first step would help > > ensure we can easily identify tests that can run against a broker. > > > > If no-one else has the time then I'd start the process by putting > > together the BrokerTestCase, or at least something we can debate about > > :) > > > > > Rupert > > > > > > On 26/09/2007, Arnaud Simon < [EMAIL PROTECTED]> wrote: > > > > > > > > Hi, > > > > > > > > I would like to know more about our testing strategy. So, the unit > > tests > > > > of the broker and client modules are run on a regular base as they > > are > > > > part of the maven build process. We will need to update the client > > > > module tests so that we can configure them to run on a remote > > broker. So > > > > far so good. > > > > They are also three other test modules: > > > > - systest > > > > - integrationtests > > > > - perftests > > > > (Note: the integrationtests module depends on the systests module) > > > > If this is clear to me what perftests are about it is less clear > > what > > > > the difference is between the systests and integrationtests modules. > > Can > > > > somebody explain me? Moreover those tests are not run as part of the > > > > maven build, so my question is when are they run? Shouldn't we run > > them > > > > as part of the maven build? > > > > > > > > Regarding the perftests I really think that we should run them (not > > all > > > > of them but some) as part of the standard build. This could help us > > > > detecting if a change has impacted performances. > > > > > > > > More generally our testing strategy should be discussed during our > > f2f. > > > > But until then, I would suggest that we convert the client module > > tests > > > > for running them against a remote broker and maybe run some > > perftests > > > > with the build. > > > > > > > > Arnaud > > > > > > > > > > > > > > > > > -- > > Martin Ritchie > > > >
