This is exactly what the test framework behind ImmediateMessageTest (and MandatoryMessageTest) does. It is not suitable for all tests, as it works around the concept of a 'test circuit' described in the distributed testing proposal. By default these tests pick up an in-vm broker, when run through vanilla JUnit. FrameworkBaseCase is the equivalent of your 'BrokerTestCase'.
You can run them through a different test runner, i.e., the Coordinator, passing in a test distribution engine (-e interop or fanout at the moment), and the test is 'projected' onto a set of distributed test nodes. You can override the use of the in-vm broker, by passing in the 'broker=tcp://localhost' option. They work on the same option set as the perftests currently do, from PingPongProducer (very misleading name, but that's code evolution for you). Also, there is a 'new style' perf test, QpidTestThroughputPerf, which is built on top of this framework, instead of PPP. All this stuff is at the infamous 80% done stage, which is also why I am very keen to stop it fragmenting amongst the different branches. On 26/09/2007, Martin Ritchie <[EMAIL PROTECTED]> wrote: > > On 26/09/2007, Rupert Smith <[EMAIL PROTECTED]> wrote: > > Arnuad, > > > > There is also a README.txt in the integration tests directory to explain > its > > purpose. > > The difference is: > > > > sys tests is for testing the Java client + broker together, as > a > > single system. > > integration tests is for testing the Java client, as an AMQP component, > > against any broker or other clients. > > > > 'sys' and 'integration' may not be the ideal names. However, please > don't > > suggest renaming them, as it will complicate merges. > > > > Sys tests are run as part of the Maven build, always using in-vm > brokers. > > > > Integration tests require the independent starting/stopping of a broker > to > > run through, as well as possibly starting/stopping test clients in other > > languages. They could be automated, but it is just a bit trickier to do. > It > > was my original intention to automate the whole interop test cycle > between > > all client languages and brokers in Qpid, and a scheme for doing so is > given > > in the interop test spec. > > > > Client tests, are supposed to be pure unit tests for the client code, > but I > > believe they also test client against an in-vm broker? As such, they > should > > not be run against a remote broker. > > > > Perftests could be run as part of a build too, although ideally nont > against > > an in-vm broker. One of the problems with running perftests to > automatically > > check performance changes on every build, is that the results of these > tests > > sometimes require 'interpretation'. It would be nice to do this > > automatically, for example outputing latency/throughput graphs to a Wiki > > page, but this is a whole project in itself. At the moment, I filter > using > > grep, and open them in a spread sheet. > > > > I have a macro... > > > > An interesting test for you to look at might be ImmediateMessageTest. > This > > one can be run in-vm, against a remote broker, or even distributed > accross > > many test nodes, all running the exact same test case. This is currently > > where I am going with the tests, also with a view to being able to run > large > > pub/sub tests, and adding *lots* more interop tests, all with a common > > framework. > > > > A situation I am very keen to avoid, is divergence of the test code > between > > different branches. The tests should be the same accross all, to show > that > > all work in the same way. Its the only sensible way I can think of, to > > ensure that when we eventually move from 0.8 to 0.10 that we carry > accross > > the behaviour from the old to the new. > > > > These tests should work at the surface of the product, that is through > the > > JMS or Qpid APIs in the respective languages. In the Java case at least, > > this should be easy because of JMS, and there should be a sub-class to > do > > Qpid/AMQP specific stuff (perhaps two one for M2/0.8 and one for > trunk/0.10 > > new client). > > > > Perhaps we could pull some of the test code (integration + perftests + > > Immediate/MandatoryMessageTest) out of the current M2/M2.1/trunk > branchfest, > > into a separate top level project? That is something that I would like > to > > do. Arnaud, you have already ported perftests on trunk to work through > pure > > JMS, so that makes it possible to do this. Thoughts? > > I'm all for this. The tests should be independent of the AMQP version. > Some tests may be version specific but they could stay with there > version. > > Talking about the systest for a moment: > I have converted some tests to use a static so it is easier to point a > single test at a broker. But it would be nice to wrap up the test > connection setup such that we have some form of a BrokerTestCase. > This would take a broker URL and if required starts an inVM broker. > From there we could then get maven to pass the InVM url for build > tests but would also allow us to set the tests to run from maven > against an external broker. > > I would say a first step. would be to separate the tests that use an > invm broker and those that don't. I'm _not_ saying that all invm and > all non_invm test should live together only that they should be > distinguishable. Example: in the systest module AckTest doesn't use an > inVM broker but MandatoryMessageTest does. This first step would help > ensure we can easily identify tests that can run against a broker. > > If no-one else has the time then I'd start the process by putting > together the BrokerTestCase, or at least something we can debate about > :) > > > Rupert > > > > On 26/09/2007, Arnaud Simon <[EMAIL PROTECTED]> wrote: > > > > > > Hi, > > > > > > I would like to know more about our testing strategy. So, the unit > tests > > > of the broker and client modules are run on a regular base as they are > > > part of the maven build process. We will need to update the client > > > module tests so that we can configure them to run on a remote broker. > So > > > far so good. > > > They are also three other test modules: > > > - systest > > > - integrationtests > > > - perftests > > > (Note: the integrationtests module depends on the systests module) > > > If this is clear to me what perftests are about it is less clear what > > > the difference is between the systests and integrationtests modules. > Can > > > somebody explain me? Moreover those tests are not run as part of the > > > maven build, so my question is when are they run? Shouldn't we run > them > > > as part of the maven build? > > > > > > Regarding the perftests I really think that we should run them (not > all > > > of them but some) as part of the standard build. This could help us > > > detecting if a change has impacted performances. > > > > > > More generally our testing strategy should be discussed during our > f2f. > > > But until then, I would suggest that we convert the client module > tests > > > for running them against a remote broker and maybe run some perftests > > > with the build. > > > > > > Arnaud > > > > > > > > > > > -- > Martin Ritchie >
