On 26/09/2007, Rupert Smith <[EMAIL PROTECTED]> wrote: > Arnuad, > > There is also a README.txt in the integration tests directory to explain its > purpose. > The difference is: > > sys tests is for testing the Java client + broker together, as a > single system. > integration tests is for testing the Java client, as an AMQP component, > against any broker or other clients. > > 'sys' and 'integration' may not be the ideal names. However, please don't > suggest renaming them, as it will complicate merges. > > Sys tests are run as part of the Maven build, always using in-vm brokers. > > Integration tests require the independent starting/stopping of a broker to > run through, as well as possibly starting/stopping test clients in other > languages. They could be automated, but it is just a bit trickier to do. It > was my original intention to automate the whole interop test cycle between > all client languages and brokers in Qpid, and a scheme for doing so is given > in the interop test spec. > > Client tests, are supposed to be pure unit tests for the client code, but I > believe they also test client against an in-vm broker? As such, they should > not be run against a remote broker. > > Perftests could be run as part of a build too, although ideally nont against > an in-vm broker. One of the problems with running perftests to automatically > check performance changes on every build, is that the results of these tests > sometimes require 'interpretation'. It would be nice to do this > automatically, for example outputing latency/throughput graphs to a Wiki > page, but this is a whole project in itself. At the moment, I filter using > grep, and open them in a spread sheet. > > I have a macro... > > An interesting test for you to look at might be ImmediateMessageTest. This > one can be run in-vm, against a remote broker, or even distributed accross > many test nodes, all running the exact same test case. This is currently > where I am going with the tests, also with a view to being able to run large > pub/sub tests, and adding *lots* more interop tests, all with a common > framework. > > A situation I am very keen to avoid, is divergence of the test code between > different branches. The tests should be the same accross all, to show that > all work in the same way. Its the only sensible way I can think of, to > ensure that when we eventually move from 0.8 to 0.10 that we carry accross > the behaviour from the old to the new. > > These tests should work at the surface of the product, that is through the > JMS or Qpid APIs in the respective languages. In the Java case at least, > this should be easy because of JMS, and there should be a sub-class to do > Qpid/AMQP specific stuff (perhaps two one for M2/0.8 and one for trunk/0.10 > new client). > > Perhaps we could pull some of the test code (integration + perftests + > Immediate/MandatoryMessageTest) out of the current M2/M2.1/trunk branchfest, > into a separate top level project? That is something that I would like to > do. Arnaud, you have already ported perftests on trunk to work through pure > JMS, so that makes it possible to do this. Thoughts?
I'm all for this. The tests should be independent of the AMQP version. Some tests may be version specific but they could stay with there version. Talking about the systest for a moment: I have converted some tests to use a static so it is easier to point a single test at a broker. But it would be nice to wrap up the test connection setup such that we have some form of a BrokerTestCase. This would take a broker URL and if required starts an inVM broker. >From there we could then get maven to pass the InVM url for build tests but would also allow us to set the tests to run from maven against an external broker. I would say a first step. would be to separate the tests that use an invm broker and those that don't. I'm _not_ saying that all invm and all non_invm test should live together only that they should be distinguishable. Example: in the systest module AckTest doesn't use an inVM broker but MandatoryMessageTest does. This first step would help ensure we can easily identify tests that can run against a broker. If no-one else has the time then I'd start the process by putting together the BrokerTestCase, or at least something we can debate about :) > Rupert > > On 26/09/2007, Arnaud Simon <[EMAIL PROTECTED]> wrote: > > > > Hi, > > > > I would like to know more about our testing strategy. So, the unit tests > > of the broker and client modules are run on a regular base as they are > > part of the maven build process. We will need to update the client > > module tests so that we can configure them to run on a remote broker. So > > far so good. > > They are also three other test modules: > > - systest > > - integrationtests > > - perftests > > (Note: the integrationtests module depends on the systests module) > > If this is clear to me what perftests are about it is less clear what > > the difference is between the systests and integrationtests modules. Can > > somebody explain me? Moreover those tests are not run as part of the > > maven build, so my question is when are they run? Shouldn't we run them > > as part of the maven build? > > > > Regarding the perftests I really think that we should run them (not all > > of them but some) as part of the standard build. This could help us > > detecting if a change has impacted performances. > > > > More generally our testing strategy should be discussed during our f2f. > > But until then, I would suggest that we convert the client module tests > > for running them against a remote broker and maybe run some perftests > > with the build. > > > > Arnaud > > > > > -- Martin Ritchie
