Arnuad, There is also a README.txt in the integration tests directory to explain its purpose. The difference is:
sys tests is for testing the Java client + broker together, as a single system. integration tests is for testing the Java client, as an AMQP component, against any broker or other clients. 'sys' and 'integration' may not be the ideal names. However, please don't suggest renaming them, as it will complicate merges. Sys tests are run as part of the Maven build, always using in-vm brokers. Integration tests require the independent starting/stopping of a broker to run through, as well as possibly starting/stopping test clients in other languages. They could be automated, but it is just a bit trickier to do. It was my original intention to automate the whole interop test cycle between all client languages and brokers in Qpid, and a scheme for doing so is given in the interop test spec. Client tests, are supposed to be pure unit tests for the client code, but I believe they also test client against an in-vm broker? As such, they should not be run against a remote broker. Perftests could be run as part of a build too, although ideally nont against an in-vm broker. One of the problems with running perftests to automatically check performance changes on every build, is that the results of these tests sometimes require 'interpretation'. It would be nice to do this automatically, for example outputing latency/throughput graphs to a Wiki page, but this is a whole project in itself. At the moment, I filter using grep, and open them in a spread sheet. I have a macro... An interesting test for you to look at might be ImmediateMessageTest. This one can be run in-vm, against a remote broker, or even distributed accross many test nodes, all running the exact same test case. This is currently where I am going with the tests, also with a view to being able to run large pub/sub tests, and adding *lots* more interop tests, all with a common framework. A situation I am very keen to avoid, is divergence of the test code between different branches. The tests should be the same accross all, to show that all work in the same way. Its the only sensible way I can think of, to ensure that when we eventually move from 0.8 to 0.10 that we carry accross the behaviour from the old to the new. These tests should work at the surface of the product, that is through the JMS or Qpid APIs in the respective languages. In the Java case at least, this should be easy because of JMS, and there should be a sub-class to do Qpid/AMQP specific stuff (perhaps two one for M2/0.8 and one for trunk/0.10 new client). Perhaps we could pull some of the test code (integration + perftests + Immediate/MandatoryMessageTest) out of the current M2/M2.1/trunk branchfest, into a separate top level project? That is something that I would like to do. Arnaud, you have already ported perftests on trunk to work through pure JMS, so that makes it possible to do this. Thoughts? Rupert On 26/09/2007, Arnaud Simon <[EMAIL PROTECTED]> wrote: > > Hi, > > I would like to know more about our testing strategy. So, the unit tests > of the broker and client modules are run on a regular base as they are > part of the maven build process. We will need to update the client > module tests so that we can configure them to run on a remote broker. So > far so good. > They are also three other test modules: > - systest > - integrationtests > - perftests > (Note: the integrationtests module depends on the systests module) > If this is clear to me what perftests are about it is less clear what > the difference is between the systests and integrationtests modules. Can > somebody explain me? Moreover those tests are not run as part of the > maven build, so my question is when are they run? Shouldn't we run them > as part of the maven build? > > Regarding the perftests I really think that we should run them (not all > of them but some) as part of the standard build. This could help us > detecting if a change has impacted performances. > > More generally our testing strategy should be discussed during our f2f. > But until then, I would suggest that we convert the client module tests > for running them against a remote broker and maybe run some perftests > with the build. > > Arnaud > >
