http://cwiki.apache.org/qpid/interop-testing-specification.html
For the interop tests automation scheme. On 26/09/2007, Rupert Smith <[EMAIL PROTECTED]> wrote: > > Arnuad, > > There is also a README.txt in the integration tests directory to explain > its purpose. > The difference is: > > sys tests is for testing the Java client + broker together, as a > single system. > integration tests is for testing the Java client, as an AMQP component, > against any broker or other clients. > > 'sys' and 'integration' may not be the ideal names. However, please don't > suggest renaming them, as it will complicate merges. > > Sys tests are run as part of the Maven build, always using in-vm brokers. > > Integration tests require the independent starting/stopping of a broker to > run through, as well as possibly starting/stopping test clients in other > languages. They could be automated, but it is just a bit trickier to do. It > was my original intention to automate the whole interop test cycle between > all client languages and brokers in Qpid, and a scheme for doing so is given > in the interop test spec. > > Client tests, are supposed to be pure unit tests for the client code, but > I believe they also test client against an in-vm broker? As such, they > should not be run against a remote broker. > > Perftests could be run as part of a build too, although ideally nont > against an in-vm broker. One of the problems with running perftests to > automatically check performance changes on every build, is that the results > of these tests sometimes require 'interpretation'. It would be nice to do > this automatically, for example outputing latency/throughput graphs to a > Wiki page, but this is a whole project in itself. At the moment, I filter > using grep, and open them in a spread sheet. > > I have a macro... > > An interesting test for you to look at might be ImmediateMessageTest. This > one can be run in-vm, against a remote broker, or even distributed accross > many test nodes, all running the exact same test case. This is currently > where I am going with the tests, also with a view to being able to run large > pub/sub tests, and adding *lots* more interop tests, all with a common > framework. > > A situation I am very keen to avoid, is divergence of the test code > between different branches. The tests should be the same accross all, to > show that all work in the same way. Its the only sensible way I can think > of, to ensure that when we eventually move from 0.8 to 0.10 that we carry > accross the behaviour from the old to the new. > > These tests should work at the surface of the product, that is through the > JMS or Qpid APIs in the respective languages. In the Java case at least, > this should be easy because of JMS, and there should be a sub-class to do > Qpid/AMQP specific stuff (perhaps two one for M2/0.8 and one for trunk/0.10 > new client). > > Perhaps we could pull some of the test code (integration + perftests + > Immediate/MandatoryMessageTest) out of the current M2/M2.1/trunk branchfest, > into a separate top level project? That is something that I would like to > do. Arnaud, you have already ported perftests on trunk to work through pure > JMS, so that makes it possible to do this. Thoughts? > > Rupert > > On 26/09/2007, Arnaud Simon <[EMAIL PROTECTED]> wrote: > > > > Hi, > > > > I would like to know more about our testing strategy. So, the unit tests > > of the broker and client modules are run on a regular base as they are > > part of the maven build process. We will need to update the client > > module tests so that we can configure them to run on a remote broker. So > > far so good. > > They are also three other test modules: > > - systest > > - integrationtests > > - perftests > > (Note: the integrationtests module depends on the systests module) > > If this is clear to me what perftests are about it is less clear what > > the difference is between the systests and integrationtests modules. Can > > somebody explain me? Moreover those tests are not run as part of the > > maven build, so my question is when are they run? Shouldn't we run them > > as part of the maven build? > > > > Regarding the perftests I really think that we should run them (not all > > of them but some) as part of the standard build. This could help us > > detecting if a change has impacted performances. > > > > More generally our testing strategy should be discussed during our f2f. > > But until then, I would suggest that we convert the client module tests > > for running them against a remote broker and maybe run some perftests > > with the build. > > > > Arnaud > > > > >
