Also, we have an internal test instructions page, that explains how to run all these tests. If Rob approves, we could put this on the Apache Wiki?
On 26/09/2007, Rupert Smith <[EMAIL PROTECTED]> wrote: > > http://cwiki.apache.org/qpid/interop-testing-specification.html > > For the interop tests automation scheme. > > On 26/09/2007, Rupert Smith <[EMAIL PROTECTED]> wrote: > > > > Arnuad, > > > > There is also a README.txt in the integration tests directory to explain > > its purpose. > > The difference is: > > > > sys tests is for testing the Java client + broker together, as > > a single system. > > integration tests is for testing the Java client, as an AMQP component, > > against any broker or other clients. > > > > 'sys' and 'integration' may not be the ideal names. However, please > > don't suggest renaming them, as it will complicate merges. > > > > Sys tests are run as part of the Maven build, always using in-vm > > brokers. > > > > Integration tests require the independent starting/stopping of a broker > > to run through, as well as possibly starting/stopping test clients in other > > languages. They could be automated, but it is just a bit trickier to do. It > > was my original intention to automate the whole interop test cycle between > > all client languages and brokers in Qpid, and a scheme for doing so is given > > in the interop test spec. > > > > Client tests, are supposed to be pure unit tests for the client code, > > but I believe they also test client against an in-vm broker? As such, they > > should not be run against a remote broker. > > > > Perftests could be run as part of a build too, although ideally nont > > against an in-vm broker. One of the problems with running perftests to > > automatically check performance changes on every build, is that the results > > of these tests sometimes require 'interpretation'. It would be nice to do > > this automatically, for example outputing latency/throughput graphs to a > > Wiki page, but this is a whole project in itself. At the moment, I filter > > using grep, and open them in a spread sheet. > > > > I have a macro... > > > > An interesting test for you to look at might be ImmediateMessageTest. > > This one can be run in-vm, against a remote broker, or even distributed > > accross many test nodes, all running the exact same test case. This is > > currently where I am going with the tests, also with a view to being able to > > run large pub/sub tests, and adding *lots* more interop tests, all with a > > common framework. > > > > A situation I am very keen to avoid, is divergence of the test code > > between different branches. The tests should be the same accross all, to > > show that all work in the same way. Its the only sensible way I can think > > of, to ensure that when we eventually move from 0.8 to 0.10 that we > > carry accross the behaviour from the old to the new. > > > > These tests should work at the surface of the product, that is through > > the JMS or Qpid APIs in the respective languages. In the Java case at least, > > this should be easy because of JMS, and there should be a sub-class to do > > Qpid/AMQP specific stuff (perhaps two one for M2/0.8 and one for trunk/0.10 > > new client). > > > > Perhaps we could pull some of the test code (integration + perftests + > > Immediate/MandatoryMessageTest) out of the current M2/M2.1/trunk branchfest, > > into a separate top level project? That is something that I would like to > > do. Arnaud, you have already ported perftests on trunk to work through pure > > JMS, so that makes it possible to do this. Thoughts? > > > > Rupert > > > > On 26/09/2007, Arnaud Simon < [EMAIL PROTECTED]> wrote: > > > > > > Hi, > > > > > > I would like to know more about our testing strategy. So, the unit > > > tests > > > of the broker and client modules are run on a regular base as they are > > > part of the maven build process. We will need to update the client > > > module tests so that we can configure them to run on a remote broker. > > > So > > > far so good. > > > They are also three other test modules: > > > - systest > > > - integrationtests > > > - perftests > > > (Note: the integrationtests module depends on the systests module) > > > If this is clear to me what perftests are about it is less clear what > > > the difference is between the systests and integrationtests modules. > > > Can > > > somebody explain me? Moreover those tests are not run as part of the > > > maven build, so my question is when are they run? Shouldn't we run > > > them > > > as part of the maven build? > > > > > > Regarding the perftests I really think that we should run them (not > > > all > > > of them but some) as part of the standard build. This could help us > > > detecting if a change has impacted performances. > > > > > > More generally our testing strategy should be discussed during our > > > f2f. > > > But until then, I would suggest that we convert the client module > > > tests > > > for running them against a remote broker and maybe run some perftests > > > with the build. > > > > > > Arnaud > > > > > > > > >
