On Thu, 2007-09-27 at 10:46 +0100, Rupert Smith wrote: > There will definitely be tests that are specific to the client > implementations of protocol version. I think client unit tests or > systests are the correct place for these. If tests are pure JMS, then > yes, moving them out to a non-client/protocol specific test pack makes > sense.
I fully agree with that. > This might also provide a good point at which to document the tests. > So if a test is to be moved, it should be fully documented as is it > moved. I would be happy to own the responsibility for cracking the > Javadoc whip. Again +1 > The idea is that if we write a test against 0-8, against the surface > of the product, that tests for some behaviour that we need to > regression test as we move to 0-10, then that will help to compensate > for the fact that the client software is being rewritten from scratch, > and may therefore lose important bug fixes that are currently going > into the 0-8 client on M2. In some cases writing a bug test against > the product surface, versus writing a lower level unit test, might not > be easy to do. For example, there is a test that checks that the store > is free of rejected messages, or something like that. Not something > you can do through the client API. This one is a broker unit test and should stay in the broker module. > Before launching in, and hand coding lots of tests, it is worth > thinking about the test generation/fitness function idea that is > described in my testing proposal. When I wrote the Immediate/Mandatory > tests they were initially written because there was a bug with > returned message processing. By varying the message size, whether it > had no body, or a body, further bugs in this area were shown up. This > led me to think about pulling out all of the different options and > flags that can be used to describe a messaging scenario, and then > thoroughly testing all reasonable combinations (or however many we can > test without it taking forever); there may be bugs lurking in some > particular combinations as has been previously shown to be the case. > > The Immediate/Mandatory tests hand code the specific set of > combinations needed to test that scenario. However, I imagine writing > a general purpose test, that is controlled by a set of Properties, and > using the fitness function gets the set of assertions that apply to a > given scenario. My starting point was all of the properties defined in > PingPongProducer, which have been pulled out into a seperate class, > MessagingTestConfigProperties. Once the general purpose test has been > written, Immediate/Mandatory tests will be replaced by it. The general > purpose test will also be able to run a lot of testing scenarios > through the Coordinator as interop tests. > > These tests need to be run over a 'test circuit' which is a somewhat > limited concept, but useful enough to test a lot of things. The > analogy is an electrical test circuit, like you might have used at > high school. There is a switch, a cell, a light bulb, a transistor > etc. and you can wire it up to produce a limited range of circuits. > The messaging equivalent has a sending side, a receiving side, and > outward route, an inward route (not used by default), and is > parametrized by message size, message rate, persistent/transient, > transactional/non-tx and so on. The sending and receiving sides may > consist of a single producer/consumer in-vm, or many > producers/consumers across many test nodes. > > This circuit idea will not be suitable without extension, to cover > more freestyle tests. The sort of tests that open a connection, create > two sessions, send a message, close one session, send another and so > on. Some of these can be pure JMS, which is good from the regression > testing perspective. They may also benefit from using a testing base > class to handle some configurable aspects of testing common to all > such tests, such as in-vm vs remote, without using the 'test circuit' > concept. > > Rupert What you are describing looks very good to me. I think that our f2f would be the ideal opportunity for us to decide for a testing strategy (taken into account available resources).
