The cancelled F2F would have been a good time to discuss. However, not wanting to appear cynical, but the F2F might be a long way off materializing. Lets just get on with it now, over the list.
On 27/09/2007, Arnaud Simon <[EMAIL PROTECTED]> wrote: > > On Thu, 2007-09-27 at 10:46 +0100, Rupert Smith wrote: > > There will definitely be tests that are specific to the client > > implementations of protocol version. I think client unit tests or > > systests are the correct place for these. If tests are pure JMS, then > > yes, moving them out to a non-client/protocol specific test pack makes > > sense. > > I fully agree with that. > > > This might also provide a good point at which to document the tests. > > So if a test is to be moved, it should be fully documented as is it > > moved. I would be happy to own the responsibility for cracking the > > Javadoc whip. > > Again +1 > > > The idea is that if we write a test against 0-8, against the surface > > of the product, that tests for some behaviour that we need to > > regression test as we move to 0-10, then that will help to compensate > > for the fact that the client software is being rewritten from scratch, > > and may therefore lose important bug fixes that are currently going > > into the 0-8 client on M2. In some cases writing a bug test against > > the product surface, versus writing a lower level unit test, might not > > be easy to do. For example, there is a test that checks that the store > > is free of rejected messages, or something like that. Not something > > you can do through the client API. > > This one is a broker unit test and should stay in the broker module. > > > Before launching in, and hand coding lots of tests, it is worth > > thinking about the test generation/fitness function idea that is > > described in my testing proposal. When I wrote the Immediate/Mandatory > > tests they were initially written because there was a bug with > > returned message processing. By varying the message size, whether it > > had no body, or a body, further bugs in this area were shown up. This > > led me to think about pulling out all of the different options and > > flags that can be used to describe a messaging scenario, and then > > thoroughly testing all reasonable combinations (or however many we can > > test without it taking forever); there may be bugs lurking in some > > particular combinations as has been previously shown to be the case. > > > > The Immediate/Mandatory tests hand code the specific set of > > combinations needed to test that scenario. However, I imagine writing > > a general purpose test, that is controlled by a set of Properties, and > > using the fitness function gets the set of assertions that apply to a > > given scenario. My starting point was all of the properties defined in > > PingPongProducer, which have been pulled out into a seperate class, > > MessagingTestConfigProperties. Once the general purpose test has been > > written, Immediate/Mandatory tests will be replaced by it. The general > > purpose test will also be able to run a lot of testing scenarios > > through the Coordinator as interop tests. > > > > These tests need to be run over a 'test circuit' which is a somewhat > > limited concept, but useful enough to test a lot of things. The > > analogy is an electrical test circuit, like you might have used at > > high school. There is a switch, a cell, a light bulb, a transistor > > etc. and you can wire it up to produce a limited range of circuits. > > The messaging equivalent has a sending side, a receiving side, and > > outward route, an inward route (not used by default), and is > > parametrized by message size, message rate, persistent/transient, > > transactional/non-tx and so on. The sending and receiving sides may > > consist of a single producer/consumer in-vm, or many > > producers/consumers across many test nodes. > > > > This circuit idea will not be suitable without extension, to cover > > more freestyle tests. The sort of tests that open a connection, create > > two sessions, send a message, close one session, send another and so > > on. Some of these can be pure JMS, which is good from the regression > > testing perspective. They may also benefit from using a testing base > > class to handle some configurable aspects of testing common to all > > such tests, such as in-vm vs remote, without using the 'test circuit' > > concept. > > > > Rupert > > What you are describing looks very good to me. I think that our f2f > would be the ideal opportunity for us to decide for a testing strategy > (taken into account available resources). > > > > >
