On 29/11/2012 12:49 AM, Dan Creswell wrote:
It would be nice to have a stable qa suite branch for testing River
development and another for refactoring and developing the test suite
itself, without interfering with the development process of River. The
test suite requires maintenance too, but right now it's a frightening
prospect.
I think it's essential to sort this out. I reckon we ultimately should be
aiming at having an up-to-date/parallel developed test suite for trunk and
indeed skunk. Bad test/compile process otherwise.
But there's a catch...
I'd like to be able to run an experimental test suite with a stable River
trunk to test the test suite so to speak.
I think that's wise equally I'm pondering how you cope with trunk moving on
and breaking a bunch of your tests for which there would be fixes in the
trunk test suite. Are you happy doing merges as and when or ???
I'm not sure, at this point I just know we've got problems, there's a
lot of coupling in the test suite and a lot of shared state; it's all
based on inheritance.
There are two test types:
1. Jini Standards Compliance
2. Implementation integration tests.
Then there's the harness itself.
I'm not sure how I'd handle trunk moving on, that could be a problem, it
may be acceptable for implementation tests, but not so for Jini
Standards Compliance.
Oh, BTW, when I turned off debugging I had 6 tests fail, then after
running again only 2 tests failed; my recent patches haven't solved
concurrency issues, debugging masked it.
In theory the implementation tests belong in trunk and could utilise a
qa harness binary distribution, similar to how we use junit or jtreg,
but we'd need to decouple the harness to achieve this.
The Jini Standards Tests should probably be bundled separately in a TCK.
Refactoring the harness won't be an easy task, the quality of trunk
depends on it and we're going to experience more concurrency headaches
until we do.
A starting point might be to copy the qa suite and tests to a skunk
directory and run the harness against River release 2.2.0 and earlier,
while refactoring.
We'd need to look at the services the harness provides and how to best
provide them, perhaps using annotations, we could potentially make
integration tests much simpler.
Regards,
Peter.
