On 10/5/2011 8:22 AM, Kristian Waagan wrote:
To detect regressions it might be better to run the (old) tests
against a newer server?
Yes, running the old tests against the new product jars almost always
pops issues, if not product issues, at least release note issues.
When I run with mixed client and server versions I always use the old
derbyTesting.jar.
Because the tests are really just a Derby application, I think they are
likely to pop issues that users will encounter. Analysis is hard when
they are just run once a year or so, but I have always found the effort
worth it.
One thought I had had would be that it would be nice if the latest
tests on the previous release branch were run and expected to continue
running against trunk. So if we made an incompatible change on the
trunk, the tests on the 10.8 branch would need to be changed in the same
we expect users to change their own applications and then we would be
more likely to flag and understand incompatibilities as they are
introduced. There is some general work that would have to be done with
the tests first, such as some of the tests are dependent on the number
of system tables. It would be nice in that behavior changes might be
recognized and documented or fixed at the time the incompatibility is
introduced. The risk is that people might be tempted to just backport
the test changes they made in trunk without taking the time to think of
the impact to users. Also it might be time consuming.
Kathey
