On 27.09.11 18:46, Kathey Marsden wrote:
On 9/27/2011 7:26 AM, Knut Anders Hatlen wrote:
Another question is whether mixed versions is a configuration we need
to support (I don't think it's stated explicitly anywhere that we
actually do support it?).
Years ago, I know this support was very important to several of our
large consumers, but perhaps needs have changed. I am sure those that
expressed the need for this to keep working are not up to 10.7 yet. I
had thought CompatibiltyTest was doing the basic testing for mixed jars.
I guess though, that test is just for basic protocol testing and must
use separate class loaders or the basic testing would have failed.
--- Compatibility test rewrite ---
I recently spent time rewriting the compatibility test (DERBY-2076), and
can confirm that it doesn't test mixed version jars. It uses separate
processes for the client and the server.
However, while rewriting the test, I did find one or two problems
related to mixing code from different releases. For instance, you cannot
shut down a really old server using a newer NetworkServerControl. In
this case I had to fork off a process to shut down the server using the
old code.
The test is dependent on being able to run the newest derbyTesting.jar
with the client and the server code (for instance to find the code for
stored procedures).
I'd be interested in hearing what people believe the compatibility test
should be able to do. The new version of the test only runs a test suite
with a set of client versions against a set of server versions. It
doesn't yet support running the client and the server with different JVM
versions.
--- Compatibility test coverage ---
I have no metrics for what kind of coverage the current compatibility
test gives, but I suspect it is rather low. Adding tests to the
compatibility suite is in theory independent of the test framework, but
running for instance suites.All with mixed versions of the client and
the server will fail for several reasons:
o existing network server setup decorators will interfere with the
running server
o features not existing in the client will be attempted tested if the
newest derbyTesting.jar is used
--- Mixed jars testing ---
--- Run only client or embedded tests when invoking _Suite ---
As an experiment I added functionality to the test framework to run only
client or embedded tests. This included minor changes in
TestConfiguration and relatively small changes in a whole lot of
suite()-methods. I only carried out the experiment for lang and jdbcapi.
When running lang._Suite with a 10.6 server and trunk client/testing, I
got this from the swing testrunner:
589/618, 67 errors, 23 failures.
Many of the issues were related to BOOLEAN and truncate table (not
supported by 10.6).
When running lang._Suite with a trunk server/testing and 10.6 client, I
got this from the swing testrunner:
611/618, 4 errors, 26 failures.
Again many of the failures were related BOOLEAN.
I don't know if it is worth doing this kind of testing, and how to
interpret the results, but the ability to run only embedded or client
tests may be something people find valuable.
Mixed jars testing requires manual inspection of the results, since the
test cases don't carry information about which versions of Derby support
the features being tested.
To detect regressions it might be better to run the (old) tests against
a newer server?
--
Kristian
[ snip ]
